|
|
|
Since the GES DISC began work with HDF in 1992, automation of data management activities has benefited significantly from the small command-line utilities provided with the HDF package. This is because the command-line processes integrate easily into the automated system through scripts. For example, ncdump is used to extract metadata in the S4PM processing system, while hdiff provides comparisons against reference data in the science software integration and test process. The most far-reaching impact has come from hrepack, which finally enabled practical internal compression of data products, in turn enabling the current EOSDIS Evolution move to online archives at the GES DISC and other sites.
Standalone HDF utiltiies also have the benefit of lowering the entrance threshold for new HDF users; tools such as ncdump and hdp have been useful in extracting data to ASCII or flat binary for input into other data analysis tools. To be fully effective in lowering the threshold, all utilities should be provided as standalone, precompiled static binaries to remove the barrier of having to build and install the HDF libraries.
Now, our most pressing need for utilities is shifting toward more comprehensive data reformatters, such as tools for reformatting exogenous formats (e.g., GRIB) to HDF. Such reformatting is complicated by data product variations in semantics and data syntax within the "standard" formats and structures. However, this might be attacked by data structure and semantic conventions (cf. COADS) or providing external information in the way of semantic or syntactic "maps" (e.g. ESML). As data fusion and data-model interoperability become more important to NASA and other agencies, format conversion will become more prominent, and such utilities could benefit not just data centers but the wider community of HDF users as well.