Practical Methods for Making HDF Data Usable by NetCDF-based Tools

A number of powerful NetCDF-based tools exist for visualizing and analyzing remote sensing data, such as IDV, McIDAS-V, Panoply, Ferret and GrADS. As a result, the GES DISC has been deploying a number of different strategies to make its native HDF, HDF-EOS2 and HDF-EOS5 data available in usable forms to these NetCDF tools. The most elaborate of these is an online, on-the-fly Web Service to convert HDF and HDF-EOS2 data into NetCDF-CF1. The end result is highly effective in making the data usable, even Level 2 swath data, which can be visualized in IDV. Unfortunately, the code is relatively convoluted and inflexible due to the difficulty of locating values in the HDF input file to describe the vertical dimension in CF1. (This could be made easier if HDF dimscales could be written through the HDF-EOS2 interface.) A similar approach has been to serve certain datasets through a Web Coverage Server developed by George Mason University, which is subject to many of the same issues. A third approach has been to use OPeNDAP to make data available to NetCDF-based tools. The HDF5 handler has proven to be effective for Level 3 gridded data in this respect at very low cost. These tools can access remote data directly through OPeNDAP, or the HDF5 data can be downloaded as a local NetCDF file. However, this is limited to HDF5 and does not work for Level 2 swath data. Recently, we have begun deploying a fourth approach, the Lats4D script running within GrADS. Though slower than our custom C++ converter, it supports (in principle) conversion of any Level 3 data that can be read in through GrADS, allowing for dimension definition through GrADS control files. Experimenting with Lats4D also led to the serendipitous discovery that NetCDF-based tools based on the netcdf-java library (e.g., IDV and Panoply) can actually read certain HDF files. If this capability could be generalized, the expense of netCDF conversion could be eliminated. Alas, the hierarchies written by HDF-EOS block this capability; we welcome ideas and contributions from the HDF and NetCDF communities to devising a solution to this impasse.

Back to Agenda

Last modified: 06/02/2017