H4CF Conversion Toolkit History =============================== This file contains development history of H4CF Conversion Toolkit (h4cf). 02. Release Information for h4cf 1.0 01. Release Information for h4cf 1.0.beta [Search on the string '%%%%' for per-release section breaks.] %%%%1.0%%%% H4CF Conversion Toolkit version 1.0 released on 2015-01-8 =============================================================== INTRODUCTION ============ This document describes the differences between version 1.0.beta and 1.0, and contains information on the platforms tested and known problems in version 1.0. For more details about the previous releases, see the file HISTORY.txt in the docs/ directory of the H4CF Conversion Toolkit source. If you have any questions or comments, please send them to the HDF-EOS forum: http://hdfeos.org/forums/ CONTENTS ======== - New Features - Bug Fixes since version 1.0.beta - Supported Platforms and Compilers - Known Problems - Others New Features ============ Product supported ------------------ Support TRMM version 7 and AIRS version 6 products to follow CF Error handling ---------------- Code is enhanced to ensure the graceful exit and the release of resources. Test suite ---------- - TRMM version 7 and AIRS version 6 tests are added to the testsuite. Support Platforms and Compilers ======================================= - Linux 2.6.32-431.11.2.el6.x86_64 #1 SMP GNU/Linux gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4) - Linux 2.6.18-308.13.1.el5PAE #1 SMP i686 i386 GNU/Linux gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-51) - Linux 2.6.18-348.1.1.el5 #1 SMP x86_64 GNU/Linux gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-52) - Darwin Kernel Version 12.5.0 x86_64 gcc version 4.2.1 (Based on Apple Inc. build 5658) Bug Fixes since version 1.0.beta ================================= - Memory leaks are checked with valgrind for fake and NASA files. Several severe memory leaking problems are fixed. - MODIS swath dimension maps apply to the corresponding data fields. - MERRA HDF-EOS2 files are fully converted to follow CF conventions. - For an HDF-EOS2 hybrid file, the subset of added SDS objects doesn't work properly. This was fixed in this release. - Fixed an HDF4 vdata field subset bug when a vdata is under a vgroup. Known Problems ============== - If your netcdf library doesn't support NetCDF-4 output, the h4tonccf utility cannot handle unsigned integer 32 type data and attribute. Either the converted file has negative values or the netcdf library throws an error during conversion because NetCDF-3 doesn't support unsigned int32 type. The behavior varies depending on netcdf library version and platform. - For some converted files (e.g., MISR-AM1_ACP-mixture-F06_0021.hdf.nc), ToolsUI (netcdf-java) will not show some file attributes although ncdump (netcdf-c) shows them correctly. This was reported to the netcdf-java developer at Unidata. - MISR SOM projection files are not supported in this release. - For an HDF4-added vdata object in an HDF-EOS2 hybrid file, the subset of a vdata field doesn't work. This is a really rare case. - The package is tested with MacOS 10.9(Darwin Kernel Version 13.4.0). The compiler is LLVM version 6.0 (clang-600.0.51) (based on LLVM 3.5svn). However, the package misses converting the HDF4-added objects in an HDF-EOS2 hybrid file. Further investigation indicates that this issue is caused in the HDF-EOS2 layer rather than in this package. Others ============== - For this release, this toolkit is tested with HDF4.2.10, HDF-EOS2.19, HDF5-1.8.13(if using netCDF-4) and netCDF-4.3.2. - Users can use NcML or NCO to edit the converted netCDF files. A NcML example is added under the example page. One may also find the information on how to use NcML to edit a netCDF file in the User's guide. NCO examples can be found in NCO's website at sourceforge. - User's guide and reference manual are untouched since 1.0.beta. - %%%%1.0.beta%%%% H4CF Conversion Toolkit version 1.0.beta released on 2013-06-15 =============================================================== INTRODUCTION ============ This document describes the differences between version 1.0.alpha and 1.0.beta, and contains information on the platforms tested and known problems in version 1.0.beta. For more details about the previous release, see the file HISTORY.txt in the docs/ directory of the H4CF Conversion Toolkit source. If you have any questions or comments, please send them to the HDF-EOS forum: http://hdfeos.org/forums/ CONTENTS ======== - New Features - Bug Fixes since version 1.0.alpha - Supported Platforms and Compilers - Known Problems - Others New Features ============ Configuration ------------- - autoconf can determine whether NetCDF-4 or NetCDF-3 will be used. API Name Change --------------- - Some attribute-related APIs are shortened. For example, h4cf_get_var_attr_type has been reduced to h4cf_get_attr_type since we also support the retrieval of file attribute type in this release. Vdata handling -------------- - In the previous version, vdata attribute values, vdata field values and vdata field attribute values are translated to string. This mapping is not right. Now original vdata data and attribute values will be kept. - Since there are no equivalent vdata objects in the h4cf output, to conform to the netCDF classic model, the vdata attributes will be mapped to file attributes and the vdata field attributes will be mapped to variable attributes. Unsigned Data Type ------------------ - h4tonccf improved the handling of unsigned data type. If NetCDF-3 output is desired, unsigned int8 and int16 type will be up-scaled to int16 and int32 type respectively. Test suite ---------- - A test suite with sample data files and API test programs is added. - Users can download and test conversion tool with sample NASA data. Examples -------- - More file-type-specific examples are provided. - An NcML example is added for post-editing the converted NetCDF file. Support Platforms and Compilers ======================================= - Linux 2.6.32-358.2.1.el6.x86_64 #1 SMP x86_64 GNU/Linux gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-3) - Linux 2.6.18-308.13.1.el5PAE #1 SMP i686 i386 GNU/Linux gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-51) - Linux 2.6.18-348.1.1.el5 #1 SMP x86_64 GNU/Linux gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-52) - Darwin Kernel Version 11.4.2 x86_64 gcc version 4.2.1 (Based on Apple Inc. build 5658) - Darwin Kernel Version 12.3.0 x86_64 Apple clang version 4.0 (tags/Apple/clang-421.0.60) Bug Fixes since version 1.0.alpha ================================= - The -O0 compiler option is no longer necessary. - Memory leaks are checked with valgrind and leaking problems are fixed. - HDF-EOS2 Grids that use the LAMAZ projection are supported. - Type mismatch problem between CF attributes and variable type is fixed. - h4cf_get_file_attrs() returns the list of attributes instead of string map. This will allow to write attributes with different types, not just string. - Bugs related to subsetting vdata field values are fixed. - The conversion tool doesn't insert an arbitrary _FillValue attribute that doesn't exist in the original file. - Name clashing routines are updated when handling HDF4 products. Now the name clashing is more robust and the original object names will be used to generate the final CF-compliant variable names if no name clashing is found. - The name clashing issue related to the hybrid HDF-EOS2 objects has been fixed. - A few critical bugs have been fixed related to making MODIS scale and offset and valid_range attributes to follow CF. Now MODIS products in our testsuite are translated to conform to CF conventions. Known Problems ============== - If your netcdf library doesn't support NetCDF-4 output, the h4tonccf utility cannot handle unsigned integer 32 type data and attribute. Either the converted file has negative values or the netcdf library throws an error during conversion because NetCDF-3 doesn't support unsigned int32 type. The behavior varies depending on netcdf library version and platform. - The library doesn't apply the scale factor for MOD04_L2 and MOD09GA products. Users should change the scale factor attribute value from "x" to "1/x" (e.g., 25 to 0.04) for these products with NCO or NcML to obtain the correct values. See the examples/ncml/README.txt directory for how to edit attributes from the converted NetCDF files. - For some converted files (e.g., MISR-AM1_ACP-mixture-F06_0021.hdf.nc), ToolsUI (netcdf-java) will not show some file attributes although ncdump (netcdf-c) shows them correctly. This is reported to the netcdf-java developer at Unidata. - MISR SOM projection files are not supported in this release. Others ============= - We remove the cdldumper source code from this release. Applications can use the current API 02. Release Information for h4cf 1.0.alpha [Search on the string '%%%%' for per-release section breaks.] %%%%1.0.alpha%%%% h4cf 1.0.alpha released on July 14, 2011 ================================================================================ I.INTRODUCTION The HDF CF (H4CF) conversion library can access various types of NASA HDF4 and HDF-EOS2 files by following CF conventions. We have incorporated the knowledge and information obtained from the NASA HDF-EOS2 and HDF4 files and the corresponding documents published by NASA data centers into this conversion library. We also have implemented an HDF to netCDF conversion tool that follows CF conventions by using this conversion library. A debugging dumper tool in Common Data Language(CDL) and several example source code files that demonstrate the usage of the conversion library are also included. The library and the tools are implemented in C++. We have tested the h4cf library against some NASA MODIS, AIRS, AMSR_E, MISR, MOPITT, TRMM, and OBPG products. In this release note, we will briefly provide some information on how we made this conversion library be CF-compliant for different NASA products. This document also describes limitations we have encountered so far. II. PRODUCT SUMMARY This release includes the following products: - One static library: h4cf 1.0 alpha - One HDF4 to netCDF-CF conversion tool: h4tonccf - One CDL dumper debugging tool: cdldumper - Several example source codes on how to use the high-level conversion library APIs. See "examples" directory. III. DOCUMENTATION In this release, we provide the following documents: Under the top directory: README - introduction of the package INSTALL - How to compile and build the conversion library and the tools from the source codes release.txt - This file Under the doc directory: USER's guide - Somewhat detailed information to help users to use the conversion library and the tools. Reference manual - Describe the usage of high-level conversion library APIs. IV. PLATFORM and COMPILER TESTED The conversion library and the utilities have been tested under Linux 32-bit and 64-bit platforms(Linux 2.6.18). The g++ compiler version is 4.1.2. V. CF-COMPLIANT In this section,we provide some information on how we make NASA HDF4 and HDF-EOS2 data products be CF-compliant. More information about the design and the implementation will be provided in a programmer's note in the future release. 1. The conversion library --------------- 1.1. HDF4 VDATA --------------- The HDF4 VDATA does not have the dimension name for the Vdata field. During the conversion, we provide a fake 1D dimension name for the variable mapped from Vdata fields. We do so because the CF conventions require that a dimension has a name. The dimension size is equal to the length of string that represents the Vdata field. The string representation is accessible through high-level APIs. For the current release, The output of the object mapped from VDATA is always in string format. We may provide other output formats in the future release. ------------------ 1.2. OBPG Products ------------------ As netCDF-3 does not support unsigned int, we convert unsigned char and unsigned int8 to int16, and unsigned int16 to int32. We also add the CF "_FillValue" attribute to the final output since the OBPG product doesn't provide the "_FillValue" attribute although we've verified that 255 is the fill value for unsigned char and unsigned int8 variables and 65535 is the fill value for the unsigned int16 variables. For OBPG L3m products, we add two additional CF attributes, "scale_factor" and "add_offset" if their scaling function is linear. The values of these two attributes are copied directly from file attributes, "Slope" and "Intercept". It should be noted that Panoply may not show these two attributes although ncdump or HDFView confirms their existence. ------------------------------------------- 1.3. Handling MODIS scale_factor and offset ------------------------------------------- Because different MODIS products use different scale_factor and add_offset equations, in order to follow CF conventions, we have to handle them differently, 1.3.1 MODIS scale_factor and add_offset There are three different scale_factor and add_offset equations. 1) For MODIS L1B, MODIS 03,05,06,07,08,09A1,17 and ATML2 level 2 swath products, MCD43B4, MCD43C1, MOD and MYD 43B4 level 3 grid products, the scale offset equation is correct_data_value = scale * (raw_data_value - offset). This equation doesn't follow the linear scale_factor and add_offset equation defined in CF conventions. 2) For MODIS 13, MODIS 09GA, and MODIS 09GHK, the scale offset equation is correct_data_value=(raw_data_value -offset)/scale_factor. This equation doesn't follow the linear scale_factor and add_offset equation defined in CF conventions. 3) For MODIS 11 level 2 swath products, the equation is correct_data_value = scale * raw_data_value + offset. This equation follows the CF conventions. 1.3.2 How to handle MODIS scale_factor and add_offset 1)For MODIS products of which the scale offset equation follows CF conventions, we have to ensure that the scale and offset attribute names to be the corresponding CF attribute names. 2)For MODIS products of which the scale offset equation doesn't follow CF conventions, we have to calculate the final values based on the provided equation. Since the datatype of most MODIS data fields are integer,we have to convert the data type from integer to float when applying the scale offset equation so that the final data values of some fields will not lose precisions. Since some netCDF visualization tools will apply the linear scale and offset equation to the data value if the CF "scale_factor" and "add_offset" attributes appear, we change the "scale_factor" to "scale_factor_modis" and change the "add_offset" to "add_offset_modis" to prevent the recalculation of the data value. ------------------------------------ 2. HDF4 to netCDF-CF conversion tool ------------------------------------ As netCDF-3 does not support unsigned int, h4tonccf converts unsigned int to int to bypass the limitation. ------------ 3. CDLDUMPER ------------ CDLDUMPER tool will not adjust data values or datatypes, It will dump the data based on their original datatype and data values. For VDATA objects, the original datatype will be kept by the conversion library. VI. LIMITATIONS 1. Auto-configuration and auto-build The auto-configuration script in this release is primitive. An extra CXXFLAGS=-O0 needs to be provide when compiling the package with "make" command. The auto-configuration and auto-build need to be more robust in the future. 2. NetCDF-4 The HDF4 to netCDF-CF conversion tool can also generate a netCDF-4 file if one configures with HDF5 and netCDF4. However, we haven't tested this option thoroughly. Based on our limited testing,the conversion from some big MODIS files will fail. 3. MISR SOM projection and LAMAZ projection The current release doesn't support HDF-EOS2 grid files that use Space-Oblique Mercator(SOM) and Lambert Azimuthal Equal Area(LAMAZ) projections. 4. Other unsupported NASA HDF4 and HDF-EOS2 products and special NASA files The current release doesn't support MERRA, AVHRR, LISTO, QuikSCAT, SeaWind and MOPITT level 2 products. Some MODIS HDF-EOS2 files have wrong projection parameters. The current release cannot handle these files. They may be handled in the future release as needed. 5. Other modules In our original design, we also planned to implement a NcML module for the conversion library to obtain supplementary metadata such as some key CF attributes. This NcML module is not implemented in this release. We may implement this module in the future based on the user's requests. 6. Misc. Some products do not have the "_FillValue" attribute. This may cause some netCDF tools, such as Panoply, to fail to plot the data correctly. During our testing, we find two such files: CER_ES4_TRMM-PFM_Edition1_009001.199808.hdf and T2010001000000.L2_LAC_SST.hdf. For MYDATML2.A2002184.2200.005.2007068182350.hdf, we use the HDF-EOS2 dimension map technique to calculate the latitude and longitude. We will use the latitude and longitude provided by the file directly in the future release. For MYDARNSS.Barrow.A2002184.2200.005.2007051063709.hdf, we cannot find a way to know which MODIS product this belongs to, so we don't try to find which scale and offset equation this product will use. Consequently, no adjustment of data values apply to this product. For MYD17A2.A2007073.h09v08.005.2007096132046.hdf, the "_FillValue" is 32767, but we find that it is actually 32766, so the recalculation may be wrong as we do not know it beforehand. For this release, we can not handle the "_FillValue" attribute correctly for MYD09A1.A2007273.h03v07.005.2007285103507.hdf, we may fix this bug in the next release.