NcML Examples for NASA HDF/HDF-EOS Files

1. Introduction

NcML is an XML representation of metadata in an NetCDF file. As an output, you can dump an NetCDF file into NcML. As an input, you can create a new NetCDF file or modify existing one.

Although NcML is mainly for NetCDF files, some NASA MEaSUREs projects produce data in NetCDF-4 and NetCDF-4 uses HDF5 as underlying storage format. In addition, NcML is useful for the NetCDF files that are converted from HDF4 files via the H4CF Conversion Toolkit. NcML is also useful for the NASA HDF4/HDF5 files served by CF-enabled HDF4 OPeNDAP Handler or HDF5 OPeNDAP handler. Finally, NcML is useful for HDF4 and HDF5 files because NcML is handled by NetCDF-Java that can access some objects in HDF (e.g., SDS in HDF4). Thus, we provide some basic and useful examples of using NcML on NASA HDF products.

Please note that these NcML examples are tested with NetCDF Java/ToolsUI version 4.3, Panoply version 4.1.8, and IDV version 4.0u1. Older versions of NetCDF Java and tools will not work.

2. Add Variable and Dimension

NcML allows you to add new variables or rename existing variables. In addition, it allows you to define dimension names and associate them with variables. Let's first explore this feature of NcML using a real NASA data.

NASA JPL PO.DAAC data center provides Aquarius HDF5 data access via OPeNDAP. However, PO.DAAC does not use CF-enabled HDF5 handler yet so PO.DAAC's OPeNDAP service does not provide latitude and longitude dataset explicitly. For example, the OPeNDAP Data Descriptor Structure (DDS) output from PO.DAAC looks as follows:

If CF option were enabled in HDF5 OPeNDAP handler, the above output would include lat/lon datasets as follows:

The missing geolocation information (i.e., lat/lon datasets) fails NetCDF-Java visualization clients like IDV, Panoply, and McIDAS-V to visualize data via OPeNDAP.

However, writing a simple NcML can solve this problem because the variable NcML element allows you to define a new variable and the dimension NcML element allows you to associate data variable with coordinate variables. The example NcML document below in Figure 1 illustrates how the NcML elements are used with the remote Aquarius HDF5 data at PO.DAAC.

In the above example, the value of location element is the PO.DAAC's OPeNDAP server data URL. From the OPeNDAP DDS output, we can easily figure out that the size of dimension is 180 and 360. From Aquarius User's Guide section 3.7, we know that level 3 mapped products have a spatial resolution of 1 degree. Thus, we first assign a name to each dimension using dimension element and then we add two new coordinate variables Latitude and Longitude with values in degrees. The value of shape attribute determines the dimension size of a variable. Therefore, it's important to define dimension with name using dimension element first before the variable element is used. Finally, we rename the original _l3m_data variable as l3m_data by dropping the first underscore character and associate the l3m_data data variable with Latitude and Longitude coordinate variables using the shape attribute that contains the names of dimensions.

You can download the above NcML file and open the NcML file with any NetCDF-Java tool like Panoply. Since we added coordinate variables, Panoply can now visualize the l3m_data as shown in Figure 2.

3. Add or Edit Attribute

Providing coordinate variables, shared dimension names, and shape information is a minimal requirement for NetCDF-Java data visualization tools that follow the CF conventions . You can add more descriptive physical meanings to data by adding or editing attributes following the CF conventions. For example, the variable name l3m_data in Figure 2 doesn't give any clue for the values [11.1, 39.2] in the color bar. What is worse, the plot doesn't provide any unit information. You can improve the plot quite easily simply by adding a few more attributes that NetCDF-Java tools can pick up automatically through the CF conventions. To add (or overwrite) a variable's attribute, you can use attribute NcML tag as shown in Figure 3.

The above NcML file extends the NcML in Figure 1 by adding several attribute elements under each variable element. The units and long_name attribute values are obtained from the OPeNDAP Data Attribute Structure (DAS) output under H5 GLOBAL as shown below:
You can also obtain the same information from the Aquarius User's Guide.

You can download the above NcML file and check the NcML file with Panoply again. Since we added units and long_name attribute, Panoply can now visualize the l3m_data with name Sea Surface Salinity and unit psu as shown in Figure 5 below.

4. Aggregate Data from Multiple Files

Another powerful feature of NcML is the ability to create one virtual data file by aggregating data from multiple files. To aggregate data in NcML, you can use aggregation tag as shown in Figure 6.

The above code attempts to aggregate 3 different remote data files of 3 different dates and create a new 3D variable from the 2D /l3m_data dataset by adding the time dimension. However, the above code will not work because shared dimensions are missing.

To make the above aggregation work, you should add shared dimensions first by writing 3 different NcML files. Please refer to the section 2 above on how to add dimensions. You may also download and review each NcML file used in this aggregation example:

• Q20131272013133.L3m_7D_SCI_V2.0_SSS_1deg.ncml
• Q20131342013140.L3m_7D_SCI_V2.0_SSS_1deg.ncml
• Q20131412013147.L3m_7D_SCI_V2.0_SSS_1deg.ncml

The below is a modified version of NcML that is derived from the Figure 6 above.
If you open the above NcML file with IDV (after putting the three additional NcML files under the same directory), you can animate the aggregated data in IDV as shown in Figure 8 below.

5. Augment Data served by OPeNDAP using NcML Handler

OPeNDAP provides a Hyrax module called NcML Handler. With this module, you can augment your data served on your Hyrax server by modifying dimensions/variables/attributes as described in the previous sections.

Here, we'll use a TRMM version 7 swath product as an example. If you try to open the OPeNDAP data URL in Figure 9 below and visualize the TRMM data through CF-enabled OPeNDAP HDF4 handler using NetCDF-Java tools like Panoply, you cannot visualize any variable from the remote TRMM version 7 swath data with the tools.

The tools fail on the above URL because some variables do not have valid CF attributes. For example, if you examine the TRMM's OPeNDAP DAS output carefully, you will notice that Swath_Longitude and Swath_Latitude coordinate variables have units attribute that has value degree. The CF conventions require values such as degrees_east and degrees_north. For another example, the data variable Swath_binDIDHmean doesn't have coordinates attribute which is another key CF attribute that visualization tools look for to associate a data variable with 2-D coordinate variables.

As we saw in the section 3 above, NcML can correct attributes easily by overwriting the existing value or adding new one. We can write a simple NcML file as shown in Figure 10 below and serve the TRMM 7 data through NcML handler that reprocesses the output from the HDF4 handler.

From the above example, Hyrax assumes that the location attribute refers to the full path (with respect to the BES data root directory) of a local dataset served by the same Hyrax server. The current version of the NcML module cannot be used to modify remote datasets.

Once you put the above NcML file under the data directory that the Hyrax serves, OPeNDAP users can visualize the TRMM version 7 data by opening the new NcML OPeNDAP URL. For example, we put the NcML file under the NASAFILES/ncml/ directory (Figure 11) and you can visualize it using Panoply (Figure 12).

In fact, NASA GES DISC already serves TRMM version 7 data via NcML handler. You can check test it here. NASA GES DISC also aggregates some MEaSUREs products using NcML via NcML handler. You can check their aggregation examples here.

If you want to know more about other NcML techniques, please read NcML Tutorial or Cookbook at Unidata.
Last modified: 06/02/2017