|
|
|
This document assumes that users have a basic knowledge of the following data formats and the corresponding software packages:
In this document we describe how to use NCL[5] to access HDF files. We use NASA HDF files to show the procedures for accessing or visualizing the data with this program. For the general usages of these packages, please refer to their user's guides[6] or websites.
One
HDF4
file and two
HDF-EOS2
files were used to demonstrate
how to access or visualize HDF data with these packages. These files
are used to store data from the NASA Special Sensor Microwave/Imager
(SSMI) Sensor
[7]
and the Advanced Microwave Scanning Radiometer for the EOS (AMSR-E)
satellite system[8].
Table 1 shows the instrument names, formats and
other information about these HDF4/HDF-EOS2 files. We also converted
HDF-EOS2 files to netCDF-4 classic model-compliant and
netCDF-4-compliant HDF5 files. In this document, we mainly refer
to the instrument and the description of physical variables when
specifying a file or a field in a file. However, the file names may
be referred to in the code examples that appear throughout this
document.
| ||||||||||||||||||||
| Table 1 Sample files |
The first file stores the NASA SSMI ocean wind fields data.
In this document, we will use the two variables u10m and
v10m that represent the U component and the
V component of the wind field.
Both variables have three dimensions:
time (from January 2005 to December 2005), longitude, and
latitude.
The file name is atlas.ssmi.ver02.level3.5_5day.s950103.hdf.
Users can download this file from
here.
The second file (AE_RnGd) stores the NASA AMSR-E data that describes the monthly average rainfall accumulation over ocean and land in July 2007.
The format of the original file is
HDF-EOS2
. The original file name is
AMSR_E_L3_RainGrid_B05_200707.hdf.
Users can download this file from
here.
We changed the extension from hdf to he2
when we accessed this file via NCL.
We used the HDF4-to-HDF5 Conversion tool
[9] to convert the AMSR-E
HDF-EOS2 file to a netCDF-4-compliant HDF5 file. The
converted file was renamed to either
AMSR_E_L3_RainGrid_B05_200707.h5 or
AMSR_E_L3_RainGrid_B05_200707.nc,
as required by the software package processing it.
Users can download this netCDF-4-compliant HDF5 file from
here.
Since the netCDF-4 classic model[10] has more restrictions
than the general netCDF-4 model, two steps are needed to create a
netCDF-4 classic model-compliant HDF5 file.
The first step is to create a netCDF-4-compliant HDF5 file as addressed in the
previous section. The second step is to follow the procedure explained in
Appendix 5.1.
The file is called AMSR_E_L3_RainGrid_B05_200707_flatten.nc, and users can download
this file from
here.
The third file (AE_SI12) contains several fields that represent brightness temperatures, sea ice concentration, and snow depth over sea ice. This file is also from AMSR-E[11]. Unlike the other two files, the polar stereographic projection is used.
The original file is an
HDF-EOS2
file and the filename is
AMSR_E_L3_SeaIce12km_B02_20020619.hdf.
Users can download this file from
here.
To create the netCDF-4 classic model-compliant
HDF5
file,
one needs to follow the same procedure described in Section
3.2.3.
The converted file name is AMSR_E_L3_SeaIce12km_B02_20020619_flatten.nc.
Users can download this file from
here.
The NCAR Command Language (NCL) is an interpreted language
designed for scientific data analysis and visualization.
In this section, we explain how to install
NCL
, read
HDF4
and
HDF5
files, and visualize them.
Although
NCL
is free, registration is required to download it.
One can find information regarding registration, downloading, and
installation from
http://www.ncl.ucar.edu/Download/.
For our examples, we used NCL 5.1.1.
NCAR distributes precompiled NCL binaries for several widely used platforms
including AIX, IRIX, Linux, Mac OSX, Solaris, and cygwin.
We used their Linux distribution, and it worked without any problems.
Source code is also available, and building NCL from source code is documented at
http://www.ncl.ucar.edu/Download/build_from_src.shtml.
The following code shows a typical way to use NCL to visualize a variable in a file. This code needs to be typed at the prompt that NCL shows.
load loads an NCL module that defines
functions used in this NCL code.
An NCL code starts with the keyword begin and ends with the
keyword end. The addfile() function is used to open a file.
Two arguments need to be provided for this function. The
first argument provides the file name and the second argument
provides the file access mode, in this case, r, which
represents read-only mode. This function returns the file
descriptor, in this case, file1.
The file descriptor can be used to refer to a variable in the file. When referring to a variable, one can use NCL's subscript feature to select a specific portion of a variable. The explanation of subscripts used in NCL can be found in the "Subscript" section of NCL Language Reference Guide: Variables[12].
As shown in Figure 1, the entire data of data field var1,
denoted as var1(:,:), are represented as an NCL variable,
variable1. A subset of a data field var2, denoted as
var2(0,:,:), is represented as NCL variable, variable2. The
subsection of the first dimension of var2 is one element, the
first element of this dimension. The subsections of the
second and third dimensions include the whole sections of
these dimensions.
NCL
tries to read dimensions, units, and fill values from
variable attributes. When a variable does not have
attributes, users need to manually provide the fill value for
better visualization. To set the fill value -1 for
variable1,
for example, one can write a statement as
variable1@_FillValue = -1. When two-dimensional dimension
scales are associated with a variable, users need to write
additional statements such as variable1@lon2d = lon,
assuming that a variable, lon, contains longitude values.
To draw a plot, one needs to call gsn_open_wks() first to get
a workstation descriptor. A workstation is an instance of an
output device such as a screen or a file. The name of the
descriptor in this example is xwks1 as shown in Figure 1. One
of its arguments determines whether NCL draws a plot on the
screen, or creates a file such as PDF or PostScript.
Users can customize a plot by setting attributes of a
resource object created by assigning True. A resource object
in NCL means configuration settings of a plot such as vector
shapes, font sizes and colors of a plot. For example, the
statement resources1@tiMainFont = 21 specifies that the font
size of the main title is 21. For more information, refer to
http://www.ncl.ucar.edu/Document/Graphics/Resources/index.shtml.
With a workstation and a resource object, one can draw a plot
of variables by calling NCL APIs such as
gsn_csm_vector_map_ce() or gsn_csm_contour_map_ce(). In this
example, gsn_csm_vector_map_ce() is called. Since this
example generates a vector plot with varible1 as one
component of the vector and variable2 as another component of
the vector, both variable1 and variable2 should be passed as
parameters to gsn_csm_vector_map_ce(). For a contour plot,
only one variable should be passed to the function
gsn_csm_contour_map_ce().
The extension of the file name passed to addfile() function is important because NCL
detects the file format based on the extension. To open an HDF-EOS2 file whose
extension is .hdf, for example, one needs to rename the actual file name or
append .he2 to the argument.
When reading HDF-EOS2 fields, users need to be aware of the name mangling
that occurs in NCL. Suppose that RrLandRain is a data field defined under
a grid MonthlyRainTotal_GeoGrid; NCL flattens this structure and appends the
grid name to the field name. As a result, the variable name becomes
RrLandRain_MonthlyRainTotal_GeoGrid. To access this data field,
users need to refer to this mangled name, not the pure data
field name, RrLandRain.
In this section, we will explain how to visualize an HDF4 SDS (4.3.3.1), HDF-EOS2 (4.3.3.2, 4.3.3.4) and netCDF-4 classic model-compliant HDF5 (4.3.3.3, 4.3.3.3).
Figure 2 shows how to use NCL to read vectors from an HDF4 file and to draw a vector plot. We used the ocean wind fields data from the SSMI instrument.
u10m and v10m)
and draws a vector plot over a cylindrical equidistant map.
With some additional settings, the plot shown in Figure 3 can be generated. The unabridged code with the full resources variable setting is provided in Appendix 5.2.
Figure 4 shows code to read an HDF-EOS2 field and draw a contour plot.
Due to name mangling, this code uses the mangled name,
RrLandRain_MonthlyRainTotal_GeoGrid to access the data field
RrLandRain in the grid MonthlyRainTotal_GeoGrid. Since this
field does not have an attribute that specifies the fill value,
this code informs the fill value to draw a more meaningful plot.
gsn_csm_contour_map_ce() draws a contour plot over a cylindrical
equidistant map. With several additional resource settings to
specify NCL plot options, we could get the result shown in
Figure 5.
The full code is provided in Appendix 5.3.
If an HDF5 file is netCDF-4 classic model-compliant, one can use NCL to visualize data.
As we mentioned in Section 3.3, this file uses the north polar stereographic projection that requires two-dimensional longitude and latitude. This projection requires additional effort.
Note: As per Section 4.3.2 file
the file, AMSR_E_L3_SeaIce12km_B02_20020619.he2
is derived by simply renaming the downloaded
example data file.
The variable resources is defined as per the
unabridged code listing of Appendix
5.4.
SI_12km_NH_18V_DAY
is an
HDF-EOS2
data field
defined in this file.
GridLon_NpPolarGrid12km and
GridLat_NpPolarGrid12km don't exist in the file, but NCL
generates both of these two-dimensional geolocation fields,
which represent longitude and latitude. Since NCL does not
automatically associate SI_12km_NH_18V_DAY with these two
geolocation fields, users should specify the associations
with statements, starting with nh18vday@lat2d and nh18vday@lon2d.
NCL will recognize the lat2d and lon2d attributes and associate
the data variable with geolocation variables.
gsn_csm_contour_map_polar() draws a plot over a polar stereographic map,
as shown in Figure 8.
The unabridged code is explained in Appendix 5.4.
This section explains how to draw a plot when an HDF5 file has two-dimensional
longitude and latitude. To accomplish this, the file must have two additional
datasets representing longitude and latitude. In Figure 9,
lon and lat are variables containing real coordinate values.
lon and lat,
which exist in the HDF5 file. The data variable and the
two coordinate variables should have the same number of elements because
coordinate variables provide longitude and latitude for each element in the
data variable. Since we can provide arbitrary longitude and latitude values
with this method, we believe that NCL can draw a plot from all kinds of projections.
Although NCL does not support HDF5, it does support the netCDF-4 classic model. This subsection explains what the netCDF-4 classic model-compliant HDF5 file is and how to get it from an HDF5 file.
A netCDF-4 classic model-compliant HDF5 file strictly follows the netCDF-3 data model. It has three requirements. The first requirement is that each dimension of an HDF5 dataset associated with a dimension dataset and the dimension should be located under the same or ancestor groups of the HDF5 dataset it is associated with. For this reason, a netCDF-4 classic model-compliant HDF5 file is a netCDF-4-compliant HDF5 file. The second requirement is that all HDF5 datasets should be defined only under the HDF5 root group. The third one is that all HDF5 datatypes should be HDF5 atomic datatypes.
Most netCDF-4-compliant HDF5 files contain several groups, which causes them not to conform to the netCDF-4 classic model. Although it is tedious, it is not impossible to manually convert a netCDF-4-compliant HDF5 file with a non-classic model into one with a classic model.
One way to accomplish this is to use ncdump and ncgen, which are part of the netCDF-4 package. One can use ncdump to generate a text file from a netCDF-4 file, which is shown in Figure 10. With this text file, one can use any text editor to remove groups, as shown in Figure 11. One may also need to rename some variables if their names are used in multiple groups. Then, the modified text file can be used as input to ncgen, which generates a netCDF-4 file. The output of ncgen can be read by NCL.
This code is an unabridged version of Figure 2. It reads and visualizes ocean wind fields in an HDF4 SDS.
The following code is an unabridged version of Figure 4.
The following code is an unabridged version of Figure 7.
.