Common Data Fields

Common Data Fields#

This notebook will walk through how to access some of the more common data fields from the SRI hdf5 data files. This should be treated as a quick reference for common scientific applications and is NOT intended to be a comprehensive map of everything in these files. Refer to metadata contained within the files for descriptions of other arrays.

import h5py
import os
import urllib.request

Download files that will be used in this tutorial.

# Download the file that we need to run these examples
filename = 'data/20200207.001_lp_5min-fitcal.h5'

if not os.path.exists(filename):
    url='https://data.amisr.com/database/dbase_site_media/PFISR/Experiments/20200207.001/DataFiles/20200207.001_lp_5min-fitcal.h5'

    print('Downloading data file...')
    urllib.request.urlretrieve(url, filename)

    print('...Done!')

Use h5py to open the hdf5 data file.

h5 = h5py.File(filename, 'r')

Throughout this notebook, the expected dimensions of each data field will be listed using the following definitions:

  • Nrecords: Number of time records

  • Nbeams: Number of radar beams

  • Nranges: Number of radar range gates

Electron Density#

Units: m\(^{-3}\)

Dimensions: Nrecords x Nbeams x Nranges

Ne = h5['FittedParams/Ne'][:]
print(Ne.shape)

(188, 11, 74)

Ion Temperature#

The fourth index corresponds to the ion species (0 = O+). Refer to Fits Array for more information.

Units: K

Dimensions: Nrecords x Nbeams x Nranges

Ti = h5['FittedParams/Fits'][:,:,:,0,1]
print(Ti.shape)

(188, 11, 74)

Electron Temperature#

Refer to Fits Array for more information.

Units: K

Dimensions: Nrecords x Nbeams x Nranges

Te = h5['FittedParams/Fits'][:,:,:,-1,1]
print(Te.shape)

(188, 11, 74)

Line-of-Sight Velocity#

Refer to Fits Array for more information.

Units: m/s

Dimensions: Nrecords x Nbeams x Nranges

Vlos = h5['FittedParams/Fits'][:,:,:,0,3]
print(Vlos.shape)

(188, 11, 74)

Time#

The Time group lists timestamps in a variety of formats, but UnixTime (seconds from January 1, 1970) is often most convenient for programming purposes.

Dimensions: Nrecords x 2

utime = h5['Time/UnixTime'][:]
print(utime.shape)

(188, 2)

Note the two columns in this array correspond to the start time and end time of each integration period. If you need a 1D array of time stamps (common for most plotting and basic analysis), it is generally acceptable to take either just the first column (start times) or the midpoint of the interval.

# start times
utime_1d = utime[:,0]

# midpoints
import numpy as np
utime_1d = np.mean(utime, axis=1)

To convert these Unix timestamps to datetime objects, use either the native python datetime library or the numpy datetime functionality.

import datetime as dt
import numpy as np

# native python datetime library
time = [dt.datetime.utcfromtimestamp(ut) for ut in utime_1d]

# numpy datetime
time = utime_1d.astype('datetime64[s]')

Geographic Position#

Geodetic latitude, longitude, and altitude of the center of each radar bin.

Dimensions: Nbeams x Nranges

glat = h5['Geomag/Latitude'][:]
glon = h5['Geomag/Longitude'][:]
galt = h5['Geomag/Altitude'][:]

print(glat.shape, glon.shape, galt.shape)

(11, 74) (11, 74) (11, 74)

Geomagnetic Position#

Geomagnetic latitude and longitude of the center of each radar bin. If your application is dependent on highly accurate magnetic position relative to some external dataset, it is recommended that you start with the geographic position and calculate the conversion to magnetic coordinates yourself to avoid ambiguity with different magnetic coordinate systems and IGRF versions.

Dimensions: Nbeams x Nranges

mlat = h5['Geomag/MagneticLatitude'][:]
mlon = h5['Geomag/MagneticLongitude'][:]

print(mlat.shape, mlon.shape)

(11, 74) (11, 74)

Beam Positions#

The beam azimuths and elevations are contained in the BeamCodes array.

Dimensions: Nbeams

az = h5['BeamCodes'][:,1]
el = h5['BeamCodes'][:,2]

print(az.shape, el.shape)

(11,) (11,)

Site Info#

Various information about the radar site and its coordinates are contained within the Site group. These are all scalar parameters. Magnetic coordinates are also available.

site_name = h5['Site/Name'][()].decode('ascii')
site_lat = h5['Site/Latitude'][()]
site_lon = h5['Site/Longitude'][()]
site_alt = h5['Site/Altitude'][()]

print(site_name, site_lat, site_lon, site_alt)

Poker flat radar 65.12992 -147.47104 213.0

Fits Array#

The FittedParams/Fits array contains all fitted parameters aside from electron density (Ti, Te, Vlos), plus collision frequency and fractional composition. It is a five dimensional array indexed as follows:

Fits[record][beam][range][species][parameter]

The species index refers to either ion species or electrons. The ion species can be determined from the IonMass array. If you are interested in a specific species, find its mass in IonMass then use that index in the Fits array.

print(h5['FittedParams/IonMass'][:])

[16. 32. 30. 28. 14.]

These correspond to the following species:

Mass

Species

16

O\(^+\)

32

O\(_2^+\)

30

NO\(^+\)

28

N\(_2^+\)

14

N\(^+\)

Note that the fourth dimension of the Fits array is one element longer than the IonMass array.

fits = h5['FittedParams/Fits'][:]
print(fits.shape)

(188, 11, 74, 6, 4)

The last spot in this is dimension is always reserved for electrons, which can be reliably selected with the index -1. Different AMISR data files may have fit for a different number of species, however, O+ will always be first and electrons will always be last.

The last dimension of the Fits array gives the parameter, always in this order:

fraction, temperature, collision frequency, LOS velocity

The last dimension should be indexed by whatever parameter you want to pull out of the array.

# Close h5 object
h5.close()
Contents