Data Files

This chapter discusses the files used by MSG to store data on disk. These files adopt the HDF5 data format, a platform-neutral binary storage format with advanced features such as transparent decompression.

File Types

There are five types of MSG data files, distinguished by their differing content:

  • specint files store spectroscopic intensity data for a single combination of photospheric parameters.

  • photint files store photometric intensity data for a single combination of photospheric parameters.

  • specgrid files store spectroscopic intensity data over a grid of photospheric parameters.

  • photgrid files store photometric intensity data over a grid of photospheric parameters.

  • passband files store passband response functions, used to convert spectroscopic intensities into corresponding photometric intensities.

For a detailed description of each file type, refer to the Data Schema chapter.

Obtaining Data

MSG ships with a limited set of data files in the $MSG_DIR/data subdirectory, sufficient to enable the walkthroughs:

  • grids/sg-demo.h5 is a specgrid file based on a solar-metallicity Kurucz atmosphere grid (ap00k2odfnew), with intensity spectra synthesized using SYNSPEC (Hubeny & Lanz, 2017).

  • passbands/pb-Generic-Johnson.*-Vega.h5 are passband files for the U, B and V filters of the Johnson photometric system.

Additional files can be downloaded separately from the grid files and passband files web pages.

Importing Data

To import an existing grid of spectra into the MSG ecosystem, first convert the individual spectra into corresponding specint files. MSG provides a number of tools to assist with this conversion; see the Grid Tools appendix for further details.

The next step is to create a manifest (named, say, manifest.txt) listing all the specint files composing the grid. This is a simple text file with each line naming one file; for instance:

specint-0001.h5
specint-0002.h5
specint-0003.h5

Then, run the specint_to_specgrid tool to create a specgrid file:

$ $MSG_DIR/bin/specint_to_specgrid manifest.txt specgrid.h5

To build a photgrid file from the data in a specgrid file, run the specgrid_to_photgrid tool:

$ $MSG_DIR/bin/specgrid_to_photgrid specgrid.h5 passband.h5 photgrid.h5

…where passband.h5 is the name of the passband file to use.

Note

It’s not always necessary to create photgrid files, because MSG supports loading a specgrid file and then convolving with a passband on-the-fly. This approach is used in the examples presented in the Walkthroughs chapter. It has the advantage of convenience, but a disadvantage in the form of a performance penalty, as discussed in the Performance Tuning chapter.