How Madrigal data is organized

In the Madrigal database, the data are organized by experiment. An experiment consists of data from a single instrument, and, as a rule, is meant to address a particular scientific goal. Most often an experiment will correspond to a particular set of operating modes run for a contiguous interval of time. For example, the data collected by an incoherent scatter radar during a world day would constitute an experiment. More complicated situations exist, and judgment may have to be exercised in determining what constitutes an experiment. For example, an experiment might be interrupted temporarily in order to use a different set of operating modes in support of a satellite overpass. In this case there would be two experiments which overlap in time.

To each experiment there corresponds a directory. These directories are of the form madroot/experiments/year/instrument/start_date[expt], where year is the four-digit year, instrument is the 3-letter mnemonic for the instrument set in the instTab.txt file, start_date is the start date of the experiment in the form ddmmmyy and expt is an optional character to distinguish different experiments with the same start date. For example, /opt/madrigal/experiments/1997/son/06jan97 contains Sondrestrom data for an experiment beginning on 6 January, 1997, where year =1997, instrument =son, start_date =06jan97, and there is no expt because there is only one Sondrestrom experiment starting on this date.

Each experiment directory must contain:

The Experiment Table entry (expTab.txt) for this experiment. This file is created automatically if you use the tools described in the creating experiments section.
The Data Availability Table entry (dataTab.txt) for this experiment. (This metadata is deprecated as discussed in the metadata section). This file is created automatically if you use the tools described in the creating experiments section.

Each experiment directory may contain:

The metadata file fileTab.txt for this experiment. This table must have one entry for each Madrigal format file in the experiment. This file is created automatically if you use the tools described in the creating experiments section.
One or more datasets in Madrigal (or any valid Cedar) format.
Subdirectories containing an html file named index.html . Links to these files will show up the experiment page.
Html pages in the main directory, and again links to these files will show up the experiment page.
Plots or other files related to individual records in the dataset. Links to these files appear next to the appropriate record in the summarizeCedarFile.cgi page. These files must be located in the subdirectory "plots/[file name]/records" under the experiment directory. In order for the script to determine which file goes with which record, the files must include a five digit number somewhere in its name. For example, plot00027.gif would appear as a link next to record 27 in summarizeCedarFile.cgi. More than one file can be associated with a given record.

An essential role for a Madrigal administrator is to run the script madroot/bin/updateMaster on a regular basis. This script performs a number of functions:

It gathers all the separate metadata files in the individual experiment directories into the central directory madroot/metadata.
It gathers metadata files from other Madrigal sites into the central directory madroot/metadata.
It updates geophysical files from the OpenMadrigal site.

It is recommended that this script be set as a cron job to run once a day.

A complete description of the various ways to add and modify experiments in a Madrigal database is given in the next section. The updateMaster script must be run after each change described in that section for the changes to take effect.

Note that an experiment need not contain any Madrigal files.