Administering OpenMadrigal

This document is meant as a manual for administering the OpenMadrigal web site. It contains information on both the developer's site www.openmadrigal.org and the archive site cedar.openmadrigal.org.

The archive site cedar.openmadrigal.org

The cedar.openmadrigal.org site serves as a central archive of all Madrigal sites. It also serves as a home for instrument data not maintained by local Madrigal sites. It is a standard Madrigal site, except that it uses a modified version of the homepage index.html. The modified version of index may be found in CVS under MillstoneCVS/Madrigal/OpenMadrigal/CEDAR_site.

All data is downloaded from remote sites using the script import_madrigal.py. This script may be found in CVS under MillstoneCVS/Madrigal/OpenMadrigal/scripts. It should be set up as a cron job to download remote Madrigal data on a regular basis, according to agreements with remote Madrigal site administrators. All data downloaded is logged in /opt/cedar/metadata/userdata/<site>_import_<year>.txt. See Madrigal crontab section below for a description of all cron jobs running on Madrigal.

Administrators can access the log files particular to their instruments using the script http://cedar.openmadrigal.org/cgi-bin/getLogAdmin.py. As the CEDAR Madrigal database administrator, there are two tasks to support this. The first is the creation of passwords. Passwords can be created using the script /opt/cedar/bin/setAdminPassword.py with usage setAdminPassword.py <username> <password>. This script can also be used to modify existing passwords. This script is located in CVS under Millstone/Madrigal/OpenMadrigal/scripts. The username is either the site name as set in $MADROOT/metadata/siteTab.txt, or set in $MADROOT/metadata/hostSite.txt. The second task is to maintain the file hostSite.txt. This is a metadata file unique to CEDAR Madrigal, and defines virtual sites for PI's who host their data directly on CEDAR Madrigal. It is kept in CVS under Millstone/Madrigal/OpenMadrigal/CEDAR_site. The columns of this file are:

<fake site id>, <fake site name>, <kinst 1> [,<kinst 2> ...]

Example:

10001,MillstoneHighResFP,5360

Whenever a new PI adds a new instrument to be hosted at CEDAR Madrigal, this file must be updated, and a new username/password pair created using setAdminPassword.py. PI's with multiple instruments can have more than one kinst code (comma separated) on the line. The user name is the <fake site name>, so either create it without spaces, or use quotes when running setAdminPassword.py. Send the PI the site name and password after you have tested it.

Local Madrigal administrators are free to modify the kind of data (kindat) descriptions of their data at any time by modifying the local file $MADROOT/metadata/typeTab.txt. The Cedar archive site needs a central repository of these kindat descriptions. To avoid problems when multiple Madrigal sites use the same kindat codes, Madrigal now allows kindat descriptions to be indexed by a combination of instrument code and kindat code. With Madrigal 3.0, the typeTab.txt file will be able to have entries of the form <kinst>_<kindat>,<description>, in addition to the present format of <kindat>,<description>. This feature is already installed on the Cedar archive Madrigal server. There is also a cron job checkCedarTypeTab.py that checks all remote Madrigal sites for any updates to their typeTab.txt file, and imports then into the Cedar archive site's typeTab.txt file. This script can be found in CVS under Millstone/Madrigal/OpenMadrigal/scripts.

The script changeInstAccess.py can be useful in administering this site. It allows you to change the status of all experiments with a given instrument kinst with one command. This script may be found in CVS under MillstoneCVS/Madrigal/OpenMadrigal/scripts.

The developer's site www.openmadrigal.org

The OpenMadrigal developer's web site serves a number of purposes:

  1. It distributes the latest version of Madrigal metadata and geophysical data to all the individual sites when they run updateMaster.
  2. It serves as a central web site to inform about the OpenMadrigal project, and to distribute the latest version of the code.

This manual also covers the building and testing of a new Madrigal release. All files relating to the OpenMadrigal.org website (including this document) are located in CVS under Millstone/Madrigal/OpenMadrigal.

Distributing Madrigal metadata and geophysical data

Two cron jobs are required to collect these data, createExpUpdate.py and createGeoUpdate.py.

createExpUpdate.py - located in CVS under Millstone/Madrigal/OpenMadrigal/scripts. Usage: createExpUpdate.py [-e emailAddress] [-e emailAddress2] ... where any number of email addresses can be notified of the results. This script collects the metadata files expTab.txt and fileTab.txt from each of the Madrigal sites listed in siteTab.txt, and stores them in the directory metadata/[site_name]_[site_id]. The site_name and site_id are also set by siteTab.txt. This cron job should be set to run at least once a day, and the results should be emailed to the OpenMadrigal administrator. The results are accessed via the web from the url http://www.haystack.mit.edu/madrigal/distributionFiles/metadata.

createGeoUpdate.py - located in CVS under Millstone/Madrigal/OpenMadrigal/scripts. Usage: createGeoUpdate.py [-e emailAddress] [-e emailAddress2] ... where any number of email addresses can be notified of the results.. This script will update the geofil.tar.Z and ig_rz.dat file on OpenMadrigal. Also updates the file status.dat which indicates the last update date of each type of geophysical data. All the files updated are accessed via the url http://www.haystack.mit.edu/madrigal/distributionFiles. Due to relatively infrequent updates to the base files, this cron job is now run only once a week.

Three core metadata files are also distributed via updateMaster:

The OpenMadrigal administrator is responsible for manually updating these three files on http://www.haystack.mit.edu/madrigal/distributionFiles/metadata whenever they are changed. You can set up up the script checkMetadataDist.py located in CVS under Millstone/Madrigal/OpenMadrigal/scripts as a cron job to check that these files are in sync.

The script checkMetadata.py can be set up as a cron job to monitor whether all Madrigal sites have these three up-to-date metadata files. The sites need to be running Madrigal 2.5 or greater in order to have automatic update of these files. This script is located in Subversion under OpenMadrigalSVN/trunk/madroot/source/madpy/scripts/bin.

Finally, this capability requires that the cgi script compareToArchive.py be installed on the Millstone server at url http://madrigal.haystack.mit.edu/cgi-bin/madrigal/compareToArchive.py. This script can be found in Subversion under OpenMadrigalSVN/madroot/scripts/madpy/scripts/cgi.

Cron jobs running on Madrigal

The latest version of the crontab.txt file used by midasop on the Madrigal machine at Haystack (which includes both the CEDAR and the Millstone Madrigal installations) can be found in CVS under Millstone/Madrigal/OpenMadrigal. All scripts in this crontab are either in CVS under Millstone/Madrigal, or in Subversion in the OpenMadrigal repository. This script includes both administrative jobs (such as updateMaster, import_madrigal.py, etc) and jobs that automatically load data into the CEDAR Madrigal site from instrument PI's who do not run their own local Madrigal sites.

Building a new release of Madrigal

Updating Madrigal derivation models

The Madrigal derivation engine contains a number of models that are under active development, and so should be updated if possible with each new Madrigal release. These models are listed below.

All Fortran code that uses reals is converted to use all doubles using nag_apt tools.

Building a release

From Subversion at Millstone, checkout OpenMadrigalSVN/trunk/madroot. The file source/configure.ac has the line AC_INIT([Madrigal] that should be updated with the Madrigal version. Since autotools is used in building a distribution, you want to be sure that your machine has up-to-date autotools. For Madrigal 2.5, I used autoconf 2.6.3, automake 1.10.2, libtool 2.2.4, and m4 1.4.12. To create the needed autotool files, cd to source and run:

  1. aclocal
  2. autoconf
  3. libtoolize
  4. autoheader
  5. automake --add-missing

To make the distribution file, cd to madroot, and run "make -f Makefile.gnu". The file MANIFEST in the madroot directory controls which files are included in the tar file. The general test procedure is to build the distribution, and then test install on any available unix machines, both as updates and as new installs. There are also regression tests to be run on the core C and Fortran libraries, and on the Python library.

Setting a release number

The release number is set in source/configure.ac and in two places in madContents.html (title and top H1 line).

Regression tests

A regression test for the C and Fortrans libraries is runMadrecDiagnostics, found in Subversion under OpenMadrigalSVN/trunk/madroot/source/madc/madrec. This regression test can also be run under purify. To do so build the Fortran library geolib with a debug flag. Then build the madrec library using the Makefile.purify file in the madrec directory.

A second regression test for the geolib Fortran library is to run madf/geolib/testGeolib. It will produce a file testGeolib.out, which should diff with testGeolib.out.rock.

The python regression tests are in OpenMadrigal/madroot/source/madpy/scripts/regressionTests. Each of the following three regression tests are run as $MADROOT/bin/python script:

Rebuilding Madrigal documentation

Rebuilding the C library documentation
See Subversion file OpenMadrigalSVN/trunk/madroot/source/madc/madrec/CREATE_DOC. If new C files have been added, this will need to be modified.
Rebuilding the Fortran geolib library documentation
In installed version of Madrigal, cd to source/madf/geolib. Copy makeGeolibProcs from Subversion OpenMadrigalSVN/trunk/ madroot/source/madf/geolib to this directory. Run <$MADROOT/bin/madtclsh makeGeolibProcs>. This will produce a file geolibProcs.html which can be pasted into geolib section of doc/dev_fortran.html. Note that makeGeolibProcs is not a robust parser, and with Madrigal 2.6 it imported code in addition to documentation, and so this step was skipped.
Rebuilding the Python Madrigal/MadrigalWeb server documentation

Make sure pdoc is installed in $MADROOT/bin/python.

Rebuilding the Python Web services documentation
Rebuilding the IDL remote API documentation
In OpenMadrigal/madroot/source/madidl, run "python create_madidl_doc.py". Then use that file to update in doc the files rt_idl.html and rr_idl.html.
Rebuilding the Matlab remote API documentation
In OpenMadrigal/madroot/source/madmatlab, run "python create_madmatlab_doc.py". Then use that file to update README and in doc the files rt_matlab.html and rr_matlab.html.
Rebuilding python source code examples in documentation.
Run pygmentize. You will need to copy in the style code in the head, and the body code.
Rebuilding the pdf documentation

To rebuilding all the Madrigal documentation into a single pdf file, cd $MADROOT/doc and run

htmldoc --batch madDocBook.book

If you have added or removed any documentation files, or changed their order, you will need to use the htmldoc UI to make those changes. Also, in order to import the graphics into the generated pdf, you will need to make a temporary copy of the $MADROOT/icons directory in the $MADROOT/doc directory.


Revised: Oct 2, 2015