From brideout at haystack.mit.edu  Wed Feb  4 13:51:28 2004
From: brideout at haystack.mit.edu (William Rideout)
Date: Mon Oct 23 17:52:57 2006
Subject: [OpenMadrigal-developers] Calling Madrigal methods to support GPS
Message-ID: <40213F30.9050600@haystack.mit.edu>

John Foster has asked me to make some changes in mapping GPS data that can 
easily be done using methods that are available via Madrigal.  In particular, I 
need a Matlab interface into madCalculator, the web page which generalizes John 
Holt's looker page.  This method calculates any derivable Madrigal parameter for 
an array of latitudes, longitudes, and altitudes.

I plan to expose this functionality from Madrigal in the way I have done 
everything recently:

1.  As a command line method (python)
2.  As a web service (aka a cgi script that calls the above script)
3.  As Matlab methods that run either locally or via the web, using 1 or 2 above.

Shunrong had commented early that he would like a simplier method for a single 
point in space and time, and one for only a point in time for parameters like Kp 
that don't vary spatially.  I'll do this, but they'll just call this one 
script/web service under the covers.

Bill

-- 
Bill Rideout
MIT Haystack Observatory
Email: brideout@haystack.mit.edu
Phone: 781 981-5624


From brideout at haystack.mit.edu  Tue Feb 10 09:34:53 2004
From: brideout at haystack.mit.edu (William Rideout)
Date: Mon Oct 23 17:52:58 2006
Subject: [OpenMadrigal-developers] Plans for the MadrigalWeb python module
Message-ID: <4028EC0D.3090507@haystack.mit.edu>

I am starting to write an entirely new Python module, called MadrigalWeb.  The 
goal of this module will be to access Madrigal data entirely via the web using 
the newly-added web services.  These web service are the major feature of the 
soon-to-be released next version of Madrigal (2.3).  This module will not 
require any other software to be installed. In particular, this module will not 
require the Madrigal server to be installed on the local machine or on any 
machine that the client can access except via the web.  It will be designed and 
tested to work both on Unix and Windows machines.  The code will be very similar 
to the Madrigal Matlab *Web methods (except with far fewer lines of code).  It 
will be written in 100% python (no C extensions needed).

The first application that will use the MadrigalWeb module will be 
globalIsprint, an application that reproduces the functionality of the global 
search web pages.  Below is the documentation for that application:

Usage:

globalIsprint --url=<Madrigal url> --parms=<Madrigal parms>
               --output=<output file> [options]

where:

--url=<Madrigal url> - url to homepage of site to be searched
                           (ie, http://www.haystack.mit.edu/madrigal/)
                           This is required.

--parms=<Madrigal parms> - a comma delimited string listing the desired Madrigal
                            parameters in mnemonic form.  (Example:
                            gdalt,dte,te).  Data will be returned
                            in the same order as given in this string.

--output=<output file name> - the file name to store the resulting data.


and options are:


--startDate=<MM/DD/YYY> - start date to filter experiments before.  Defaults to
                           allow all experiments.

--endDate=<MM/DD/YYY> - end date to filter experiments after.  Defaults to allow
                         all experiments.

--inst=<instrument list> - comma separated list of instrument codes or names.
                            See Madrigal documentation for this list.  Defaults
                            to allow all instruments. If names are given, the
                            argument must be enclosed in double quotes.

                            Examples::

                                --inst=10,30

                                --inst="Jicamarca IS Radar,Arecibo IS Radar"

--kindat=<kind of data list> - comma separated list of kind of data codes.  See
                                Madrigal documentation for this list.  Defaults
                                to allow all kinds of data.  If names are given,
                                the  argument must be enclosed in double quotes.

                                Examples::

                                     --kindat=3001,13201

                                     --kindat="INSCAL Basic Derived
                                       Parameters,(efwind) Derived Parameters:
                                       E+F regions: winds/electric fields/ion
                                       drifts"

--filter=<filter>=<[mnemonic] or [mnemonic1,[+-*/]mnemonic2]>,<lower
                     limit1>,<upper limit1>[or<lower limit2>,<upper limit2>...]

                     a filter using any measured or derived Madrigal parameter,
                     or two Madrigal parameters either added, subtracted,
                     multiplied or divided.  Each filter has one or more allowed
                     ranges.  The filter accepts data that is in any allowed
                     range.  If the Madrigal parameter value is missing, the
                     filter will always reject that data.  Multiple filter
                     arguments are allowed on the command line.  To skip either a
                     lower limit or an upper limit, leave it blank.  Examples::


                         filter=ti,500,1000  (Accept when 500 <= Ti <= 1000)

                         filter=gdalt,-,sdwht,0,  (Accept when gdalt >
                            shadowheight - that is, point in direct sunlight)

                         filter=gdalt,200,300,1000,1200 (Accept when 200 <= gdalt
                           <= 300 OR 1000 <= gdalt <= 1200)

--seasonalStartDate=<MM/DD> - seasonal start date to filter experiments before.
                               Use this to select only part of the year to
                               collect data.  Defaults to Jan 1.  Example:
                               --seasonalStartDate=07/01 would only allow
                               experiments after July 1st from each year.


--seasonalEndDate=<MM/DD> - seasonal end date to filter experiments after.  Use
                             this to select only part of the year to collect
                             data.  Defaults to Dec 31.  Example:
                             --seasonalEndDate=10/31 would only allow
                             experiments before Oct 31 of each year.

--showFiles - if given, show file names.  Default is to not show file names.

--showSummary - if given, summarize all arguments at the beginning.  Default is
                 to not show summary.

Let me know if you have any comments.

Bill

-- 
Bill Rideout
MIT Haystack Observatory
Email: brideout@haystack.mit.edu
Phone: 781 981-5624


From shunrong at haystack.mit.edu  Tue Feb 10 11:59:50 2004
From: shunrong at haystack.mit.edu (Shunrong Zhang)
Date: Mon Oct 23 17:52:58 2006
Subject: [OpenMadrigal-developers] Plans for the MadrigalWeb python module
In-Reply-To: <4028EC0D.3090507@haystack.mit.edu>
Message-ID: <Pine.GSO.4.21.0402101111110.10443-100000@hyperion>

Hi Bill,

Minor comments about the the options for globalIsprint:

> 
> Usage:
> 
> globalIsprint --url=<Madrigal url> --parms=<Madrigal parms>
>                --output=<output file> [options]
> 
> where:

> 
> --inst=<instrument list> - comma separated list of instrument codes or names.
>                             See Madrigal documentation for this list.  Defaults
>                             to allow all instruments. If names are given, the
>                             argument must be enclosed in double quotes.
> 
>                             Examples::
> 
>                                 --inst=10,30
> 
>                                 --inst="Jicamarca IS Radar,Arecibo IS Radar"

Can we not use the full name of the instruments, but something like, 
                                  --inst="Jica,Arecibo"
that returns data for instrument name containing "Jica" and "Arecibo" ?


> 
> --kindat=<kind of data list> - comma separated list of kind of data codes.  See
>                                 Madrigal documentation for this list.  Defaults
>                                 to allow all kinds of data.  If names are given,
>                                 the  argument must be enclosed in double quotes.
> 
>                                 Examples::
> 
>                                      --kindat=3001,13201
> 
>                                      --kindat="INSCAL Basic Derived
>                                        Parameters,(efwind) Derived Parameters:
>                                        E+F regions: winds/electric fields/ion
>                                        drifts"

Similar to --inst, can we use
                                     --kindat="INSCAL"
to search everthing with "INSCAL"? In the Sondrestrom case where there are
31 kindat's, this will be great if we use --kindat="FITTED" to represent
those 10 kindat's containing "FITTED".


> 
>                          filter=gdalt,-,sdwht,0,  (Accept when gdalt >
>                             shadowheight - that is, point in direct sunlight)
> 

I don't quite understand filter=gdalt,-,sdwht,0. I thought I could use
filter=gdalt,sdwht,- to define the low limit as sdwht and leave the upper
limit open to accept data with gdalt>sdwht ...



>                          filter=gdalt,200,300,1000,1200 (Accept when 200 <= gdalt
>                            <= 300 OR 1000 <= gdalt <= 1200)
I like this very much ...


> 
> --showFiles - if given, show file names.  Default is to not show file names.
  and not to leave a blank line between two data files ?



> 
> --showSummary - if given, summarize all arguments at the beginning. Default is
>                  to not show summary.
but always to show the mnemonics ?


Other than the above, I still like the isprint option,

 badval=<bad value string> (defaults to "missing")

I know we can use a filter to get rid of data with "missing",
but sometimes we don't want to reject data for other parameters even
though this particular one is missing, or a filter isn't a good
choice. For example, when By and Bz (GSMBY, GSMBZ) data are missing, we
still want to list Nel, Te, etc. Then it'll be nice to simply set the
"missing" to -9999.0.

This is all I can think of so far.

Shunrong



From brideout at haystack.mit.edu  Tue Feb 10 14:24:52 2004
From: brideout at haystack.mit.edu (William Rideout)
Date: Mon Oct 23 17:52:58 2006
Subject: [OpenMadrigal-developers] Plans for the MadrigalWeb python module
References: <Pine.GSO.4.21.0402101111110.10443-100000@hyperion>
Message-ID: <40293004.6040901@haystack.mit.edu>

Shunrong Zhang wrote:

> Minor comments about the the options for globalIsprint:

> 
> Can we not use the full name of the instruments, but something like, 
>                                   --inst="Jica,Arecibo"
> that returns data for instrument name containing "Jica" and "Arecibo" ?


I'll allow the use of * in this argument to mean the same thing it does in the 
Unix shell - and also for kindat names.


> 
>>--showFiles - if given, show file names.  Default is to not show file names.
> 
>   and not to leave a blank line between two data files ?
> 

That's correct.


> 
>>--showSummary - if given, summarize all arguments at the beginning. Default is
>>                 to not show summary.
> 
> but always to show the mnemonics ?

When --showSummary not given, the report will still have the header line listing 
the mnemonics requested.  So the default case will be what you want - only a 
single header line at top, and all other lines data without blank lines.

> 
> 
> Other than the above, I still like the isprint option,
> 
>  badval=<bad value string> (defaults to "missing")
> 

Whoops, I meant to add these three arguments:

--missing=<missing string> (defaults to "missing")

--assumed=<assumed string> (defaults to "assumed")

--knownbad=<knownbad string> (defaults to "knownbad")

Thanks for the comments.


Bill



-- 
Bill Rideout
MIT Haystack Observatory
Email: brideout@haystack.mit.edu
Phone: 781 981-5624


From brideout at haystack.mit.edu  Tue Feb 24 14:00:14 2004
From: brideout at haystack.mit.edu (William Rideout)
Date: Mon Oct 23 17:52:58 2006
Subject: [OpenMadrigal-developers] Adding Tsyganenko model to Madrigal
Message-ID: <403B9F3E.9040507@haystack.mit.edu>

John F,

I'm almost done incorporating the Tsyganenko model into Madrigal.  As always 
when incorporating Fortran, the biggest challenge is to get the code to handle 
error conditions correctly.

The Tsyganenko model basically draws the field line given various conditions. 
Because of your request for adding this feature to mapping GPS mapping, I have 
added the following four parameters to Madrigal:

1 & 2:

given a point in space and time, find the point where the corresponding field 
line crosses the GSM XY plane (that is, the magnetic dipole equatorial plane). 
The two returned values are:

TSYG_EQ_XGSM
TSYG_EQ_YGSM

3 & 4:

given a point in space and time, find the point where the corresponding field 
line crosses the GSE XY plane (that is, the ecliptic plane).  The two returned 
values are:

TSYG_EQ_XGSE
TSYG_EQ_YGSE

My question is - should I expose anything else?  I haven't exposed the apex 
coordinates, although TSYG_EQ_XGSM and TSYG_EQ_YGSM are close.  And if so, 
should they be returned as GSM or GSE?  Also, do I need to convert anything back 
to Geodetic coordinates?  I can also return footprints, conjugate points, etc.

Finally, a separate method in Tsyganenko's library does the following:

FOR ANY POINT OF SPACE WITH COORDINATES (XGSM,YGSM,ZGSM) AND SPECIFIED 
CONDITIONS IN THE INCOMING SOLAR WIND, THIS SUBROUTINE:

(1) DETERMINES IF THE POINT (XGSM,YGSM,ZGSM) LIES INSIDE OR OUTSIDE THE
       MODEL MAGNETOPAUSE OF SHUE ET AL. (JGR-A, V.103, P. 17691, 1998).

(2) CALCULATES THE GSM POSITION OF A POINT {XMGNP,YMGNP,ZMGNP}, LYING AT THE 
MODEL MAGNETOPAUSE AND ASYMPTOTICALLY TENDING TO THE NEAREST BOUNDARY POINT WITH
  RESPECT TO THE OBSERVATION POINT {XGSM,YGSM,ZGSM}, AS IT APPROACHES THE 
MAGNETO-PAUSE.

Let me know if anyone thinks this would be a helpful parameter for Madrigal to 
expose.

Bill

-- 
Bill Rideout
MIT Haystack Observatory
Email: brideout@haystack.mit.edu
Phone: 781 981-5624


From jcf at haystack.mit.edu  Wed Feb 25 09:55:06 2004
From: jcf at haystack.mit.edu (John Foster)
Date: Mon Oct 23 17:52:58 2006
Subject: [OpenMadrigal-developers] Re: Adding Tsyganenko model to Madrigal
Message-ID: <200402251455.i1PEt6p21490@hyperion.haystack.edu>

Bill,

What you've done sounds very useful. I imagine that the TSYG_EQ_XGSM
TSYG_EQ_YGSM points where the field line crosses the GSM XY plane will 
be the most useful for our immediate needs.

A useful back-conversion wuld be to go from a EQ_XGSM, EQ_YGSM position
(inside the Tsyganenko magnetopause) back to geographic lat and long (at
a specified altitude, e.g. 400 km) for a given point in time. Then, we could
take the IMAGE fetures and project them onto our ionospheric TEC
viewing plane.

The last-mentioned features of the Tsyganenko model, don't sound too
useful for our work.

Keep up the good work!

John F


From brideout at haystack.mit.edu  Wed Feb 25 12:55:50 2004
From: brideout at haystack.mit.edu (William Rideout)
Date: Mon Oct 23 17:52:58 2006
Subject: [OpenMadrigal-developers] Re: Adding Tsyganenko model to Madrigal
References: <200402251455.i1PEt6p21490@hyperion.haystack.edu>
Message-ID: <403CE1A6.2000509@haystack.mit.edu>

John,

Here's my 3 suggested matlab methods I'll add to madmatlab.  These are general 
methods to map in either direction from geodetic into the GSM XY plane for a 
given time using the Tsyganenko model.

The general idea is to call the getGsmMap method once for every time of 
interest, which traces field lines from a grid around the globe (and so takes 
about 8 minutes to run).  You then call mapGeodeticToGsm or mapGsmToGeodetic 
using the returned matrix as an input parameter for each point of interest. 
These two methods are very fast.

Please let me know if they meet your needs.

I'll also create a script to make a jpeg image of TEC data in the GSM plane from 
either hemisphere using these methods, but that script is GPS specific and so I 
won't add it to the Madrigal Matlab API.

The three methods are:

1.

[gsmMap] = getGsmMap(madrigalUrl, alt, year, month, day, hour, min, sec)

     This method will create a mapping matrix from geodetic parameters at a 
given alt into the GSM XY plane.  The resultant gsmMap matrix is then used as an 
argument in the next two methods that go from geodetic to GSM, and GSM to 
geodetic.  (This method traces 5400 field lines and takes about 8 minutes to run).

     Inputs:

         madrigalUrl: home page of madrigal site to use.  Until installed on
         hyperion and released, this must be "http://grail/madrigal"

         alt: the reference altitude (for GPS data, use 450.0)

         year, month, day, hour, min, sec - specify time

2.

[xgsm, ygsm] = mapGeodeticToGsm(gsmMap, lat, long)

     Returns xgsm and ygsm given the gsmMap calculated above, and a lat and long

     If mapping fails, returns [NaN, NaN]

3.

[lat, long] = mapGsmToGeodetic(gsmMap, xgsm, ygsm, hemisphereFlag)

     Returns lat and long given the gsmMap calculated above, xgsm, ygsm,
and a hemisphereFlag (1=northern, 2=southern).  The hemisphere flag is required
since one GSM point can map into up to two geodetic points.  The hemisphere 
referred to is the magnetic one,so for example its still possible to get 
negitive latitudes when hemisphereFlag == 1.

     If mapping fails, returns [NaN, NaN]


Bill



John Foster wrote:
> Bill,
> 
> What you've done sounds very useful. I imagine that the TSYG_EQ_XGSM
> TSYG_EQ_YGSM points where the field line crosses the GSM XY plane will 
> be the most useful for our immediate needs.
> 
> A useful back-conversion wuld be to go from a EQ_XGSM, EQ_YGSM position
> (inside the Tsyganenko magnetopause) back to geographic lat and long (at
> a specified altitude, e.g. 400 km) for a given point in time. Then, we could
> take the IMAGE fetures and project them onto our ionospheric TEC
> viewing plane.
> 
> The last-mentioned features of the Tsyganenko model, don't sound too
> useful for our work.
> 
> Keep up the good work!
> 
> John F
> 
> _______________________________________________
> OpenMadrigal-developers mailing list
> OpenMadrigal-developers@openmadrigal.org
> http://www.openmadrigal.org/mailman/listinfo/openmadrigal-developers
> 
> 


-- 
Bill Rideout
MIT Haystack Observatory
Email: brideout@haystack.mit.edu
Phone: 781 981-5624