From brideout at haystack.mit.edu Tue Jan 6 15:53:48 2004 From: brideout at haystack.mit.edu (William Rideout) Date: Mon Oct 23 17:52:55 2006 Subject: [OpenMadrigal-developers] Update of Madrigal on hyperion with new Matlab API Message-ID: <3FFB205C.6020906@haystack.mit.edu> I'm planning to upgrade Hyperion with the newest version of Madrigal Wednesday afternoon. This means that our version of Madrigal will be off line for about 1-2 hours. Please let me know if this time is not good. The new feature that will be installed is the complete Madrigal Matlab API. This new feature is really two new features: 1. All Madrigal data, from metadata to individual file data, is now available as web services. This means that Madrigal can now be accessed by any language (Matlab, IDL, Tcl, etc) that can get data from the web. 2. A Matlab API has been written to use these web services. This Matlab API can run entirely over the internet, so you can write a Matlab program that does everything our present web interface does, except it can run on any machine that Matlab will run on (including Windows machines). See the Madrigal documentation page (under API's/Matlab) for a full description of each method, or type "help madmatlab" when running Matlab on hyperion. Note that Matlab is not required on the Madrigal web server to support these new features. From the web server's point of view, I simply added some cgi scripts written in python. Matlab is only required to access these scripts via the Matlab API (feature 2 above). The web services are not implemented according to the SOAP or XMLRPC standard since not all scripting languages have support for these standards (or for XML parsing). Instead they use the simple approach of returning data requested via a query as a delimited text file. These web services are fully documented as part of the Python API. This would allow another scripting language, such as IDL, to access these web services. The Matlab api methods in general simply convert their arguments into a query url, and then parse the returned text file into a Matlab array of structures. To test the Matlab API, there is a test program testMadmatlab in source/madmatlab. It simply asks for the main url of the site you wish to test, and then calls the various Matlab methods using the standard test files. If that Madrigal site does not yet have this release of Madrigal, an error will be returned. John Holt is planning to write a Matlab application using this Matlab API. When he does, I plan to release that as a Matlab application, so that any user with Matlab on a Windows or Unix machine can install and run it without needing to install a Madrigal server. Please let me know if you have any comments. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From shunrong at haystack.mit.edu Tue Jan 6 16:23:05 2004 From: shunrong at haystack.mit.edu (Shunrong Zhang) Date: Mon Oct 23 17:52:56 2006 Subject: [OpenMadrigal-developers] new global search report (fwd) Message-ID: Bill, When you update Hyperion with the newest version of Madrigal, could you consider also update the output format issue for the global search that I mentioned sometime ago (see below)? Thanks! Shunrong ---------- Forwarded message ---------- Date: Thu, 23 Oct 2003 15:36:40 -0400 (EDT) From: Shunrong Zhang To: William Rideout Cc: John Holt Subject: new global search report Hi Bill, I've checked the new global search and shown the new report format to John. The 'assmued' characters issue, the UTH bug are nicely solved. The html scripts and the info between two experiments are removed. These are great for serious analyses. As the report will be used by Fortran programs, and maybe by matlab as well for data presentation, we tend to like a simpler format, although we can live with this current format that will add some comlexity for Fortran (and matlab) coding. It'd be better to remove the blank line between two experiment data blocks. The info before the parameter mnemonics table isn't necessary (In fact, it repeats what has been shown in the email informing the availability of the report) and makes difficult for automatic reading by Fortran without a common line of statements (characters) marking the start point of data. That may be the case for matlab too. So it would be nice for the report file to have only one line for mnemonics (the first line), and others lines are simply data without blank line in between. Thanks! Shunrong From brideout at haystack.mit.edu Wed Jan 7 09:17:16 2004 From: brideout at haystack.mit.edu (William Rideout) Date: Mon Oct 23 17:52:56 2006 Subject: [OpenMadrigal-developers] new global search report (fwd) References: Message-ID: <3FFC14EC.2070808@haystack.mit.edu> Shunrong and I discussed his needs for global searching, and we decided that what he needs is a global search application. The present global search web interface is designed to be interactive, with the assumption that its not done often enough to require automation. Shunrong needs an automated approach to global searching, so there's no need for any web interface. So my plan is to write a new application called globalIsprint. It will have a command line very similar to isprint's, except that instead of specifying a single file, there will be extra arguments to limit instruments, kindats, seasons, and Madrigal sites to search. Unlike the present global search, which returns quickly and the user later receives an email when things are done, this application will be synchronous (since it is assumed the user is running it in batch mode). As requested by Shunrong, one of the report formats will be pure data, without any blank lines or metadata. Implementation: It seems natural to use the new web services described below to implement this application, since we want it to work seamlessly across all Madrigal sites. No new web services are required - the application would simply call isprintService for every file that meets the selection criteria. Even when searching a local Madrigal site, there is not a large performance difference when using isprintService via http and using isprint directly. I'll write the application in python, not Matlab, since there is really not much in the Matlab API except kludgy parsing that can be done in python in 4 lines or less. This should be a pure python application, which means it should run anywhere python is installed (Windows or Unix). Please let me know your thoughts. If this idea seems okay, I'll send out a complete description of the command line arguments. Bill Shunrong Zhang wrote: > Bill, > > When you update Hyperion with the newest version of Madrigal, could you > consider also update the output format issue for the global search that I > mentioned sometime ago (see below)? Thanks! > > Shunrong > > ---------- Forwarded message ---------- > Date: Thu, 23 Oct 2003 15:36:40 -0400 (EDT) > From: Shunrong Zhang > To: William Rideout > Cc: John Holt > Subject: new global search report > > > Hi Bill, > > I've checked the new global search and shown the new report format to > John. > > The 'assmued' characters issue, the UTH bug are nicely solved. The html > scripts and the info between two experiments are removed. These are great > for serious analyses. > > As the report will be used by Fortran programs, and maybe by matlab as > well for data presentation, we tend to like a simpler format, although we > can live with this current format that will add some comlexity for Fortran > (and matlab) coding. It'd be better to remove the blank line between two > experiment data blocks. The info before the parameter mnemonics table > isn't necessary (In fact, it repeats what has been shown in the email > informing the availability of the report) and makes difficult for > automatic reading by Fortran without a common line of statements > (characters) marking the start point of data. That may be the case for > matlab too. > > So it would be nice for the report file to have only one line for > mnemonics (the first line), and others lines are simply data without blank > line in between. > > Thanks! > Shunrong > > > > > _______________________________________________ > OpenMadrigal-developers mailing list > OpenMadrigal-developers@openmadrigal.org > http://www.openmadrigal.org/mailman/listinfo/openmadrigal-developers > > -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Thu Jan 15 11:17:57 2004 From: brideout at haystack.mit.edu (William Rideout) Date: Mon Oct 23 17:52:57 2006 Subject: [OpenMadrigal-developers] Madrigal patch for SRI References: <3FE2E856.68B01A19@sri.com> <3FE35FDE.9040502@haystack.mit.edu> <3FE2F2CA.1C2484D3@sri.com> <3FE36631.4010108@haystack.mit.edu> <4002CDBB.E331912B@sri.com> <400423BF.3020807@haystack.mit.edu> <40052400.3818A378@sri.com> Message-ID: <4006BD35.9020309@haystack.mit.edu> Angela, As promised, here's the patch that should make your global search function work correctly. Here's how to install it: 1. Copy the attached file SRI_patch.tar to $MADROOT 2. cd $MADROOT 3. tar -xf SRI_patch.tar 4. make -f Makefile.gnu madrec 5. make -f Makefile.gnu binary 6. cd $MADROOT/source/madpy 7. /usr/bin/python setup.py install This should make the global search work correctly. Please email me when its done so I can check. Technical discussion: There were actually two problems that I corrected: 1. The python method os.spawnlp uses the $PATH variable to execute a process even when its passed a full path to the executable (which I didn't realize). So even though I was passing it "/usr/bin/python", it was running a different version of python when it was kicking off a background job to produce the global search report. This lead to the "madrigal.metadata not found" error. I changed os.spawnlp to os.spawnl, which will not try to use the $PATH variable, but instead insists on a full path to the executable. This was changed in the file report.py, and is the reason you ran /usr/bin/python setup.py install in step 7. 2. Cycle marks: We assumed cycle marks (Cedar parameter 95) in a file always would go in sequence, and I used to abort if I found an out of order cycle mark. SRI clearly uses this parameter differently, so I changed the C engine to simply ignore out of order cycle marks. This is why you rebuilt and re-installed the C madrec library in steps 4 and 5. Both these changes have been checked into CVS and will be part of any future release, so they are not SRI specific. Bill Angela Li wrote: > [Wed Jan 14 19:07:58 2004] [error] [client 192.52.65.66] Problem found > with cycle numbers: Success, referer: > http://transport.sri.com/madrigal/cgi-bin/GetEmailFromUser?callingpage=madSearch& > -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 -------------- next part -------------- A non-text attachment was scrubbed... Name: SRI_patch.tar Type: application/x-tar Size: 286720 bytes Desc: not available Url : http://www.haystack.mit.edu/pipermail/openmadrigal-developers/attachments/20040115/c95a3489/SRI_patch.tar