From brideout at haystack.mit.edu Wed Jul 7 17:10:52 2004 From: brideout at haystack.mit.edu (William Rideout) Date: Mon Oct 23 17:53:03 2006 Subject: [OpenMadrigal-developers] Header and Catalog records for new Millstone ISR data Message-ID: <40EC66DC.2030004@haystack.mit.edu> Before we send any of our recent data to the Cedar database, we need to add header and catalog records to our data. As Larisa pointed out at the integrated data workshop at Cedar, this metadata can be very important to users, so I'd like to make an effort to get as much information in as possible. Previously these headers were generated by filing in templates, with (I assume) the ability to add experiment specific comments. Attached below is an example header and catalog template for a standard experiment. Please let me know if any parts of the template are out-of-date. Also, please let me know if an different template is appropriate for other types of data. Since our experiments tend to run in certain modes (extra wide coverage, local 10 position, etc), I'd like to get concise descriptions of these modes and incorporate these descriptions. Also important to header and catalog records are descriptions of the non-standard parameters measured. The Millstone-specific parameters in the new files that I can't find described in the old headers are: 3351 Millstone Hill radar operatn par 52 3352 Millstone Hill radar operatn par 53 3353 Millstone Hill radar operatn par 54 3321 Ti, Tr correlation coefficient 3322 Ti, Ph correlation coefficient 3323 Ti, Co correlation coefficient 3324 Tr, Ph correlation coefficient So if someone could write a one paragraph description of these, I'd appreciate it. *****************CATALOG template******************** KRECC 2001 Catalogue Record, Version 1 KINSTE 30 Millstone Hill - MISA Steerable/ Zenith Fixed Antennas MODEXP 0 Millstone Hill Incoherent Scatter Radar Data CMODEXP More details are available from: http://www.haystack.edu/ C TIMCY 5 minutes ALT1 93 km. Lowest altitude measured ALT2 1171 km. Highest altitude measured GGLAT1 42 degrees. Lowest geographic latitude measured GGLAT2 43 degrees. Highest geographic latitude measured GGLON1 289 degrees. Westmost geographic longitude measured GGLON2 289 degrees. Eastmost geographic longitude measured PL1 40 Shortest radar pulse length PL2 2000 Longest radar pulse length IBYRE 2000 Beginning year IBDTE 105 Beginning month and day IBHME 1315 Beginning UT hour and minute IBCSE 1000 Beginning centisecond IEYRE 2000 Ending year IEDTE 105 Ending month and day IEHME 2041 Ending UT hour and minute IECSE 1600 Ending centisecond C CPURP Incoherent scatter observations of the ionosphere C CIREM See: http://www.haystack.edu/ CSREM See: http://www.haystack.edu/ C CPI J. Holt CPREPDAT Wed Apr 24 14:36:43 EDT 2002 *****************HEADER template******************** KRECH 3002 Header Record, Version 2 KINST 3 30 Millstone Hill - MISA Steerable / Zenith Fixed Antennas KINDAT 4 3408 Basic Parameter Set via INSCAL Version 8.1 C CKINDAT STANDARD MILLSTONE HILL PROCESSING METHOD USING INSCAL. CKINDAT CKINDAT INSCAL analyzes incoherent scatter autocorrelation functions (acfs) CKINDAT to determine ionospheric plasma parameters. CKINDAT CKINDAT The acfs are formed from the measured lag-products using a trapezoidal CKINDAT summation rule. A multidimensional non-linear least squares fit to each CKINDAT acf is then performed to compute estimates of the plasma parameters. CKINDAT Parameter error bars are computed by assuming that chi square is 1.0. CKINDAT Analysis parameters for this experiment are summarized below. More CKINDAT details, including the actual INSCAL input parameters, output listing CKINDAT and error messages are avaialable from http://www.haystack.edu. CKINDAT CKINDAT 1. The search parameters for 90-130 [km] were: CKINDAT CKINDAT Ion Temperature, ACF Normalization Factor, Collision Frequency CKINDAT and Ion Drift Velocity CKINDAT CKINDAT Assumed or model parameters for 90-130 [km]: CKINDAT CKINDAT Temperature Ratio = 1.0 CKINDAT CKINDAT Density (Before temperature correction) = CKINDAT Calfac(i) * [S/N](i) * Systmp * Range(i)**2 / Xmtr_power CKINDAT where Calfac = radar calibration factor CKINDAT S/N = signal to noise ratio CKINDAT Systmp = system temperature (K) CKINDAT Range = range (km) CKINDAT Xmtr_power = transmitter peak power (MW) CKINDAT i = range index CKINDAT CKINDAT n(H+)/Ne = 0.0 CKINDAT CKINDAT n(mass 31)/Ne: if Altitude < 120 km then = 1.0 CKINDAT else = 1. - 2./(1. + SQRT(1.+8.*EXP(ZZ2))) CKINDAT where: CKINDAT ZZ2 = MIN(-(Altitude-180.)/H, 50.) CKINDAT H = 10. - 6.*EXP(ZZ1) CKINDAT ZZ1 = MIN(-(Altitude-120.)/40., 50.) CKINDAT CKINDAT The measurements were not sensitive to the H+ Drift velocity CKINDAT CKINDAT 2. The search parameters for 130-400 [km] were: CKINDAT CKINDAT Ion Temperature, ACF Normalization Factor, Temperature Ratio, CKINDAT and Ion Drift Velocity CKINDAT CKINDAT Assumed or model parameters for 130-400 [km]: CKINDAT CKINDAT Collision Frequency = 0.0 CKINDAT CKINDAT Density (Before temperature correction) = CKINDAT Calfac(i) * [S/N](i) * Systmp * Range(i)**2 / Xmtr_power CKINDAT where Calfac = radar calibration factor CKINDAT S/N = signal to noise ratio CKINDAT Systmp = system temperature (K) CKINDAT Range = range (km) CKINDAT Xmtr_power = transmitter peak power (MW) CKINDAT i = range index CKINDAT CKINDAT n(H+)/Ne = 0.0 CKINDAT n(mass 31)/Ne: if Altitude < 120 km then = 1.0 CKINDAT else = 1. - 2./(1. + SQRT(1.+8.*EXP(ZZ2))) CKINDAT where: CKINDAT ZZ2 = MIN(-(Altitude-180.)/H, 50.) CKINDAT H = 10. - 6.*EXP(ZZ1) CKINDAT ZZ1 = MIN(-(Altitude-120.)/40., 50.) CKINDAT CKINDAT The measurements were not sensitive to the H+ Drift velocity CKINDAT CKINDAT The search parameters for 400-1200 [km] were: CKINDAT CKINDAT Ion Temperature, Temperature Ratio, n(H+)/Ne, ACF Normalization CKINDAT Factor, and Ion Drift Velocity CKINDAT CKINDAT Assumed or model parameters 400-1200 [km]: CKINDAT CKINDAT Collision Frequency = 0.0 CKINDAT CKINDAT Density (Before temperature correction) = CKINDAT Calfac(i) * [S/N](i) * Systmp * Range(i)**2 / Xmtr_power CKINDAT where Calfac = radar calibration factor CKINDAT S/N = signal to noise ratio CKINDAT Systmp = system temperature (K) CKINDAT Range = range (km) CKINDAT Xmtr_power = transmitter peak power (MW) CKINDAT i = range index CKINDAT CKINDAT n(mass 31)/Ne = 1. - 2./(1. + SQRT(1.+8.*EXP(ZZ2))) CKINDAT where: CKINDAT ZZ2 = MIN(-(Altitude-180.)/H, 50.) CKINDAT H = 10. - 6.*EXP(ZZ1) CKINDAT ZZ1 = MIN(-(Altitude-120.)/40., 50.) CKINDAT CKINDAT The measurements were not sensitive to the H+ Drift velocity CKINDAT CKINDAT Chirp correction: CKINDAT CKINDAT A chirp correction has been applied to the line of sight velocities to CKINDAT compensate for a frequency offset produced in the UHF transmitter. CKINDAT This is calculated from the applied dc voltage and current in the CKINDAT transmitter klystron which are measured continuously, and from the CKINDAT known pulse length, interpulse period, and pulse rise and fall times. CKINDAT For a given pulse length this chirp correction normally varies little CKINDAT over the course of an experiment, and in practice a single chirp CKINDAT correction is usually applied for each pulse length for the whole CKINDAT experiment. Typical values of the chirp correction vary from 10-20 CKINDAT m/s. CKINDAT CKINDAT Density Calibration: CKINDAT CKINDAT The densities have been calibrated using the Umass Lowell Digisonde. CKINDAT Millstone Hill Incoherent Scatter electron densities are calculated by CKINDAT inserting into the radar equation a calibration factor relating radar CKINDAT signal temperature to electron density. This calibration constant is CKINDAT determined by direct comparison of high elevation measurements of CKINDAT signal temperature from the F-region peak with local digisonde CKINDAT measurements of peak electron density. C CHIST See http://www.haystack.edu/ CHIST A complete history of the analysis of this experiment is maintained CHIST at the above Web site. In some cases, an updated analysis of this CHIST experiment may be found there. C IBYRT 2000 Beginning year IBDTT 105 Beginning month and day IBHMT 1315 Beginning UT hour and minute IBCST 1000 Beginning centisecond IEYRT 2000 Ending year IEDTT 105 Ending month and day IEHMT 2041 Ending UT hour and minute IECST 1600 Ending centisecond LPROL 13 16 Length of prologue in data records JPAR 14 11 Number of single-valued parameters MPAR 15 24 Number of multiple-values parameters NROW 16 17 Number of entries for multiple valued parameter C 1D Parameters: KODS(1) 17 132 Beginning azimuth (0=geog N,90=east) 1.0e-02 deg KODS(2) 18 133 Ending azimuth (0=geog N,90=east) 1.0e-02 deg KODS(3) 19 142 Beginning elevation angle 1.0e-02 deg KODS(4) 20 143 Ending elevation angle 1.0e-02 deg KODS(5) 21 402 Pulse length 1.0e-06 sec KODS(6) 22 482 System temperature 1.0e+00 K KODS(7) 23 483 Additional increment to system temp 1.0e-04 K KODS(8) 24 486 Peak power 1.0e+00 kW KODS(9) 25 490 Transmitted frequency 1.0e+05 Hz KODS(10) 26 3318 D.P. Power Normalization constant 1.0e-03 N/A KODS(11) 27 3319 Additional increment to D.P. Power NrmK 1.0e-07 N/A C 2D Parameters: KODM(1) 39 -3350 Error in Line of sight Doppler Vlos (po 1.0e+00 m/s KODM(2) 40 -3313 Error in ACF Normalization Factor 1.0e-03 N/A KODM(3) 41 -710 Error in Ion-neutral collision frequenc 1.0e+00 s-1 KODM(4) 42 -690 Error in Comp - (ions with mol wt 28 to 1.0e-03 N/A KODM(5) 43 -660 Error in Composition - [H+]/Ne 1.0e-03 N/A KODM(6) 44 -580 Error in Line of sight ion velocity (po 1.0e+00 m/s KODM(7) 45 -570 Error in Temperature ratio (Te/Ti) 1.0e-03 N/A KODM(8) 46 -550 Error in Ion temperature (Ti) 1.0e+00 K KODM(9) 47 -505 Error in Log10(uncorrected electron den 1.0e-03 lg(m-3) KODM(10) 48 120 Range 1.0e+00 km KODM(11) 49 121 Additional increment to range 1.0e-01 m KODM(12) 50 411 Signal to noise ratio 1.0e-03 N/A KODM(13) 51 420 Reduced-chi square of fit 1.0e-03 N/A KODM(14) 52 430 Goodness of fit 1.0e+00 N/A KODM(15) 53 461 Millstone Hill data quality code 1 1.0e+00 N/A KODM(16) 54 505 Log10(uncorrected electron density) 1.0e-03 lg(m-3) KODM(17) 55 550 Ion temperature (Ti) 1.0e+00 K KODM(18) 56 570 Temperature ratio (Te/Ti) 1.0e-03 N/A KODM(19) 57 580 Line of sight ion velocity (pos = away) 1.0e+00 m/s KODM(20) 58 660 Composition - [H+]/Ne 1.0e-03 N/A KODM(21) 59 690 Comp - (ions with mol wt 28 to 32)/Ne 1.0e-03 N/A KODM(22) 60 710 Ion-neutral collision frequency 1.0e+00 s-1 KODM(23) 61 3313 ACF Normalization Factor 1.0e-03 N/A KODM(24) 62 3350 Line of sight Doppler Vlos (pos = away) 1.0e+00 m/s C C 420 Reduced-chi square of fit C If all statistical assumptions are correct, the expectation value of C chi-square is one. In fact, it tends to be much smaller when the C signal-to-noise ratio is large. This is probably due to the large C correlations between the lag products in this case, which are not C taken into account in the fit. C C 430 Goodness of fit C This is 1000 times the root mean square deviation of the fit from C the the measured autocorrelation function (ACF). Since the ACF is C normalized to 1.0, values of ~1000 indicate ~100% deviations, values C of ~100 indicate ~10% deviations and values ~10 indicate ~1% C deviations. C C 461 Millstone Hill data quality code 1 C The data quality parameter is generated by an algorithm which C attempts to detect the presence of either a satellite echo or C radio frequency interference (RFI), resulting in a 0 for clean C spectra and a 1 for contaminated spectra. The algorithm's C thresholds are deliberately set to avoid false detections on C genuine incoherent scatter signals, and therefore data C contaminated by weaker satellite or RFI signals will not always C be flagged. In particular, the algorithm will miss C satellites/RFI at altitudes with significant heavy ion (mass > C O+) fractions, or for altitudes with temperatures < 300 K. This C parameter should not be used as the sole data quality flag. C C -505 Uncertainty in Log10(uncorrected electron density) C This is computed from the statistical uncertainty of the fit ACF at C zero lag. In conformity with the CEDAR standard, it is the logarithm C of the uncertainty, not the uncertainty of the logarithm. If the fit C fails, the density itself is still stored in the data record, and C the uncertainty is missing. This statistical uncertainty is C normally much smaller than the larger uncertainty in the density C calibration, which is ~20%. C C 3350 Line of sight Doppler Vlos (pos = away) C This is a separate non-linear least squares calculation of the C Doppler velocity, which, unlike parameter 580, does not assume an C incoherent scatter spectrum. This may be particularly useful in C coherent echo studies. C CAREM Analysis included a correction for possible spectral asymmetry CANALYST J. M. Holt CANDATE Mon Jan 10 10:50:52 2000 Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Tue Jul 13 11:22:38 2004 From: brideout at haystack.mit.edu (William Rideout) Date: Mon Oct 23 17:53:03 2006 Subject: [OpenMadrigal-developers] New version of globalSearchDelayedGeo.py Message-ID: <40F3FE3E.9040605@haystack.mit.edu> Shunrong, The new version of the script globalSearchDelayedGeo.py is available at /opt/madrigal/bin. To make the script faster, I've implemented a new web service, madTimeCalculator. It's similar to the web service madCalculator, which calculates parameters over a gird of lat, long, and alt; madTimeCalculator calculates values over a range of times. Since it does not specify any point in space, it will only work for location-independent parameters such as Kp or Bz. In this way I can access geoparms for a large number of times at once. The command line is exactly the same as before - but much faster. Let me know what you think. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Thu Jul 15 09:33:36 2004 From: brideout at haystack.mit.edu (William Rideout) Date: Mon Oct 23 17:53:03 2006 Subject: [OpenMadrigal-developers] Issues with Madrigal Message-ID: <40F687B0.2@haystack.mit.edu> John, Frank mentioned to me that you and he had some problems using Madrigal. In particular, Frank said that you were looking to access some geophysical data, so you selected "Access Data" -> "Database Inventory Form". You then selected "Geophysical Indicies" and modified the date range to select some limited amount of data. You then got a no data returned message. (Let me know if I'm off here). Madrigal is geared around experiments, and this page is called "Madrigal Experiment Selector". We organize geophysical parameters into one experiment that started in 1950 and continues up until the last time we updated the data. By choosing a limited date range, you effectively eliminated that experiment. You would have seen the experiment if you had left the default dates. However, I think what you did was something other reasonable users might do, and so I changed the logic of that search. Now the search returns a list of experiments that overlap the selected time range, rather than are included in the selected date range. That is, it now returns a superset of what it used to return. This is also a reaction to Larisa's point that we should try to avoid "No data found" messages as much as possible, and help users to avoid them. So please try again your approach, and let me know if you think this is reasonable solution, or if you have a better suggestion. Some other points about geophysical parameters: If you select any other instrument in the Madrigal database, and choose isprint as the way to view the data, you'll see that geophysical parameters, magnetic field, IMF, MSIS, etc parameters are always listed as though they were recorded in the data. This is done through Madrigal's derivation engine. If you want parameter data independently of any experiment in the Madrigal database, go to "Run Models" -> "Calculate any Madrigal Parameter". Please let me know if you have any other problems with Madrigal, because feedback from first-time users is extremely valuable. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From lpg at haystack.mit.edu Thu Jul 15 12:20:52 2004 From: lpg at haystack.mit.edu (Larisa Goncharenko) Date: Mon Oct 23 17:53:03 2006 Subject: [OpenMadrigal-developers] Re: Issues with Madrigal Message-ID: <200407151620.i6FGKqt02265@hyperion.haystack.edu> Bill, I like a lot your idea of returning experiments that overlap selected time range. Good move! I expect a lot of users have troubles similar to those you just described. I just tried to check how it works now, but I had trouble reaching 2004 KP data. The search returns files with geophysical parameters ending on 04/30/2004, but when I proceed with the file, the end date is shown as 03/31/2003, and I don't get any info for selected 2004 dates. Could you check where inconsistency in dates come from and where is 2004 KP data? Thank a lot, Larisa From brideout at haystack.mit.edu Fri Jul 16 09:12:00 2004 From: brideout at haystack.mit.edu (William Rideout) Date: Mon Oct 23 17:53:03 2006 Subject: [OpenMadrigal-developers] Adding method getIonosphericTerminator to madmatlab Message-ID: <40F7D420.8070803@haystack.mit.edu> As per a request by John Foster, I'm planning to add a new method to the madmatlab library called getIonosphericTerminator. It's format will be: [term] = getIonosphericTerminator(madrigalUrl, time, alt, gridSize) where madrigalUrl = any Madrigal site with version 2.3 or greater, time = Matlab time, alt = altitude in km at which to find the terminator, gridSize = size of grid in degrees to define the terminator term is a n by 2 array of lat, longitude pairs defining the terminator. Since this method uses the standard web service madCalculator, it will work on any Madrigal 2.3 site or greater. Algorithm: Madrigal will calculate shadowHeight for each point on a global grid at the given time. Each latitude will be scanned, and any point where the change in shadowheight crosses the given alt will be added to term array. Let me know if you have any comments. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624 From brideout at haystack.mit.edu Mon Jul 19 09:59:29 2004 From: brideout at haystack.mit.edu (William Rideout) Date: Mon Oct 23 17:53:03 2006 Subject: [OpenMadrigal-developers] Dropping support for Matlab 5 Message-ID: <40FBD3C1.8080409@haystack.mit.edu> I'm planning to drop support for Matlab 5 in the Madmatlab client API (this is the API that allows you to access Madrigal data using Matlab from anywhere on the internet). The basic problem is that Matlab 5 did not support accessing data over the internet; so I've been using a third-party module called pnet. However, this module has two serious problems: 1. Eiscat has had a problem using this module through its firewall. 2. It fails when downloading large amounts of data (actually, it just downloads some of the data, and returns without generating an error). Problem 2 can be avoided by using the urlread method that was introduced in Matlab 6. I'm hoping that by switching to this standand method, the Eiscat firewall problem will also disappear (hopefully I can get someone there to try it in a few days when I put a new version on-line). Please let me know you need Matlab 5 support; however, problem 2 can lead to corrupt results, so I really want to make the switch. Bill -- Bill Rideout MIT Haystack Observatory Email: brideout@haystack.mit.edu Phone: 781 981-5624