Version 170 (modified by mark1, 11 years ago) (diff) |
---|
Arrival of new flight data
This procedure should be followed on receipt of new flight data from ARSF.
If in any doubt about something (e.g. a dataset has two project codes), contact Gary.
Copy the data onto the system
After successfully copying the data onto the system, email arsf-processing to confirm the data transferred ok (but hasn't yet been checked!).
Unpacking projects
proj_tidy.sh
Run proj_tidy.sh:
proj_tidy.sh -p <path_to_project>
This will do three things:
- Run a series of checks which you should go through and fix if necessary
- Fix a few things automatically (permissions/common spelling mistakes)
- Provide a list of commands (if any) that will convert the project from the old style structure to the current standard. These should be checked then pasted into the terminal.
Manual steps
- Check time stamps of files vs julian day to check we use the correct julian day
- Prune empty directories (check these aren't stubs for later data first)
- Rename any remaining spelling mistakes in directories or files. Check eagle and hawk raws have VNIR/SWIR prefix.
- Check all data belongs to this flight and move out/delete if belongs to a different flight day.
Verification
unpack_file_check.py
knpa: This will need to be rewritten to work with the new file structure (or merged with proj_tidy)
Manual checks
- Look at the logsheet and verify that we have copies of all relevant data mentioned there.
- In some cases, the flight crew may fly two projects back-to-back but enter all the data onto a single logsheet. If so, you may need to split the project directory into two, particularly if there's a large time gap (navigation needs separate processing) or the PIs are different (different delivery addresses/tracking). If you do need to split a project, ensure both copies have copies of common files (logsheets, rinex, etc), but that non-common files are not duplicated (ie. don't include hawk data for part 1 in part 2..). Also note in the ticket what was done for tracking purposes.
- Check the filesizes of all data files (Eagle, Hawk, ATM, CASI) to make sure none are zero bytes (or obviously broken in some way).
- Verify the details on the logsheet (esp. PI) by calling/emailing ARSF-Ops (probably Gary) - the application form and logsheet are not reliable enough, nor do they track any changes in PI over the lifetime of the application. Get the alpha grading as well.
Move to permanent PML data storage
Move to appropriate location in the repository (~arsf/arsf_data/2011/flight_data/...)
- ensure the project directory names conform to the standard - PROJECTCODE-YYYY_JJJxx_SITENAME, e.g. GB07_07-2007_102a_Inverclyde, boresight-2007_198, etc
Tickets and tracking
Status Page
Add details to the processing status page. Under 'Data location' use the full path to the project (e.g /users/rsg/arsf/... ).
Ticket
Raise a new trac ticket (type 'flight processing') for the new data.
- Ticket summary should be of the form EU10/03, flight day 172/2011, Dennys Wood
- Add short version of scientific purpose to guide processing (check ARSF application in ~arsf/arsf_data/2009/ARSF_Applications)
- Note arrival time of data
- Set priority of ticket from project grading (try the grades subpages on Projects or hassle ARSF-Ops)
- Note any specific comments that might help with processing
- Owner should be blank
- Verify the sensors that were requested in the application (primary) and those that weren't (secondary). Note in the ticket which these were.
- Ticket body should contain:
Data location: ~arsf/arsf_data/2011/flight_data/..... FILL IN Data arrived from ARSF via SATA disk LETTER OR network transfer on DATE. Scientific objective: FILL IN FROM APPLICATION (just enough to guide processing choices) Priority: FILL IN FROM APPLICATION/WIKI PAGE (e.g. alpha-5 low), set ticket priority appropriately PI: A. N. Other EUFAR Project ID: Any other notes.. = Sensors: = * Eagle (requested) * Hawk (requested) * Leica LIDAR (not requested but flown anyway) * RCD (requested)
Create KML overview
When happy the directory is unpacked correctly run make_kmloverview.py to generate the password for the email below.
Vectors
If the site is in the UK, check the vector page to see if we have vectors for the site in question. If not then email ARSF-Ops (Gary) and ask for them, specifying the corner points of the area covered, in OS BNG grid coordinates. Use bounding_box.py (needs to be ran in the project directory) or use the generated nextmap DEM (below) to get the range of tiles, then do the conversion. It can take a couple of weeks before we get the vectors.
E-mail PI
Email the PI to inform them that their data has arrived for processing. Sample text:
- fill in the 7 fields: <PI_NAME>, <PROJECT>, <EUFAR ID> <TICKET_NO>, <DATE_OF_ARRIVAL>, <USERNAME>, <PASSWORD>, <INSERTWEBLINKHERE>
- the date of arrival should be when the disks arrived or when the download begun
- the username and password are available in the .htaccess file in processing/kml_overview or ~arsf/usr/share/kmlpasswords.csv. Note that make_kmloverview.py will need to have been run to create a password first unless the project code already exists for previous data.
- cc to arsf-processing
- also cc to neodc@rl.ac.uk
- set reply-to to arsf-processing
- subject: ARSF data arrival notification (<PROJECT> [<EUFAR ID>])
Dear <PI_NAME>, This is a notification that your ARSF data for <PROJECT> [<EUFAR ID>], flown on <CALENDAR_DAY(S)>, are at the ARSF Data Analysis Node for processing (data received from ARSF Operations on <DATE_OF_ARRIVAL>). We aim to deliver as quickly as possible - our current processing priority order can be found at http://arsf-dan.nerc.ac.uk/status/order/order.php, though please note that this queue is subject to change. You can follow progress at the following webpages: http://arsf-dan.nerc.ac.uk/status/progress/progress.php - general status page http://arsf-dan.nerc.ac.uk/trac/ticket/<TICKETNO> - our notes during processing (may be technical) Also available are KML (Google Earth) files that will allow you to view flight data coverage. These files are updated as the data are processed and more information becomes available. Click the following link and enter username and password details as below: <INSERTWEBLINKHERE> username: <USERNAME> password: <PASSWORD> Further information on KML previews: http://arsf-dan.nerc.ac.uk/trac/wiki/ProcessingAtDAN/KMLoverview If you would like any more information, please feel free to contact us at arsf-processing@pml.ac.uk Regards,
Basestation
Check there is basestation data for the flight. If not, ask ops if they have some or if we need to download it.
Create DEM
Run the DEM generating script if a UK flight. nextmapdem.sh
If the flight is outside the UK you'll need to process the LiDAR data first or use the ASTER/SRTM dem (see http://arsf-dan.nerc.ac.uk/trac/wiki/Processing/SRTMDEMs)