= Arrival of new flight data = This procedure should be followed on receipt of new flight data from ARSF. If in any doubt about something (e.g. a dataset has two project codes), contact Gary. == Copy the data onto the system == * [wiki:Procedures/NewDataArrival/Disk Data by SATA disk or DVD] * [wiki:Procedures/NewDataArrival/Network Data by network transfer] After successfully copying the data onto the system, email arsf-processing to confirm the data transferred ok (but hasn't yet been checked!). == Unpacking projects == === proj_tidy.sh === Run proj_tidy.sh: {{{ proj_tidy.sh -p }}} This will do three things: - Run a series of checks which you should go through and fix if necessary[[BR]] - Fix a few things automatically (permissions/common spelling mistakes)[[BR]] - Provide a list of commands (if any) that will convert the project from the old style structure to the current standard. These should be checked then pasted into the terminal. === Manual steps === * Prune empty directories (check these aren't stubs for later data first) * Rename any remaining spelling mistakes in directories or files. Check eagle and hawk raws have VNIR/SWIR prefix. * Check all data belongs to this flight and move out/delete if belongs to a different flight day. * Remove any spaces, brackets or other Unix-upsetting characters in filenames * use `find -regex '.*[^-0-9a-zA-Z/._].*' | ~arsf/usr/bin/fix_naughty_chars.py` * gives suggested commands, but check before pasting commands! == Verification == === unpack_file_check.py === knpa: This will need to be rewritten to work with the new file structure (or merged with proj_tidy) === Manual checks === * Look at the logsheet and verify that we have copies of all relevant data mentioned there. * In some cases, the flight crew may fly two projects back-to-back but enter all the data onto a single logsheet. If so, you may need to split the project directory into two, particularly if there's a large time gap (navigation needs separate processing) or the PIs are different (different delivery addresses/tracking). If you do need to split a project, ensure both copies have copies of common files (logsheets, rinex, etc), but that non-common files are not duplicated (ie. don't include hawk data for part 1 in part 2..). Also note in the ticket what was done for tracking purposes. * Check the filesizes of all data files (Eagle, Hawk, ATM, CASI) to make sure none are zero bytes (or obviously broken in some way). * Verify the details on the logsheet (esp. PI) by calling/emailing ARSF-Ops (probably Gary) - the application form and logsheet are not reliable enough, nor do they track any changes in PI over the lifetime of the application. Get the alpha grading as well. == Move to permanent PML data storage == Move to appropriate location in the repository (~arsf/arsf_data/2011/flight_data/...) * ensure the project directory names conform to the standard - '''PROJECTCODE-YYYY_JJJxx_SITENAME''', e.g. ''GB07_07-2007_102a_Inverclyde'', ''boresight-2007_198'', etc == Tickets and tracking == === Status Page === Add details to the [http://arsf-dan.nerc.ac.uk/status/addflight/addflight.php processing status page]. Under 'Data location' use the full path to the project (e.g /users/rsg/arsf/... ). === Ticket === Raise a [http://arsf-dan.nerc.ac.uk/trac/newticket new trac ticket] (type 'flight processing') for the new data. * Ticket summary should be of the form '''EU10/03, flight day 172/2011, Dennys Wood''' * Add short version of scientific purpose to guide processing (check ARSF application in ~arsf/arsf_data/2009/ARSF_Applications) * Note arrival time of data * Set priority of ticket from project grading (try the grades subpages on [wiki:Projects] or hassle ARSF-Ops) * Note any specific comments that might help with processing * Owner should be blank * Verify the sensors that were requested in the application (primary) and those that weren't (secondary). Note in the ticket which these were. * Ticket body should contain: {{{ Data location: ~arsf/arsf_data/2011/flight_data/..... FILL IN Data arrived from ARSF via SATA disk LETTER OR network transfer on DATE. Scientific objective: FILL IN FROM APPLICATION (just enough to guide processing choices) Priority: FILL IN FROM APPLICATION/WIKI PAGE (e.g. alpha-5 low), set ticket priority appropriately PI: A. N. Other EUFAR Project ID: Any other notes.. = Sensors: = * Eagle (requested) * Hawk (requested) * Leica LIDAR (not requested but flown anyway) * RCD (requested) }}} == Vectors == If the site is in the UK, check the [http://rsg.pml.ac.uk/intranet/trac/wiki/Projects/ARSF-DAN/Vector_Requests vector page] to see if we have vectors for the site in question. If not then email ARSF-Ops (Gary) and ask for them, specifying the corner points of the area covered, in OS BNG grid coordinates. Use bounding_box.py (needs to be ran in the project directory) or use the generated nextmap DEM (below) to get the range of tiles, then do the conversion. It can take a couple of weeks before we get the vectors. == E-mail PI == Email the PI to inform them that their data has arrived for processing. Sample text: * fill in the 5 fields: , , , * the date of arrival should be when the disks arrived or when the download begun * cc to arsf-processing * also cc to !neodc@rl.ac.uk * set reply-to to arsf-processing * subject: `ARSF data arrival notification ( [])` {{{ Dear , This is a notification that your ARSF data for [] is at the ARSF Data Analysis Node for processing (data received from ARSF Operations on ). We aim to deliver as quickly as possible - our current processing priority order can be found at http://arsf-dan.nerc.ac.uk/status/order/order.php, though please note that this queue is subject to change. You can follow progress at the following webpages: http://arsf-dan.nerc.ac.uk/status/progress/progress.php - general status page http://arsf-dan.nerc.ac.uk/trac/ticket/ - our notes during processing (may be technical) If you would like any more information, please feel free to contact us at arsf-processing@pml.ac.uk Regards, }}} == Create DEM == Run the DEM generating script if a UK flight. `nextmapdem.sh` If the flight is outside the UK you'll need to process the LiDAR data first or use the ASTER/SRTM dem (see http://arsf-dan.nerc.ac.uk/trac/wiki/Processing/SRTMDEMs)