Arrival of new flight data

This procedure should be followed on receipt of new flight data from ARSF.

If in any doubt about something (e.g. a dataset has two project codes), contact Gary.

Permissions fixing

Ensure permissions on the SATA disk are correct and all files readable ("chmod a+rX /mnt/satadisk" as root)

Copying to PML data storage

Copy to appropriate location on the filesystem (~arsf/arsf_data/in_progress/2007/flight_data/...)

• when copying from the source media, preserve timestamps with cp -a or rsync -av
• ensure the directory name conforms to the standard - PROJECTCODE-YYYY_JJJxx_SITENAME, e.g. GB07_07-2007_102a_Inverclyde, boresight-2007_198, etc
• prune empty directories (check these aren't stubs for later data first) (use rmdir * in the project directory)
• rename any capitalised subdirectories
• remove any spaces, brackets or other Unix-upsetting characters in filenames
  • use find -regex '.*[^-0-9a-zA-Z/._].*' | ~arsf/usr/bin/fix_naughty_chars.py to give suggested commands, but check before pasting commands!
• remove executable bit on all files -> find -type f -exec chmod a-x {} \;
• move DCALM* files to applanix/Raw
• bzip lidar files, bzip lidar/*
• remove group & other write permission, chmod go-w . -R

In some cases, the flight crew may fly two projects back-to-back but enter all the data onto a single logsheet. If so, you may need to split the project directory into two, particularly if there's a large time gap (navigation needs separate processing) or the PIs are different (different delivery addresses/tracking). If you do need to split a project, ensure both copies have copies of common files (logsheets, rinex, etc), but that non-common files are not duplicated (ie. don't include hawk data for part 1 in part 2..). Also note in the ticket what was done for tracking purposes.

Verification

Look at the logsheet and verify that we have copies of all relevant data mentioned there.

Check the filesizes of all data files (Eagle, Hawk, ATM, CASI) to make sure none are zero bytes (or obviously broken in some way).

Tickets and tracking

Raise a trac ticket (type 'flight processing') for the new data, then add details to the ​processing status page.

• ticket summary should be of the form BGS07/02, flight day 172/2007, Keyworth
• add scientific summary (check ARSF application in ~arsf/arsf_data/in_progress/2007/ARSF_Applications-GB_2007)
• note arrival time of data
• note any specific comments that might help with processing
• owner should be blank
• ticket body should contain:

  Data location: ~arsf/arsf_data/in_progress/2007/flight_data/uk/GB04_19-2007_102b_Nigg_Bay
  
  Data arrived from ARSF via initial SATA disk transfer in June.
  
  Scientific details: (details of 2004 applications not available at PML)
  
  PI: A. N. Other
  
  Any other notes..
  
  == Sensors: ==
   * ATM
   * CASI
   * Eagle
   * Hawk
  
  == Quicklook ==
  
  TBD.