NEODC Data Delivery

This page documents the procedure to follow when delivering data to NEODC.

First, we need to have some data to send - these should be datasets that are 'completed'.

Work in progress'''

1. Prepare the relevant datasets:
   1. Make sure everything is where it should be! (see this wiki page? for the layout)
   2. Check everything is there (PDF logsheet, delivery directories, scripts, applanix/rinex data)
   3. Add a copy of the relevant trac ticket(s) ; run:

      mkdir -p admin/trac_ticket
      pushd admin/trac_ticket
      wget --recursive --level 1 --convert-links --html-extension http://arsf-dan.nerc.ac.uk/trac/ticket/TICKETNUMBER
      popd
      

   4. Delete the contents of the lev1/ subdirectory, where these are duplicates of the delivery directory.
2. Scan the filesystem for any 'bad' things and fix them:
   1. delete any unnecessary files - backups of DEMs that weren't used, temp files created by gedit (~ at end of filename), etc
   2. Remove executable bit on all files (except bin/az* and the run[aceh] scripts) -> find -type f -not -wholename '*bin/az*' -and -not -regex '.*/run[aceh]/.*sh' -and -perm /a=x -exec chmod a-x {} \;
   3. Find all files/dirs with unusual characters (space, brackets, etc) - use find -regex '.*[^-0-9a-zA-Z/._].*' | ~arsf/usr/bin/fix_naughty_chars.py to give suggested commands, but check first
   4. Give everyone read permissions (and execute if it has user execute) for the current directory and below - chmod -R a+rX .
3. Request DTG archives the data
   1. note request on ticket
   2. switch ticket component to "archiving" for ease of tracking.
4. When DTG completes the archiving (on date YYYYMMDD):
   1. Create a directory as /data/larsenb1/archived/arsf/YYYYMMDD/..<original path from ~arsf/arsf_data/> and move the data to it
      1. e.g. ~arsf/2008/flight_data/uk/CEH08_01... backed up on 1st Feb 2009 would be /data/larsenb1/archived/arsf/20090201/2008/flight_data/uk/CEH08_01...
   2. Create a symlink to the archive location under ~arsf/arsf_data/archived/YYYYMMDD/..<original path from ~arsf/arsf_data/>
   3. copy and delete the DTG metadata from ~utils/logs/archiving/scratch/arsf... to ~arsf/dtg_archive_metadata/YYYYMMDD/....
5. Create an archive tarball for NEODC to download:

   su - arsf
   cd ~/arsf_data/archived/
    # figure out what you want to archive, e.g. all in 20080219 archive dir, then
   ./archive_helper-justdoit.sh 20080219/2007/flight_data/uk/*
    # note you need to specify dirs at the project level, hence the * above
   

   1. When complete, this will have dumped the data into ~arsf/arsf_data/archived/neodc_transfer_area/staging/. Check it looks ok then move it up one level so NEODC can rsync it.
6. Notify NEODC they can download the data.
7. When NEODC have the data, note on ticket that it's been archived/sent to neodc and close it.