Version 25 (modified by knpa, 14 years ago) (diff)


NEODC Data Delivery

This page documents the procedure to follow when delivering data to NEODC.

First, we need to have some data to send - these should be datasets that are 'completed'.

Work in progress'''

  1. Prepare the relevant datasets:
    1. Make sure everything is where it should be! (see Processing/FilenameConventions for the layout)
    2. Check everything is there (PDF logsheet, delivery directories, scripts, applanix/rinex data)
    3. Add a copy of the relevant trac ticket(s) ; run:
      mkdir -p admin/trac_ticket
      pushd admin/trac_ticket
      wget --recursive --level 1 --convert-links --html-extension
    4. Delete the contents of the lev1/ subdirectory, where these are duplicates of the delivery directory.
  2. Scan the filesystem for any 'bad' things and fix them:
    1. delete any unnecessary files - backups of DEMs that weren't used, temp files created by gedit (~ at end of filename), etc
    2. Remove executable bit on all files (except bin/az* and the run[aceh] scripts) -> find -type f -not -wholename '*bin/az*' -and -not -regex '.*/run[aceh]/.*sh' -and -perm /a=x -exec chmod a-x {} \;
    3. Find all files/dirs with unusual characters (space, brackets, etc) - use find -regex '.*[^-0-9a-zA-Z/._].*' | ~arsf/usr/bin/ to give suggested commands, but check first
    4. Give everyone read permissions (and execute if it has user execute) for the current directory and below - chmod -R a+rX .
  3. Request DTG archives the data
    1. note request on ticket
    2. switch ticket component to "archiving" for ease of tracking.
  4. When DTG completes the archiving (on date YYYYMMDD):
    1. Create a directory as /data/larsenb1/archived/arsf/YYYYMMDD/..<original path from ~arsf/arsf_data/> and move the data to it
      1. e.g. ~arsf/2008/flight_data/uk/CEH08_01... backed up on 1st Feb 2009 would be /data/larsenb1/archived/arsf/20090201/2008/flight_data/uk/CEH08_01...
    2. Create a symlink in ~arsf/arsf_data/archived to the archived data at /data/larsenb1/archived/arsf/YYYYMMDD/
    3. copy and delete the DTG metadata from ~utils/logs/archiving_scratch/arsf... to ~arsf/dtg_archive_metadata/YYYYMMDD/
  5. Create an archive tarball for NEODC to download:
    su - arsf
    cd ~/arsf_data/archived/
     # figure out what you want to archive, e.g. all in 20080219 archive dir, then 20080219/2007/flight_data/uk/*
     # note you need to specify dirs at the project level, hence the * above
     # To run the archiving locally rather than via the grid engine, use:
    ./ 20080219/2007/flight_data/uk/*
    1. When complete, this will have dumped the data into ~arsf/arsf_data/archived/neodc_transfer_area/staging/. Check it looks ok then move it up one level so NEODC can rsync it.
  6. Notify NEODC they can download the data.
  7. When NEODC have the data:
    • remove it from the transfer area
    • note on ticket that it's been archived/sent to neodc
    • close the ticket