Version 39 (modified by knpa, 14 years ago) (diff)


NEODC Data Delivery

This page documents the procedure to follow when delivering data to NEODC.

First, we need to have some data to send - these should be datasets that are 'completed'.
All sensors need to be delivered and r-synced. Check with whoever processed each sensor if need be.

  1. Move workspace project into workspace/being_archived
  2. Prepare the relevant datasets:
    1. Make sure everything is where it should be! (see Processing/FilenameConventions for the layout)
    2. Check everything is there (LIDAR DELIVERY DEM, PDF logsheet, delivery directories, scripts, applanix/rinex data)
    3. Add a copy of the relevant trac ticket(s) ; run:
      mkdir -p admin/trac_ticket
      pushd admin/trac_ticket
      wget --recursive --level 1 --convert-links --html-extension
    4. Delete the contents of the lev1/ subdirectory, where these are duplicates of the delivery directory.
  3. Scan the filesystem for any 'bad' things and fix them:
    1. delete any unnecessary files - backups of DEMs that weren't used, temp files created by gedit (~ at end of filename), hidden files etc
    2. Remove executable bit on all files (except the point cloud filter and the run[aceh] scripts) -> find -type f -not -wholename '*pt_cloud_filter*' -and -not -regex '.*/run[aceh]/.*sh' -and -perm /a=x -exec chmod a-x {} \;
    3. Find all files/dirs with unusual characters (space, brackets, etc) - use find -regex '.*[^-0-9a-zA-Z/._].*' | ~arsf/usr/bin/ to give suggested commands, but check first
    4. Give everyone read permissions (and execute if it has user execute) for the current directory and below - chmod -R a+rX .
  4. Request DTG archives the data and note request on ticket
  5. When DTG completes the archiving (on date YYYYMMDD):
    1. Create a directory as /data/larsenb1/archived/arsf/YYYYMMDD/..<original path from ~arsf/arsf_data/> and move the data to it
      1. e.g. ~arsf/2008/flight_data/uk/CEH08_01... backed up on 1st Feb 2009 would be /data/larsenb1/archived/arsf/20090201/2008/flight_data/uk/CEH08_01...
    2. Create a symlink to the project in it's original repository location (e.g. under ~arsf/arsf_data/2008/flight_data/uk)
    3. Create a symlink in ~arsf/arsf_data/archived to the archived data at /data/larsenb1/archived/arsf/YYYYMMDD/
    4. copy and delete the DTG metadata from ~utils/logs/archiving_scratch/arsf... to ~arsf/dtg_archive_metadata/YYYYMMDD/
  6. Create an archive tarball for NEODC to download:
    su - arsf
    cd ~/arsf_data/archived/
     # figure out what you want to archive, e.g. all in 20080219 archive dir, then 20080219/2007/flight_data/uk/*
     # note you need to specify dirs at the project level, hence the * above
     # To run the archiving locally rather than via the grid engine, use:
    ./ 20080219/2007/flight_data/uk/*
    1. When complete, this will have dumped the data into ~arsf/arsf_data/archived/neodc_transfer_area/staging/. Check it looks ok then move it up one level so NEODC can rsync it.
  7. Notify NEODC they can download the data.
  8. When NEODC have the data:
    • remove it from the transfer area
    • note on ticket that it's been archived/sent to neodc (with date.)
    • close the ticket
    • Delete the workspace copy in being_archived - Maybe wait a couple of weeks for NEODC have time to check nothing is missing?