Version 45 (modified by knpa, 14 years ago) (diff)


NEODC Data Delivery

This page documents the procedure to follow when delivering data to NEODC.

First, we need to have some data to send - these should be datasets that are 'completed'.
All sensors need to be delivered and r-synced. Check with whoever processed each sensor if need be.

  1. Move workspace project into workspace/being_archived
  2. Prepare the relevant datasets:
    1. Make sure everything is where it should be! (see Processing/FilenameConventions for the layout)
    2. Check everything is there (LIDAR DELIVERY DEM, PDF logsheet, delivery directories, scripts, applanix/rinex data)
    3. Add a copy of the relevant trac ticket(s) ; run:
      mkdir -p admin/trac_ticket
      pushd admin/trac_ticket
      wget --recursive --level 1 --convert-links --html-extension
    4. Delete the contents of the lev1/ subdirectory, where these are duplicates of the delivery directory.
    5. Scan the filesystem for any 'bad' things and fix them:
      1. delete any unnecessary files - backups of DEMs that weren't used, temp files created by gedit (~ at end of filename), hidden files etc
      2. Remove executable bit on all files (except the point cloud filter and the run[aceh] scripts) -> find -type f -not -wholename '*pt_cloud_filter*' -and -not -regex '.*/run[aceh]/.*sh' -and -perm /a=x -exec chmod a-x {} \;
      3. Find all files/dirs with unusual characters (space, brackets, etc) - use find -regex '.*[^-0-9a-zA-Z/._].*' | ~arsf/usr/bin/ to give suggested commands, but check first
      4. Give everyone read permissions (and execute if it has user execute) for the current directory and below - chmod -R a+rX .
  3. Create an archive tarball for NEODC to download:
    su - arsf
    cd ~/arsf_data/archived/
     # figure out what you want to archive, e.g. all in 20080219 archive dir, then 20080219/2007/flight_data/uk/*
     # note you need to specify dirs at the project level, hence the * above
     # To run the archiving locally rather than via the grid engine, use:
    ./ 20080219/2007/flight_data/uk/*
    1. When complete, this will have dumped the data into ~arsf/arsf_data/archived/neodc_transfer_area/staging/. Check it looks ok then move it up one level so NEODC can rsync it.
  4. Notify NEODC they can download the data.
  5. When NEODC have the data:
    1. Remove it from the transfer area
    2. Note on ticket that it's been sent to neodc (with date.)
  6. When NEODC confirm they have backed up the data:
    1. Create a directory on /data/larsenb1/archived/arsf/<original path from ~arsf/arsf_data/> and move the data to it.
      e.g. mv ~arsf/2008/flight_data/uk/CEH08_01/ /data/larsenb1/archived/arsf/2008/flight_data/uk/CEH08_01
    2. Create a symlink to the project in it's original location. Point the symlink through ~arsf/arsf_data/archived rather than directly to larsen
      e.g. ln -s ~arsf/arsf_data/archived/2008/flight_data/uk/CEH08_01 ~arsf/arsf_data/2008/flight_data/uk/CEH08_01
    3. Note in ticket that it has been backed up NEODC and give new data location.
  7. When NEODC confirm that everything appears to be in order (maybe wait a week):
    • Close the ticket
    • Delete the workspace copy in being_archived