Version 31 (modified by knpa, 14 years ago) (diff) |
---|
NEODC Data Delivery
This page documents the procedure to follow when delivering data to NEODC.
First, we need to have some data to send - these should be datasets that are 'completed'.
All sensors need to be delivered and r-synced. Check with whoever processed each sensor if need be.
Work in progress'''
- Prepare the relevant datasets:
- Make sure everything is where it should be! (see Processing/FilenameConventions for the layout)
- Check everything is there (PDF logsheet, delivery directories, scripts, applanix/rinex data)
- Add a copy of the relevant trac ticket(s) ; run:
mkdir -p admin/trac_ticket pushd admin/trac_ticket wget --recursive --level 1 --convert-links --html-extension http://arsf-dan.nerc.ac.uk/trac/ticket/TICKETNUMBER popd
- Delete the contents of the lev1/ subdirectory, where these are duplicates of the delivery directory.
- Scan the filesystem for any 'bad' things and fix them:
- delete any unnecessary files - backups of DEMs that weren't used, temp files created by gedit (~ at end of filename), hidden files etc
- Remove executable bit on all files (except the point cloud filter and the run[aceh] scripts) -> find -type f -not -wholename '*pt_cloud_filter*' -and -not -regex '.*/run[aceh]/.*sh' -and -perm /a=x -exec chmod a-x {} \;
- Find all files/dirs with unusual characters (space, brackets, etc) - use find -regex '.*[^-0-9a-zA-Z/._].*' | ~arsf/usr/bin/fix_naughty_chars.py to give suggested commands, but check first
- Give everyone read permissions (and execute if it has user execute) for the current directory and below - chmod -R a+rX .
- Request DTG archives the data
- note request on ticket
- switch ticket component to "archiving" for ease of tracking.
- When DTG completes the archiving (on date YYYYMMDD):
- Create a directory as /data/larsenb1/archived/arsf/YYYYMMDD/..<original path from ~arsf/arsf_data/> and move the data to it
- e.g. ~arsf/2008/flight_data/uk/CEH08_01... backed up on 1st Feb 2009 would be /data/larsenb1/archived/arsf/20090201/2008/flight_data/uk/CEH08_01...
- Create a symlink to the project in it's original repository location (e.g. under ~arsf/arsf_data/2008/flight_data/uk)
- Create a symlink in ~arsf/arsf_data/archived to the archived data at /data/larsenb1/archived/arsf/YYYYMMDD/
- copy and delete the DTG metadata from ~utils/logs/archiving_scratch/arsf... to ~arsf/dtg_archive_metadata/YYYYMMDD/
- Create a directory as /data/larsenb1/archived/arsf/YYYYMMDD/..<original path from ~arsf/arsf_data/> and move the data to it
- Create an archive tarball for NEODC to download:
su - arsf cd ~/arsf_data/archived/ # figure out what you want to archive, e.g. all in 20080219 archive dir, then qsub_archiver.sh 20080219/2007/flight_data/uk/* # note you need to specify dirs at the project level, hence the * above # To run the archiving locally rather than via the grid engine, use: ./archive_helper-justdoit.sh 20080219/2007/flight_data/uk/*
- When complete, this will have dumped the data into ~arsf/arsf_data/archived/neodc_transfer_area/staging/. Check it looks ok then move it up one level so NEODC can rsync it.
- Notify NEODC they can download the data.
- When NEODC have the data:
- remove it from the transfer area
- note on ticket that it's been archived/sent to neodc
- close the ticket
- Delete the workspace copy