= NEODC Data Delivery =
This page documents the procedure to follow when delivering data to NEODC.
First, we need to have some data to send - these should be datasets that are 'completed'.
'''Work in progress!'''
1. Prepare the relevant datasets:
1. Make sure everything is where it should be! (see [wiki:Procedures/DataFlow this wiki page] for the layout)
1. Check everything is there (PDF logsheet, delivery directories, scripts, applanix/rinex data)
1. Add a copy of the relevant trac ticket(s) ; run:
{{{
mkdir -p admin/trac_ticket
pushd admin/trac_ticket
wget --recursive --level 1 --convert-links --html-extension http://arsf-dan.nerc.ac.uk/trac/ticket/TICKETNUMBER
popd
}}}
1. Delete the contents of the lev1/ subdirectory, where these are duplicates of the delivery directory.
1. Scan the filesystem for any 'bad' things and fix them:
1. delete any unnecessary files - backups of DEMs that weren't used, temp files created by gedit (~ at end of filename), etc
1. Remove executable bit on all files (except bin/az* and the run[aceh] scripts) -> `find -type f -not -wholename '*bin/az*' -and -not -regex '.*/run[aceh]/.*sh' -and -perm /a=x -exec chmod a-x {} \;`
1. Find all files/dirs with unusual characters (space, brackets, etc) - use `find -regex '.*[^-0-9a-zA-Z/._].*' | ~arsf/usr/bin/fix_naughty_chars.py` to give suggested commands, but check first
1. Give everyone read permissions (and execute if it has user execute) for the current directory and below - `chmod -R a+rX .`
1. Request DTG archives the data
1. note request on ticket
1. switch ticket component to "archiving" for ease of tracking.
1. When DTG completes the archiving (on date YYYYMMDD):
1. Create a directory as /data/larsenb1/archived/arsf/YYYYMMDD/.. and move the data to it
1. e.g. `~arsf/2008/flight_data/uk/CEH08_01...` backed up on 1st Feb 2009 would be `/data/larsenb1/archived/arsf/20090201/2008/flight_data/uk/CEH08_01...`
1. Create a symlink to the archive location under `~arsf/arsf_data/archived/YYYYMMDD/..`
1. copy and delete the DTG metadata from `~utils/logs/archiving/scratch/arsf...` to `~arsf/dtg_archive_metadata/YYYYMMDD/....`
1. Create an archive tarball for NEODC to download:
{{{
su - arsf
cd ~/arsf_data/archived/
# figure out what you want to archive, e.g. all in 20080219 archive dir, then
./archive_helper-justdoit.sh 20080219/2007/flight_data/uk/*
# note you need to specify dirs at the project level, hence the * above
}}}
1. When complete, this will have dumped the data into `~arsf/arsf_data/archived/neodc_transfer_area/staging/`. Check it looks ok then move it up one level so NEODC can rsync it.
1. Notify NEODC they can download the data.
1. When NEODC have the data:
* remove it from the transfer area
* note on ticket that it's been archived/sent to neodc
* close the ticket