= NEODC Data Delivery = This page documents the procedure to follow when delivering data to NEODC. First, we need to have some data to send - these should be datasets that are 'completed'.[[BR]] All sensors need to be delivered and r-synced. Run rsync with a --dry-run to check it is all up to date. Check with whoever processed each sensor if need be. [[BR]] 1. If there is a workspace version, move it into workspace/being_archived 1. Prepare the repository version for archiving: 1. Make sure everything is present and where it should be! (see [wiki:Processing/FilenameConventions] for the required layout and name formats) [[BR]] Things to look out for: '''Delivery folders, applanix/rinex data, las files, DEMs''' [[BR]] Use proj_tidy.sh to highlight any problems: {{{ proj_tidy.sh -p -c }}} 1. Add a copy of the relevant trac ticket(s) ; run: {{{ mkdir -p admin/trac_ticket pushd admin/trac_ticket wget --recursive --level 1 --convert-links --html-extension http://arsf-dan.nerc.ac.uk/trac/ticket/TICKETNUMBER popd }}} 1. Scan the filesystem for any 'bad' things and fix them: 1. Delete any unnecessary files - backups of DEMs that weren't used, temp files created by gedit (~ at end of filename), hidden files, duplicates in lev1 dir etc 1. Find all files/dirs with unusual characters (space, brackets, etc), ignoring the admin/trac_ticket folder: {{{ find -regex '.*[^-0-9a-zA-Z/._].*' -o -path './admin/trac_ticket' -prune | ~arsf/usr/bin/fix_naughty_chars.py }}} This will give suggested commands, but check first. 1. Set permissions: 1. Remove executable bit on all files (except the point cloud filter and the run[aceh] scripts): {{{ find -type f -not -wholename '*pt_cloud_filter*' -and -not -regex '.*/run[aceh]/.*sh' -and -perm /a=x -exec chmod a-x {} \; }}} 1. Give everyone read permissions (and execute if it has user execute) for the current directory and below: {{{ chmod -R a+rX . }}} 1. Create the tarballs for NEODC to download:[[BR]][[BR]] ''If AIMMS or GRIMM data is present then you will first need to separate these and put into separate tarballs. [[BR]] Use: tar czf [[BR]] Tarball name should be in format: GB09_05-2009_278b_Leighton_moss-AIMMS.tar.gz [[BR]] Create a md5sum file for the AIMMS/GRIMM data also [[BR]] Use: md5sum > -MD5SUM.txt'' {{{ su - arsf ~/arsf_data/archived/qsub_archiver.sh (e.g. ~arsf/arsf_data/2011/flight_data/spain_portugal/EU11_03-2011_142_Jimena/) # To run the archiving locally rather than via the grid engine, use: ~arsf/usr/bin/archiving_tarballer.sh }}} When complete, this will have dumped the data into `~arsf/arsf_data/archived/neodc_transfer_area/staging/`. Check it looks OK then move it up one level so NEODC can rsync it. Logs will be in ~arsf/arsf_data/archived/archiver_logs/. [[BR]][[BR]] 1. Notify NEODC they can download the data (Current contact is: wendy.garland@stfc.ac.uk) and record the date in the ticket. 1. When NEODC confirm they have backed up the data: 1. Remove tarball from the transfer area 1. Move the repository project to archive disk at: ~arsf/arsf_data/archived/ [[BR]] e.g. `mv ~arsf/arsf_data/2008/flight_data/uk/CEH08_01/ ~arsf/arsf_data/archived/2008/flight_data/uk/CEH08_01` [[BR]] You may need to create parent directories if they don't yet exist. 1. Create a symlink to the project in it's original location. Point the symlink through ~arsf/arsf_data/archived rather than directly to specific disk. [[BR]] e.g. `ln -s ~arsf/arsf_data/archived/2008/flight_data/uk/CEH08_01 ~arsf/arsf_data/2008/flight_data/uk/CEH08_01` 1. Note in ticket that it has been backed up by NEODC and moved to archive disk. 1. Final steps - maybe wait a month: 1. If workspace version present, delete from being_archived. 1. Close the ticket