= Data arrival by network transfer = 1. change to arsf user on a safe machine {{{ ssh arsf@gridmaster2 # (or su - arsf and don't turn your machine off!) }}} 1. find the exact data path on Ops server {{{ ssh arsfdan@thelma.nerc-arsf.ac.uk # go find the path to the data to download # these will vary.. cd /data/Data_2009/ ##Also try /mnt/synology1/data/Data\ 2014/ or synology2 cd UK cd 281-09_GB08-02_Delamere # check path pwd # returns /data/Data_2009/UK/281-09_GB08-02_Delamere # # check size isn't crazy big # > 150GB and think about a disk transfer du -hsc . # if you get any permission denied errors, fix up permissions then redo the du. # fix permissions with: # /data/permissions_fixer/fix_perms_for_arsfdan.sh DIRECTORY_NEEDING_FIX # **be very careful** with this as you could easily screw up the whole thelma server if you get the directory wrong! logout }}} 1. create a temporary directory in ~arsf_data/YYYY/flight_data/unpacking/ to download to, e.g. {{{ DOWNDIR=~/arsf_data/`date +%Y`/flight_data/unpacking/download-`date +%Y%m%d%H%M%S` mkdir $DOWNDIR cd $DOWNDIR }}} 1. do the download with rsync, using compression (-z flag). Note there is '''no trailing /''' on the PROJECTNAME part of the path below - this is important! * example rsync command follows, with a --dry-run option to prevent any damage occurring on the trial attempt. If it lists the files you'd expect it to transfer, remove the --dry-run option and repeat. If not, check your slashes! {{{ date rsync --dry-run -avz arsfdan@thelma.nerc-arsf.ac.uk:/PATH/YOU/FOUND/EARLIER/PROJECTNAME . ; date # date commands are optional, they just let you see when it started and stopped }}} If something goes wrong after an hour or two, don't delete what you've already downloaded. Instead, repeat the rsync command and it'll pick up where it left off. == Unpacking data == Return to [wiki:Procedures/NewDataArrival]