| Version 4 (modified by mggr, 15 years ago) (diff) |
|---|
Data arrival by network transfer
- change to arsf user on a safe machine
ssh arsf@gridmaster2 # (or su - arsf and don't turn your machine off!)
- find the exact data path on Ops server
ssh arsfdan@thelma.nerc-arsf.ac.uk # go find the path to the data to download # these will vary.. cd /data/Data_2009/ cd UK cd 281-09_GB08-02_Delamere # check path pwd # returns /data/Data_2009/UK/281-09_GB08-02_Delamere # # check size isn't crazy big # > 150GB and think about a disk transfer du -hsc . logout
- create a temporary directory in ~arsf_data/YYYY/flight_data/unpacking/ to download to, e.g.
DOWNDIR=~/arsf_data/`date +%Y`/flight_data/unpacking/download-`date +%Y%m%d%H%M%S` mkdir $DOWNDIR cd $DOWNDIR
- do the download with rsync, using compression (-z flag). Note there is no trailing / on the PROJECTNAME part of the path below - this is important!
date rsync -avz arsfdan@thelma.nerc-arsf.ac.uk:/PATH/YOU/FOUND/EARLIER/PROJECTNAME . ; date # date commands are optional, they just let you see when it started and stopped
If something goes wrong after an hour or two, don't delete what you've already downloaded. Instead, repeat the rsync command and it'll pick up where it left off.
Unpacking data
Return to Procedures/NewDataArrival