Changes between Initial Version and Version 1 of Procedures/NewDataArrival/Network


Ignore:
Timestamp:
Oct 9, 2009 11:00:48 AM (11 years ago)
Author:
mggr
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Procedures/NewDataArrival/Network

    v1 v1  
     1= Data arrival by network transfer =
     2
     3 1. change to arsf user on a safe machine
     4   * `ssh arsf@gridmaster2` # (or `su - arsf` and don't turn your machine off!)
     5 1. find the exact data path on Ops server
     6{{{
     7ssh arsfdan@thelma.nerc-arsf.ac.uk
     8 # go find the path to the data to download
     9 # these will vary..
     10cd /data/Data_2009/
     11cd UK
     12cd 281-09_GB08-02_Delamere
     13 # check path
     14pwd
     15 # returns /data/Data_2009/UK/281-09_GB08-02_Delamere
     16 #
     17 # check size isn't crazy big
     18 # > 150GB and think about a disk transfer
     19du -hsc .
     20logout
     21}}}
     22 1. create a temporary directory in ~arsf_data/YYYY/flight_data/unpacking/ to download to, e.g.
     23{{{
     24DOWNDIR=~/arsf_data/`date +%Y`/flight_data/unpacking/download-`date +%Y%m%d%H%M%S`
     25mkdir $DOWNDIR
     26cd $DOWNDIR
     27}}}
     28 1. do the download with rsync.  Note there is '''no trailing /''' on the PROJECTNAME part of the path below - this is important!
     29   * `rsync -avz arsfdan@thelma.nerc-arsf.ac.uk:/PATH/YOU/FOUND/EARLIER/PROJECTNAME . ; date`
     30
     31If something goes wrong after an hour or two, don't delete what you've already downloaded.  Instead, repeat the rsync command and it'll pick up where it left off.
     32
     33== Unpacking data ==
     34
     35Return to [wiki:Procedures/NewDataArrival]