Changes between Version 189 and Version 190 of Procedures/NewDataArrival


Ignore:
Timestamp:
Feb 3, 2017, 12:03:05 PM (7 years ago)
Author:
asm
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Procedures/NewDataArrival

    v189 v190  
    11= Arrival of new flight data =
    22
    3 This procedure should be followed on receipt of new flight data from ARSF.
     3This procedure should be followed on receipt of new flight data from NERC-ARF.
    44
    55If in any doubt about something (e.g. a dataset has two project codes), contact Gary.
    6 
    7 == Copy the data onto the system ==
    8 
    9  * [wiki:Procedures/NewDataArrival/Disk Data by SATA disk or DVD]
    10  * [wiki:Procedures/NewDataArrival/Network Data by network transfer]
    11 
    12 After successfully copying the data onto the system, email arsf-processing to confirm the data transferred ok (but hasn't yet been checked!).
    13 
    14 == Unpacking projects ==
    15 
    16 It is recommended you create a copy of the data and unpack this, incase anything goes wrong. Once the data has been successfully unpacked and moved to permanent storage the original copy from ops can be removed.
    176
    187== Project Set Up ==
    198'''Warning''': proj_tidy.sh is becoming deprecated so use this instructions instead -> [wiki:Procedures/NewDataArrival/ProjectSetUp]
    209
    21 === proj_tidy.sh  ===
    22 
    23 Run proj_tidy.sh:
    24 {{{
    25 proj_tidy.sh -e -p <path_to_project>
    26 }}}
    27 
    28 This will do three things:
    29 
    30 - Run a series of checks which you should go through and fix if necessary[[BR]]
    31 - Fix a few things automatically (permissions/common spelling mistakes)[[BR]]
    32 - Provide a list of commands (if any) that will convert the project from the old style structure to the current standard. These should be checked then pasted into the terminal.
    33 
    34 === Manual steps ===
    35  * Check time stamps of files vs julian day to check we use the correct julian day
    36  * Prune empty directories (check these aren't stubs for later data first)
    37  * Rename any remaining spelling mistakes in directories or files. Check hyperspectral raws have VNIR/SWIR/FENIX prefix.
    38  * Check all data belongs to this flight and move out/delete if belongs to a different flight day.
    3910
    4011== Verification ==
    … …  
    4516   * In some cases, the flight crew may fly two projects back-to-back but enter all the data onto a single logsheet.  If so, you may need to split the project directory into two, particularly if there's a large time gap (navigation needs separate processing) or the PIs are different (different delivery addresses/tracking).  If you do need to split a project, ensure both copies have copies of common files (logsheets, rinex, etc), but that non-common files are not duplicated (ie. don't include hawk data for part 1 in part 2..).  Also note in the ticket what was done for tracking purposes.
    4617 * Check the filesizes of all data files (Eagle, Hawk, ATM, CASI) to make sure none are zero bytes (or obviously broken in some way).
    47  * Verify the details on the logsheet (esp. PI) by calling/emailing ARSF-Ops (probably Gary) - the application form and logsheet are not reliable enough, nor do they track any changes in PI over the lifetime of the application. Get the alpha grading as well.
     18 * Verify the details on the logsheet (esp. PI) by calling/emailing NERC-ARF Ops (probably Gary) - the application form and logsheet are not reliable enough, nor do they track any changes in PI over the lifetime of the application. Get the alpha grading as well.
    4819
    49 
    50 == Move to permanent PML data storage ==
    51 
    52 Move to appropriate location in the repository (~arsf/arsf_data/2011/flight_data/...)
    53   * ensure the project directory names conform to the standard - '''PROJECTCODE-YYYY_JJJxx_SITENAME''', e.g. ''GB07_07-2007_102a_Inverclyde'', ''boresight-2007_198'', etc
    5420
    5521== Generate Logsheet ==
    … …  
    5824 * Run from top level - writes into 'admin' dir. You may wish to rename this dir before running to prevent any accidental overwriting.
    5925 * Fill in the yellow boxes with information from supplied logsheet.
    60  * Blue boxes should be automatically filled in when you start the program but check the values are correct (or at least that they appear so).
     26 * Blue boxes should be automatically filled in when you start the program but check the values are correct.
    6127 * Click 'fill post manual'. This will calculate the values for the green fields based on the values you have entered into the yellow fields. You should also check that these fields look correct. If the flight was flown over midnight, you may need to correct them.
    6228
    … …  
    6632=== Status Page ===
    6733
    68 Add details to the [http://arsf-dan.nerc.ac.uk/status/addflight/ processing status page]. Under 'Data location' use the full path to the project and do NOT include server name (e.g use /users/rsg/arsf/... and not /data/vishyuan/...).
     34Add details to the [http://arsf-dan.nerc.ac.uk/status/addflight/ processing status page]. Under 'Data location' use the full path to the project and do NOT include server name (e.g use /users/rsg/arsf/... and not /data/visuyuan/...).
    6935
    7036=== Ticket ===
    … …  
    7238Raise a [http://arsf-dan.nerc.ac.uk/trac/newticket new trac ticket] (type 'flight processing') for the new data.
    7339 * Ticket summary should be of the form '''EU10/03, flight day 172/2011, Dennys Wood'''
    74  * Add short version of scientific purpose to guide processing (check ARSF application in ~arsf/arsf_data/2009/ARSF_Applications)
     40 * Add short version of scientific purpose to guide processing (check NERC-ARF application)
    7541 * Note arrival time of data
    76  * Set priority of ticket from project grading (try the grades subpages on [wiki:Projects] or hassle ARSF-Ops)
     42 * Set priority of ticket from project grading (try the grades subpages on [wiki:Projects])
    7743 * Note any specific comments that might help with processing
    7844 * Owner should be blank
    … …  
    8248Data location: ~arsf/arsf_data/2011/flight_data/..... FILL IN
    8349
    84 Data arrived from ARSF via SATA disk LETTER OR network transfer on DATE.
     50Data arrived from NERC-ARF via SATA disk LETTER OR network transfer on DATE.
    8551
    8652Scientific objective: FILL IN FROM APPLICATION (just enough to guide processing choices)
    … …  
    11177== Vectors ==
    11278
    113 If the site is in the UK, check the [wiki:vectors vector page] to see if we have vectors for the site in question. If not then email ARSF-Ops (Gary) and ask for them, specifying the corner points of the area covered, in OS BNG grid coordinates. Use bounding_box.py (needs to be ran in the project directory) or use the generated nextmap DEM (below) to get the range of tiles, then do the conversion. It can take a couple of weeks before we get the vectors.
     79If the site is in the UK, check the [wiki:vectors vector page] to see if we have vectors for the site in question. If not then email NERC-ARF-Ops (Gary) and ask for them, specifying the corner points of the area covered, in OS BNG grid coordinates. Use bounding_box.py (needs to be ran in the project directory) or use the generated nextmap DEM (below) to get the range of tiles, then do the conversion. It can take a couple of weeks before we get the vectors.
    11480
    11581== E-mail PI ==
    … …  
    11985   * the date of arrival should be when the disks arrived or when the download begun
    12086   * the username and password are available in the .htaccess file in processing/kml_overview or ~arsf/usr/share/kmlpasswords.csv. Note that make_kmloverview.py will need to have been run to create a password first unless the project code already exists for previous data.
    121  * cc to arsf-processing
     87 * cc to nerc-arf-processing
    12288 * also cc to !neodc@rl.ac.uk
    123  * set reply-to to arsf-processing
    124  * subject: `ARSF data arrival notification (<PROJECT> [<EUFAR ID>])`
     89 * set reply-to to nerc-arf-processing
     90 * subject: `NERC-ARF data arrival notification (<PROJECT> [<EUFAR ID>])`
    12591{{{
    12692Dear <PI_NAME>,
    12793
    128 This is a notification that your ARSF data for <PROJECT> [<EUFAR ID>], flown on
    129 <CALENDAR_DAY(S)>, are at the ARSF Data Analysis Node for processing (data
    130 received from ARSF Operations on <DATE_OF_ARRIVAL>).
     94This is a notification that your NERC-ARF data for <PROJECT> [<EUFAR ID>], flown on
     95<CALENDAR_DAY(S)>, are at the NERC-ARF Data Analysis Node for processing (data
     96received from NERC-ARF Operations on <DATE_OF_ARRIVAL>).
    13197
    13298We aim to deliver as quickly as possible - our current processing