Changes between Version 202 and Version 203 of Procedures/NewDataArrival


Ignore:
Timestamp:
Sep 21, 2018 8:35:11 AM (10 months ago)
Author:
dac
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Procedures/NewDataArrival

    v202 v203  
    99A semi-automated procedure -> [wiki:Procedures/NewDataArrival/ProjectSetUp]
    1010
     11== Tickets and tracking ==
     12
     13=== Status Page ===
     14
     15Add details to the [https://nerc-arf-dan.pml.ac.uk/status/addflight/ processing status page]. Under 'Data location' use the full path to the project and do NOT include server name (e.g use /users/rsg/arsf/... and not /data/visuyuan/...).
     16
     17=== Ticket ===
     18
     19Raise a [https://nerc-arf-dan.pml.ac.uk/trac/newticket new trac ticket] (type 'flight processing') for the new data.
     20 * Ticket summary should be of the form '''EU10/03, flight day 172/2011, Dennys Wood'''
     21 * Add short version of scientific purpose to guide processing (check NERC-ARF application)
     22 * Note arrival time of data
     23 * Set priority of ticket from project grading (try the grades subpages on [wiki:Projects])
     24 * Note any specific comments that might help with processing
     25 * Owner should be blank
     26 * Verify the sensors that were requested in the application (primary) and those that weren't (secondary).  Note in the ticket which these were.
     27 * Ticket body should contain:
     28{{{
     29Data location: ~arsf/arsf_data/2011/flight_data/..... FILL IN
     30
     31Data arrived from NERC-ARF via SATA disk LETTER OR network transfer on DATE.
     32
     33Scientific objective: FILL IN FROM APPLICATION (just enough to guide processing choices)
     34
     35Priority: FILL IN FROM APPLICATION/WIKI PAGE (e.g. alpha-5 low), set ticket priority appropriately
     36
     37PI: A. N. Other
     38EUFAR Project ID:
     39CAFAM code:
     40
     41Any other notes..
     42
     43= Sensors: =
     44 * Fenix (requested/not requested, flown/not flown)
     45 * Leica LIDAR (requested/not requested, flown/not flown)
     46 * OWL (requested/not requested, flown/not flown)
     47 * RCD (requested/not requested, flown/not flown)
     48}}}
     49[[BR]]
     50
     51== Create KML overview ==
     52
     53When happy the directory is unpacked correctly run [wiki:Processing/makekmloverview make_kmloverview.py]. This will generate the web page and quicklooks of the data that are made available to the user. It will also generate a password which should be included in the email below. Check the username and password works by commenting out the following line:
     54
     55== E-mail PI ==
     56
     57Email the PI to inform them that their data has arrived to the Data Analysis Node for processing.  Sample text:
     58 * fill in the 7 fields: <PI_NAME>, <PROJECT>, <EUFAR ID> <TICKET_NO>, <DATE_OF_ARRIVAL>, <USERNAME>, <PASSWORD>, <INSERTWEBLINKHERE>
     59   * the date of arrival should be when the disks arrived or when the download begun
     60   * the username and password are available in the .htaccess file in processing/kml_overview or ~arsf/usr/share/kmlpasswords.csv. Note that make_kmloverview.py will need to have been run to create a password first unless the project code already exists for previous data.
     61 * cc to nerc-arf-processing
     62 * set reply-to to nerc-arf-processing
     63 * subject: `NERC-ARF data arrival notification (<PROJECT> [<EUFAR ID>])`
     64{{{
     65Dear <PI_NAME>,
     66
     67This is a notification that your NERC-ARF data for <PROJECT> [<EUFAR ID>], flown on
     68<CALENDAR_DAY(S)>, are at the NERC-ARF Data Analysis Node for processing (data
     69received from NERC-ARF Operations on <DATE_OF_ARRIVAL>).
     70
     71We aim to deliver as quickly as possible. You can follow progress at the following webpages:
     72
     73 https://nerc-arf-dan.pml.ac.uk/status/progress/
     74  - general status page
     75
     76 https://nerc-arf-dan.pml.ac.uk/trac/ticket/631 <TICKETNO>
     77  - our notes during processing (may be technical)
     78
     79Also available is the PI page which will allow you to view flight data coverage and download quicklooks. This page will be updated as the data are processed and more information becomes available. Click the following link and enter username and password details as below:
     80
     81<INSERTWEBLINKHERE>
     82
     83username: <USERNAME>
     84password: <PASSWORD>
     85
     86If there are any specific processing requirements, or if you have additional data that may help with the processing (e.g. GPS basestation data, ground or atmospheric measurements), please contact us directly so we can take account of this before beginning processing.
     87
     88If you would like any more information, please feel free to contact us at nerc-arf-processing@pml.ac.uk
     89
     90Regards,
     91
     92}}}
     93
     94Once the PI has been emailed commit the updated KML password file to ~arsf/live_git_repos/config_files/ (no need to push changes).
     95[[BR]]
     96
     97== Basestation ==
     98
     99Check there is basestation data for the flight. If not, ask ops if they have some or if we need to [wiki:Processing/GNSSStatinons download it].
    11100
    12101
     
    68157
    69158[[BR]]
    70 
    71 == Tickets and tracking ==
    72 
    73 === Status Page ===
    74 
    75 Add details to the [https://nerc-arf-dan.pml.ac.uk/status/addflight/ processing status page]. Under 'Data location' use the full path to the project and do NOT include server name (e.g use /users/rsg/arsf/... and not /data/visuyuan/...).
    76 
    77 === Ticket ===
    78 
    79 Raise a [https://nerc-arf-dan.pml.ac.uk/trac/newticket new trac ticket] (type 'flight processing') for the new data.
    80  * Ticket summary should be of the form '''EU10/03, flight day 172/2011, Dennys Wood'''
    81  * Add short version of scientific purpose to guide processing (check NERC-ARF application)
    82  * Note arrival time of data
    83  * Set priority of ticket from project grading (try the grades subpages on [wiki:Projects])
    84  * Note any specific comments that might help with processing
    85  * Owner should be blank
    86  * Verify the sensors that were requested in the application (primary) and those that weren't (secondary).  Note in the ticket which these were.
    87  * Ticket body should contain:
    88 {{{
    89 Data location: ~arsf/arsf_data/2011/flight_data/..... FILL IN
    90 
    91 Data arrived from NERC-ARF via SATA disk LETTER OR network transfer on DATE.
    92 
    93 Scientific objective: FILL IN FROM APPLICATION (just enough to guide processing choices)
    94 
    95 Priority: FILL IN FROM APPLICATION/WIKI PAGE (e.g. alpha-5 low), set ticket priority appropriately
    96 
    97 PI: A. N. Other
    98 EUFAR Project ID:
    99 CAFAM code:
    100 
    101 Any other notes..
    102 
    103 = Sensors: =
    104  * Fenix (requested/not requested, flown/not flown)
    105  * Leica LIDAR (requested/not requested, flown/not flown)
    106  * OWL (requested/not requested, flown/not flown)
    107  * RCD (requested/not requested, flown/not flown)
    108 }}}
    109 [[BR]]
    110 
    111 == Create KML overview ==
    112 
    113 When happy the directory is unpacked correctly run [wiki:Processing/makekmloverview make_kmloverview.py]. This will generate the web page and quicklooks of the data that are made available to the user. It will also generate a password which should be included in the email below. Check the username and password works by commenting out the following line:
    114 {{{
    115 Allow from env=internal_proxy
    116 }}}
    117 from processing/kml_overview/.htaccess as this means a password isn't required for internal access. Remember to uncomment once you have finished.
    118 [[BR]]
    119 
    120 == Vectors ==
    121 
    122 If the site is in the UK, check the [wiki:vectors vector page] to see if we have vectors for the site in question. If not, then email NERC-ARF-Ops and ask for them specifying the corner points of the area covered  in OS BNG grid coordinates. Use bounding_box.py (needs to be run in the project directory) or use the generated nextmap DEM (below) to get the range of tiles, then do the conversion. It can take a couple of weeks before we get the vectors.
    123 [[BR]]
    124 
    125 == E-mail PI ==
    126 
    127 Email the PI to inform them that their data has arrived to the Data Analysis Node for processing.  Sample text:
    128  * fill in the 7 fields: <PI_NAME>, <PROJECT>, <EUFAR ID> <TICKET_NO>, <DATE_OF_ARRIVAL>, <USERNAME>, <PASSWORD>, <INSERTWEBLINKHERE>
    129    * the date of arrival should be when the disks arrived or when the download begun
    130    * the username and password are available in the .htaccess file in processing/kml_overview or ~arsf/usr/share/kmlpasswords.csv. Note that make_kmloverview.py will need to have been run to create a password first unless the project code already exists for previous data.
    131  * cc to nerc-arf-processing
    132  * also cc to !neodc@rl.ac.uk
    133  * set reply-to to nerc-arf-processing
    134  * subject: `NERC-ARF data arrival notification (<PROJECT> [<EUFAR ID>])`
    135 {{{
    136 Dear <PI_NAME>,
    137 
    138 This is a notification that your NERC-ARF data for <PROJECT> [<EUFAR ID>], flown on
    139 <CALENDAR_DAY(S)>, are at the NERC-ARF Data Analysis Node for processing (data
    140 received from NERC-ARF Operations on <DATE_OF_ARRIVAL>).
    141 
    142 We aim to deliver as quickly as possible. You can follow progress at the following webpages:
    143 
    144  https://nerc-arf-dan.pml.ac.uk/status/progress/
    145   - general status page
    146 
    147  https://nerc-arf-dan.pml.ac.uk/trac/ticket/631 <TICKETNO>
    148   - our notes during processing (may be technical)
    149 
    150 Also available is the PI page which will allow you to view flight data coverage and download quicklooks. This page will be updated as the data are processed and more information becomes available. Click the following link and enter username and password details as below:
    151 
    152 <INSERTWEBLINKHERE>
    153 
    154 username: <USERNAME>
    155 password: <PASSWORD>
    156 
    157 If there are any specific processing requirements, or if you have additional data that may help with the processing (e.g. GPS basestation data, ground or atmospheric measurements), please contact us directly so we can take account of this before beginning processing.
    158 
    159 If you would like any more information, please feel free to contact us at nerc-arf-processing@pml.ac.uk
    160 
    161 Regards,
    162 
    163 }}}
    164 
    165 Once the PI has been emailed commit the updated KML password file to ~arsf/live_git_repos/config_files/ (no need to push changes).
    166 [[BR]]
    167 
    168 == Basestation ==
    169 
    170 Check there is basestation data for the flight. If not, ask ops if they have some or if we need to [wiki:Processing/GNSSStatinons download it].