| | 11 | == Tickets and tracking == |
| | 12 | |
| | 13 | === Status Page === |
| | 14 | |
| | 15 | Add details to the [https://nerc-arf-dan.pml.ac.uk/status/addflight/ processing status page]. Under 'Data location' use the full path to the project and do NOT include server name (e.g use /users/rsg/arsf/... and not /data/visuyuan/...). |
| | 16 | |
| | 17 | === Ticket === |
| | 18 | |
| | 19 | Raise a [https://nerc-arf-dan.pml.ac.uk/trac/newticket new trac ticket] (type 'flight processing') for the new data. |
| | 20 | * Ticket summary should be of the form '''EU10/03, flight day 172/2011, Dennys Wood''' |
| | 21 | * Add short version of scientific purpose to guide processing (check NERC-ARF application) |
| | 22 | * Note arrival time of data |
| | 23 | * Set priority of ticket from project grading (try the grades subpages on [wiki:Projects]) |
| | 24 | * Note any specific comments that might help with processing |
| | 25 | * Owner should be blank |
| | 26 | * Verify the sensors that were requested in the application (primary) and those that weren't (secondary). Note in the ticket which these were. |
| | 27 | * Ticket body should contain: |
| | 28 | {{{ |
| | 29 | Data location: ~arsf/arsf_data/2011/flight_data/..... FILL IN |
| | 30 | |
| | 31 | Data arrived from NERC-ARF via SATA disk LETTER OR network transfer on DATE. |
| | 32 | |
| | 33 | Scientific objective: FILL IN FROM APPLICATION (just enough to guide processing choices) |
| | 34 | |
| | 35 | Priority: FILL IN FROM APPLICATION/WIKI PAGE (e.g. alpha-5 low), set ticket priority appropriately |
| | 36 | |
| | 37 | PI: A. N. Other |
| | 38 | EUFAR Project ID: |
| | 39 | CAFAM code: |
| | 40 | |
| | 41 | Any other notes.. |
| | 42 | |
| | 43 | = Sensors: = |
| | 44 | * Fenix (requested/not requested, flown/not flown) |
| | 45 | * Leica LIDAR (requested/not requested, flown/not flown) |
| | 46 | * OWL (requested/not requested, flown/not flown) |
| | 47 | * RCD (requested/not requested, flown/not flown) |
| | 48 | }}} |
| | 49 | [[BR]] |
| | 50 | |
| | 51 | == Create KML overview == |
| | 52 | |
| | 53 | When happy the directory is unpacked correctly run [wiki:Processing/makekmloverview make_kmloverview.py]. This will generate the web page and quicklooks of the data that are made available to the user. It will also generate a password which should be included in the email below. Check the username and password works by commenting out the following line: |
| | 54 | |
| | 55 | == E-mail PI == |
| | 56 | |
| | 57 | Email the PI to inform them that their data has arrived to the Data Analysis Node for processing. Sample text: |
| | 58 | * fill in the 7 fields: <PI_NAME>, <PROJECT>, <EUFAR ID> <TICKET_NO>, <DATE_OF_ARRIVAL>, <USERNAME>, <PASSWORD>, <INSERTWEBLINKHERE> |
| | 59 | * the date of arrival should be when the disks arrived or when the download begun |
| | 60 | * the username and password are available in the .htaccess file in processing/kml_overview or ~arsf/usr/share/kmlpasswords.csv. Note that make_kmloverview.py will need to have been run to create a password first unless the project code already exists for previous data. |
| | 61 | * cc to nerc-arf-processing |
| | 62 | * set reply-to to nerc-arf-processing |
| | 63 | * subject: `NERC-ARF data arrival notification (<PROJECT> [<EUFAR ID>])` |
| | 64 | {{{ |
| | 65 | Dear <PI_NAME>, |
| | 66 | |
| | 67 | This is a notification that your NERC-ARF data for <PROJECT> [<EUFAR ID>], flown on |
| | 68 | <CALENDAR_DAY(S)>, are at the NERC-ARF Data Analysis Node for processing (data |
| | 69 | received from NERC-ARF Operations on <DATE_OF_ARRIVAL>). |
| | 70 | |
| | 71 | We aim to deliver as quickly as possible. You can follow progress at the following webpages: |
| | 72 | |
| | 73 | https://nerc-arf-dan.pml.ac.uk/status/progress/ |
| | 74 | - general status page |
| | 75 | |
| | 76 | https://nerc-arf-dan.pml.ac.uk/trac/ticket/631 <TICKETNO> |
| | 77 | - our notes during processing (may be technical) |
| | 78 | |
| | 79 | Also available is the PI page which will allow you to view flight data coverage and download quicklooks. This page will be updated as the data are processed and more information becomes available. Click the following link and enter username and password details as below: |
| | 80 | |
| | 81 | <INSERTWEBLINKHERE> |
| | 82 | |
| | 83 | username: <USERNAME> |
| | 84 | password: <PASSWORD> |
| | 85 | |
| | 86 | If there are any specific processing requirements, or if you have additional data that may help with the processing (e.g. GPS basestation data, ground or atmospheric measurements), please contact us directly so we can take account of this before beginning processing. |
| | 87 | |
| | 88 | If you would like any more information, please feel free to contact us at nerc-arf-processing@pml.ac.uk |
| | 89 | |
| | 90 | Regards, |
| | 91 | |
| | 92 | }}} |
| | 93 | |
| | 94 | Once the PI has been emailed commit the updated KML password file to ~arsf/live_git_repos/config_files/ (no need to push changes). |
| | 95 | [[BR]] |
| | 96 | |
| | 97 | == Basestation == |
| | 98 | |
| | 99 | Check there is basestation data for the flight. If not, ask ops if they have some or if we need to [wiki:Processing/GNSSStatinons download it]. |
| 70 | | |
| 71 | | == Tickets and tracking == |
| 72 | | |
| 73 | | === Status Page === |
| 74 | | |
| 75 | | Add details to the [https://nerc-arf-dan.pml.ac.uk/status/addflight/ processing status page]. Under 'Data location' use the full path to the project and do NOT include server name (e.g use /users/rsg/arsf/... and not /data/visuyuan/...). |
| 76 | | |
| 77 | | === Ticket === |
| 78 | | |
| 79 | | Raise a [https://nerc-arf-dan.pml.ac.uk/trac/newticket new trac ticket] (type 'flight processing') for the new data. |
| 80 | | * Ticket summary should be of the form '''EU10/03, flight day 172/2011, Dennys Wood''' |
| 81 | | * Add short version of scientific purpose to guide processing (check NERC-ARF application) |
| 82 | | * Note arrival time of data |
| 83 | | * Set priority of ticket from project grading (try the grades subpages on [wiki:Projects]) |
| 84 | | * Note any specific comments that might help with processing |
| 85 | | * Owner should be blank |
| 86 | | * Verify the sensors that were requested in the application (primary) and those that weren't (secondary). Note in the ticket which these were. |
| 87 | | * Ticket body should contain: |
| 88 | | {{{ |
| 89 | | Data location: ~arsf/arsf_data/2011/flight_data/..... FILL IN |
| 90 | | |
| 91 | | Data arrived from NERC-ARF via SATA disk LETTER OR network transfer on DATE. |
| 92 | | |
| 93 | | Scientific objective: FILL IN FROM APPLICATION (just enough to guide processing choices) |
| 94 | | |
| 95 | | Priority: FILL IN FROM APPLICATION/WIKI PAGE (e.g. alpha-5 low), set ticket priority appropriately |
| 96 | | |
| 97 | | PI: A. N. Other |
| 98 | | EUFAR Project ID: |
| 99 | | CAFAM code: |
| 100 | | |
| 101 | | Any other notes.. |
| 102 | | |
| 103 | | = Sensors: = |
| 104 | | * Fenix (requested/not requested, flown/not flown) |
| 105 | | * Leica LIDAR (requested/not requested, flown/not flown) |
| 106 | | * OWL (requested/not requested, flown/not flown) |
| 107 | | * RCD (requested/not requested, flown/not flown) |
| 108 | | }}} |
| 109 | | [[BR]] |
| 110 | | |
| 111 | | == Create KML overview == |
| 112 | | |
| 113 | | When happy the directory is unpacked correctly run [wiki:Processing/makekmloverview make_kmloverview.py]. This will generate the web page and quicklooks of the data that are made available to the user. It will also generate a password which should be included in the email below. Check the username and password works by commenting out the following line: |
| 114 | | {{{ |
| 115 | | Allow from env=internal_proxy |
| 116 | | }}} |
| 117 | | from processing/kml_overview/.htaccess as this means a password isn't required for internal access. Remember to uncomment once you have finished. |
| 118 | | [[BR]] |
| 119 | | |
| 120 | | == Vectors == |
| 121 | | |
| 122 | | If the site is in the UK, check the [wiki:vectors vector page] to see if we have vectors for the site in question. If not, then email NERC-ARF-Ops and ask for them specifying the corner points of the area covered in OS BNG grid coordinates. Use bounding_box.py (needs to be run in the project directory) or use the generated nextmap DEM (below) to get the range of tiles, then do the conversion. It can take a couple of weeks before we get the vectors. |
| 123 | | [[BR]] |
| 124 | | |
| 125 | | == E-mail PI == |
| 126 | | |
| 127 | | Email the PI to inform them that their data has arrived to the Data Analysis Node for processing. Sample text: |
| 128 | | * fill in the 7 fields: <PI_NAME>, <PROJECT>, <EUFAR ID> <TICKET_NO>, <DATE_OF_ARRIVAL>, <USERNAME>, <PASSWORD>, <INSERTWEBLINKHERE> |
| 129 | | * the date of arrival should be when the disks arrived or when the download begun |
| 130 | | * the username and password are available in the .htaccess file in processing/kml_overview or ~arsf/usr/share/kmlpasswords.csv. Note that make_kmloverview.py will need to have been run to create a password first unless the project code already exists for previous data. |
| 131 | | * cc to nerc-arf-processing |
| 132 | | * also cc to !neodc@rl.ac.uk |
| 133 | | * set reply-to to nerc-arf-processing |
| 134 | | * subject: `NERC-ARF data arrival notification (<PROJECT> [<EUFAR ID>])` |
| 135 | | {{{ |
| 136 | | Dear <PI_NAME>, |
| 137 | | |
| 138 | | This is a notification that your NERC-ARF data for <PROJECT> [<EUFAR ID>], flown on |
| 139 | | <CALENDAR_DAY(S)>, are at the NERC-ARF Data Analysis Node for processing (data |
| 140 | | received from NERC-ARF Operations on <DATE_OF_ARRIVAL>). |
| 141 | | |
| 142 | | We aim to deliver as quickly as possible. You can follow progress at the following webpages: |
| 143 | | |
| 144 | | https://nerc-arf-dan.pml.ac.uk/status/progress/ |
| 145 | | - general status page |
| 146 | | |
| 147 | | https://nerc-arf-dan.pml.ac.uk/trac/ticket/631 <TICKETNO> |
| 148 | | - our notes during processing (may be technical) |
| 149 | | |
| 150 | | Also available is the PI page which will allow you to view flight data coverage and download quicklooks. This page will be updated as the data are processed and more information becomes available. Click the following link and enter username and password details as below: |
| 151 | | |
| 152 | | <INSERTWEBLINKHERE> |
| 153 | | |
| 154 | | username: <USERNAME> |
| 155 | | password: <PASSWORD> |
| 156 | | |
| 157 | | If there are any specific processing requirements, or if you have additional data that may help with the processing (e.g. GPS basestation data, ground or atmospheric measurements), please contact us directly so we can take account of this before beginning processing. |
| 158 | | |
| 159 | | If you would like any more information, please feel free to contact us at nerc-arf-processing@pml.ac.uk |
| 160 | | |
| 161 | | Regards, |
| 162 | | |
| 163 | | }}} |
| 164 | | |
| 165 | | Once the PI has been emailed commit the updated KML password file to ~arsf/live_git_repos/config_files/ (no need to push changes). |
| 166 | | [[BR]] |
| 167 | | |
| 168 | | == Basestation == |
| 169 | | |
| 170 | | Check there is basestation data for the flight. If not, ask ops if they have some or if we need to [wiki:Processing/GNSSStatinons download it]. |