Changes between Version 152 and Version 153 of Procedures/NewDataArrival
- Timestamp:
- Jul 26, 2011, 12:21:17 PM (13 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
Procedures/NewDataArrival
v152 v153 14 14 == Unpacking projects == 15 15 16 The next stage can be manual or semi-automated. Even if the script works, go through and check it has correctly done each of the manual steps. 16 === proj_tidy.sh === 17 17 18 === Script === 19 * Run unpack_convert_proj.sh in the target project directory. This will change the project from the old-syle to the new whilst correcting a few errors. 18 Run proj_tidy.sh: 19 {{{ 20 proj_tidy.sh -p <path_to_project> 21 }}} 20 22 21 === Other steps === 22 * Prune empty directories (check these aren't stubs for later data first) (use `rmdir *` in the project directory) 23 * Rename any capitalised subdirectories 23 This will do three things: 24 25 - Run a series of checks which you should go through and fix if necessary[[BR]] 26 - Fix a few things automatically (permissions/common spelling mistakes)[[BR]] 27 - Provide a list of commands (if any) that will convert the project from the old style structure to the current standard. These should be checked then pasted into the terminal. 28 29 === Manual steps === 30 * Prune empty directories (check these aren't stubs for later data first) 31 * Rename any remaining spelling mistakes in directories or files. Check eagle and hawk raws have VNIR/SWIR prefix. 24 32 * Remove any spaces, brackets or other Unix-upsetting characters in filenames 25 33 * use `find -regex '.*[^-0-9a-zA-Z/._].*' | ~arsf/usr/bin/fix_naughty_chars.py` 26 34 * gives suggested commands, but check before pasting commands! 27 * Remove executable bit on all files28 * use `find -type f -exec chmod a-x {} \;`29 * Remove group & other write permission30 * `chmod go-w . -R`31 * Convert the .doc logsheet to .pdf32 * `ooffice -invisible "macro:///Standard.Module1.SaveAsPDF(FULL_PATH_TO_FILE)"`33 35 34 = Verification=36 == Verification == 35 37 36 == Unpack file check==38 === unpack_file_check.py === 37 39 38 knpa: This will need to be rewritten to work with the new file structure .40 knpa: This will need to be rewritten to work with the new file structure (or merged with proj_tidy) 39 41 40 == proj_tidy.sh == 41 42 knpa: ditto! 43 44 == Final manual checks == 42 === Manual checks === 45 43 46 44 Look at the logsheet and verify that we have copies of all relevant data mentioned there. … … 51 49 Check the filesizes of all data files (Eagle, Hawk, ATM, CASI) to make sure none are zero bytes (or obviously broken in some way). 52 50 53 = Move to permanent PML data storage=51 == Move to permanent PML data storage == 54 52 55 Move to appropriate location in the repository (~arsf/arsf_data/20 09/flight_data/...)53 Move to appropriate location in the repository (~arsf/arsf_data/2011/flight_data/...) 56 54 * ensure the project directory names conform to the standard - '''PROJECTCODE-YYYY_JJJxx_SITENAME''', e.g. ''GB07_07-2007_102a_Inverclyde'', ''boresight-2007_198'', etc 57 55 58 = Tickets and tracking=56 == Tickets and tracking == 59 57 60 61 == Status Page == 58 === Status Page === 62 59 63 60 Add details to the [http://arsf-dan.nerc.ac.uk/status/addflight/addflight.php processing status page] 64 61 65 == Ticket==62 === Ticket === 66 63 67 64 Raise a [http://arsf-dan.nerc.ac.uk/trac/newticket new trac ticket] (type 'flight processing') for the new data. … … 95 92 }}} 96 93 97 = Vectors=94 == Vectors == 98 95 99 96 If the site is in the UK, check the [http://rsg.pml.ac.uk/intranet/trac/wiki/Projects/ARSF-DAN/Vector_Requests vector page] to see if we have vectors for the site in question. If not then email ARSF-Ops (Gary) and ask for them, specifying the corner points of the area covered, in OS BNG grid coordinates. Use bounding_box.py (needs to be ran in project/hyperspectral at the moment) or use the generated nextmap DEM to get the range of tiles, then do the conversion. This can take a couple of weeks before we get the vectors. 100 97 101 = E-mail PI=98 == E-mail PI == 102 99 103 100 Email the PI to inform them that their data has arrived for processing. Sample text: … … 133 130 }}} 134 131 135 = Create subsidiary files=132 == Create DEM == 136 133 137 134 Run the DEM generating script if a UK flight. `nextmapdem.sh` 138 135 139 136 If the flight is outside the UK you'll need to process the LiDAR data first or use the ASTER/SRTM dem (see http://arsf-dan.nerc.ac.uk/trac/wiki/Processing/SRTMDEMs) 140 141 Create the calibration sym link. This is done automatically if the unpacking scripts have been run.