Changes between Version 152 and Version 153 of Procedures/NewDataArrival


Ignore:
Timestamp:
Jul 26, 2011 12:21:17 PM (8 years ago)
Author:
knpa
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Procedures/NewDataArrival

    v152 v153  
    1414== Unpacking projects ==
    1515
    16 The next stage can be manual or semi-automated. Even if the script works, go through and check it has correctly done each of the manual steps.
     16=== proj_tidy.sh  ===
    1717
    18 === Script ===
    19 * Run unpack_convert_proj.sh in the target project directory. This will change the project from the old-syle to the new whilst correcting a few errors.
     18Run proj_tidy.sh:
     19{{{
     20proj_tidy.sh -p <path_to_project>
     21}}}
    2022
    21 === Other steps ===
    22  * Prune empty directories (check these aren't stubs for later data first) (use `rmdir *` in the project directory)
    23  * Rename any capitalised subdirectories
     23This will do three things:
     24
     25- Run a series of checks which you should go through and fix if necessary[[BR]]
     26- Fix a few things automatically (permissions/common spelling mistakes)[[BR]]
     27- Provide a list of commands (if any) that will convert the project from the old style structure to the current standard. These should be checked then pasted into the terminal.
     28
     29=== Manual steps ===
     30 * Prune empty directories (check these aren't stubs for later data first)
     31 * Rename any remaining spelling mistakes in directories or files. Check eagle and hawk raws have VNIR/SWIR prefix.
    2432 * Remove any spaces, brackets or other Unix-upsetting characters in filenames
    2533   * use `find -regex '.*[^-0-9a-zA-Z/._].*' | ~arsf/usr/bin/fix_naughty_chars.py`
    2634   * gives suggested commands, but check before pasting commands!
    27  * Remove executable bit on all files
    28    * use `find -type f -exec chmod a-x {} \;`
    29  * Remove group & other write permission
    30    * `chmod go-w . -R`
    31  * Convert the .doc logsheet to .pdf
    32    * `ooffice -invisible  "macro:///Standard.Module1.SaveAsPDF(FULL_PATH_TO_FILE)"`
    3335
    34 = Verification =
     36== Verification ==
    3537
    36 == Unpack file check ==
     38=== unpack_file_check.py ===
    3739
    38 knpa: This will need to be rewritten to work with the new file structure.
     40knpa: This will need to be rewritten to work with the new file structure (or merged with proj_tidy)
    3941
    40 == proj_tidy.sh  ==
    41 
    42 knpa: ditto!
    43 
    44 == Final manual checks ==
     42=== Manual checks ===
    4543
    4644Look at the logsheet and verify that we have copies of all relevant data mentioned there.
     
    5149Check the filesizes of all data files (Eagle, Hawk, ATM, CASI) to make sure none are zero bytes (or obviously broken in some way).
    5250
    53 = Move to permanent PML data storage =
     51== Move to permanent PML data storage ==
    5452
    55 Move to appropriate location in the repository (~arsf/arsf_data/2009/flight_data/...)
     53Move to appropriate location in the repository (~arsf/arsf_data/2011/flight_data/...)
    5654  * ensure the project directory names conform to the standard - '''PROJECTCODE-YYYY_JJJxx_SITENAME''', e.g. ''GB07_07-2007_102a_Inverclyde'', ''boresight-2007_198'', etc
    5755
    58 = Tickets and tracking =
     56== Tickets and tracking ==
    5957
    60 
    61 == Status Page ==
     58=== Status Page ===
    6259
    6360Add details to the [http://arsf-dan.nerc.ac.uk/status/addflight/addflight.php processing status page]
    6461
    65 == Ticket ==
     62=== Ticket ===
    6663
    6764Raise a [http://arsf-dan.nerc.ac.uk/trac/newticket new trac ticket] (type 'flight processing') for the new data.
     
    9592}}}
    9693
    97 = Vectors =
     94== Vectors ==
    9895
    9996If the site is in the UK, check the [http://rsg.pml.ac.uk/intranet/trac/wiki/Projects/ARSF-DAN/Vector_Requests vector page] to see if we have vectors for the site in question. If not then email ARSF-Ops (Gary) and ask for them, specifying the corner points of the area covered, in OS BNG grid coordinates. Use bounding_box.py (needs to be ran in project/hyperspectral at the moment) or use the generated nextmap DEM to get the range of tiles, then do the conversion. This can take a couple of weeks before we get the vectors.
    10097
    101 = E-mail PI =
     98== E-mail PI ==
    10299
    103100Email the PI to inform them that their data has arrived for processing.  Sample text:
     
    133130}}}
    134131
    135 = Create subsidiary files =
     132== Create DEM ==
    136133
    137134Run the DEM generating script if a UK flight. `nextmapdem.sh`
    138135
    139136If the flight is outside the UK you'll need to process the LiDAR data first or use the ASTER/SRTM dem (see http://arsf-dan.nerc.ac.uk/trac/wiki/Processing/SRTMDEMs)
    140 
    141 Create the calibration sym link. This is done automatically if the unpacking scripts have been run.