Changes between Version 147 and Version 148 of Procedures/NewDataArrival


Ignore:
Timestamp:
May 26, 2011 3:51:10 PM (8 years ago)
Author:
knpa
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Procedures/NewDataArrival

    v147 v148  
    2121The next stage can be manual or semi-automated. Even if the script works, go through and check it has correctly done each of the manual steps.
    2222
    23 === Semi-scripted method ===
    24   * In the directory above the new project directories (eg '.../flight_data/unpacking/') run 'unpack_folder_structure.py' [[BR]] NOTE: If you want to just run this on just one project (and as we usually only unpack one at a time) then you will need to move this project into a temporary containing folder when running this script (will change script at some point so can specify a single project).
    25     * By default, it runs safely in a dry-run mode and will output the commands it will run to the terminal. Check these look ok.
    26     * If happy, either re-run with --execute (and optionally --verbose)
    27     * Each project directory should be re-formatted to the current standard
     23=== Script ===
     24* Run unpack_convert_proj.sh in the target project directory. This will change the project from the old-syle to the new whilst correcting a few errors.
    2825
    29 === Non-scripting  method ===
     26=== Other steps ===
    3027 * Prune empty directories (check these aren't stubs for later data first) (use `rmdir *` in the project directory)
    3128 * Rename any capitalised subdirectories
    32  * Move DCALM* files to applanix/raw
    3329 * Remove any spaces, brackets or other Unix-upsetting characters in filenames
    3430   * use `find -regex '.*[^-0-9a-zA-Z/._].*' | ~arsf/usr/bin/fix_naughty_chars.py`
     
    4036 * Convert the .doc logsheet to .pdf
    4137   * `ooffice -invisible  "macro:///Standard.Module1.SaveAsPDF(FULL_PATH_TO_FILE)"`
    42  * Rename 'rinex' to 'basestation' in top directory (if present). Move all basestation data to here, then create a symlink to it from inside both applanix and leica/ipas.
    4338
    4439= Verification =
     
    4641== Unpack file check ==
    4742
    48 In each project directory run 'unpack_file_check.py -l <admin/logsheet.doc(.txt)>'
    49   * This will convert .doc logsheet to .txt, or use the .txt if one available. NOTE to convert to .doc requires ooffice macro
    50   * Will then do various checks of data against logsheet as listed below. Information will be output to terminal. Important (error) messages are printed again at the end.
    51     * Check file sizes against a 'suitable' size and also against header file (Eagle + Hawk)
    52     * Check number of files against logsheet
    53     * Check number of logsheets
    54     * Check GPS start/stop times in header file (Eagle + Hawk)
    55     * Check .raw, .nav, .log, .hdr for each Eagle + Hawk line
    56     * Check for nav-sync issues - THIS IS BASED ON A FALSE ASSUMPTION AND PROBABLY IS USELESS
     43knpa: This will need to be rewritten to work with the new file structure.
    5744
    5845== proj_tidy.sh  ==
    5946
    60 In the base of the project directory run proj_tidy:[[BR]]
    61 `proj_tidy.sh -p ./ -e`[[BR]]
    62 This will check that the directory structures look sensible and tell you where they don't. Correct anything significant that gets flagged up.
     47knpa: ditto!
    6348
    6449== Final manual checks ==
     
    6752 * In some cases, the flight crew may fly two projects back-to-back but enter all the data onto a single logsheet.  If so, you may need to split the project directory into two, particularly if there's a large time gap (navigation needs separate processing) or the PIs are different (different delivery addresses/tracking).  If you do need to split a project, ensure both copies have copies of common files (logsheets, rinex, etc), but that non-common files are not duplicated (ie. don't include hawk data for part 1 in part 2..).  Also note in the ticket what was done for tracking purposes.
    6853
    69 Verify the details on the logsheet (esp. PI) by calling/emailing ARSF-Ops (probably Gary) - the application form and logsheet are not reliable enough, nor do they track any changes in PI over the lifetime of the application.
     54Verify the details on the logsheet (esp. PI) by calling/emailing ARSF-Ops (probably Gary) - the application form and logsheet are not reliable enough, nor do they track any changes in PI over the lifetime of the application. Get the alpha grading as well.
    7055
    7156Check the filesizes of all data files (Eagle, Hawk, ATM, CASI) to make sure none are zero bytes (or obviously broken in some way).
    7257
    73 = Move to permanent PML data storage =
    74 
    75 Move to appropriate location in the repository (~arsf/arsf_data/2009/flight_data/...)
    76   * ensure the project directory names conform to the standard - '''PROJECTCODE-YYYY_JJJxx_SITENAME''', e.g. ''GB07_07-2007_102a_Inverclyde'', ''boresight-2007_198'', etc
    77 
    78 = Create subsidiary files =
    79 
    80 Run the DEM generating script if a UK flight. `nextmapdem.sh`
    81 
    82 If the flight is outside the UK you'll need to process the LiDAR data first or use the ASTER/SRTM dem (see http://arsf-dan.nerc.ac.uk/trac/wiki/Processing/SRTMDEMs)
    83 
    84 Create the calibration sym link. This is done automatically if the unpacking scripts have been run.
    8558
    8659= Tickets and tracking =
     
    160133
    161134}}}
     135
     136= Move to permanent PML data storage =
     137
     138Move to appropriate location in the repository (~arsf/arsf_data/2009/flight_data/...)
     139  * ensure the project directory names conform to the standard - '''PROJECTCODE-YYYY_JJJxx_SITENAME''', e.g. ''GB07_07-2007_102a_Inverclyde'', ''boresight-2007_198'', etc
     140
     141= Create subsidiary files =
     142
     143Run the DEM generating script if a UK flight. `nextmapdem.sh`
     144
     145If the flight is outside the UK you'll need to process the LiDAR data first or use the ASTER/SRTM dem (see http://arsf-dan.nerc.ac.uk/trac/wiki/Processing/SRTMDEMs)
     146
     147Create the calibration sym link. This is done automatically if the unpacking scripts have been run.