Changes between Version 3 and Version 4 of Processing/proj_tidy


Ignore:
Timestamp:
Oct 9, 2014 12:35:31 PM (5 years ago)
Author:
knpa
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Processing/proj_tidy

    v3 v4  
    11= proj_tidy.sh =
     2
     3== Intro ==
    24
    35proj_tidy (project tidy) is a tool to keep the arsf flight directories in a neat and standard data structure with correct layout and filenames. It also runs a multitude of checks to flag any problems with the data.
    46
    57It also serves as our "unpacking script", it prints a list of commands that you should use to convert it from the format we get it from ARSF-OPS to our desired standard.
     8
     9== Location ==
    610
    711It lives here:[[BR]]
     
    1115~arsf/usr/bin/proj_tidy/
    1216
    13 Also present there, are regex/templates. These are designed to hold filename/folder structure information for each year. This is because we use proj_tidy to archive flights in previous years. We want to make sure these are standard for a given year but we don't care if they change in-between years (which things do). We therefore need a record of past conventions. The regex files are used by proj_tidy, the templates are the same thing but in a form where things like flightDay can be readily inserted to build correct paths/names. I think this was so proj_tidy can give suggested changes so you can copy and paste without having to correct it manually, but this has not yet been implemented. For pre-2011 an older version of proj_tidy is ran (proj_tidy_old.sh), this is because the project structure/filename conventions were originally hardcoded into the script. When we changed to APL in 2011, I decided to rewrite proj_tidy in the new, flexible year format, and kept the old one around for 2010 stuff. If you run the regular proj_tidy on something pre-2011, it will call proj_tidy_old instead.
     17
     18== Basic guide to functions ==
    1419
    1520
    1621
     22== Changes between years ==
     23
     24Also present in the module directory, are regex/templates. These are designed to hold filename/folder structure information for each year. This is because we use proj_tidy to archive flights in previous years. We want to make sure these are standard for a given year but we don't care if they change in-between years (which things do). We therefore need a record of past conventions. The regex files are used by proj_tidy, the templates are the same thing but in a form where things like flightDay can be readily inserted to build correct paths/names. I think this was so proj_tidy can give suggested changes so you can copy and paste without having to correct it manually, but this has not yet been implemented. For pre-2011 an older version of proj_tidy is ran (proj_tidy_old.sh), this is because the project structure/filename conventions were originally hardcoded into the script. When we changed to APL in 2011, I decided to rewrite proj_tidy in the new, flexible year format, and kept the old one around for 2010 stuff. If you run the regular proj_tidy on something pre-2011, it will call proj_tidy_old instead.
     25
     26== Improvements/fixes ==
     27
     28* Currently it determines if raw data for a sensor exists by looking if there is e.g. hyperspectral/fenix/ dir. This should be changed so it actually looks for data files e.g. FENIX*.raw
     29
     30
     31