Changes between Version 10 and Version 11 of Processing/proj_tidy


Ignore:
Timestamp:
Oct 9, 2014 4:07:36 PM (4 years ago)
Author:
knpa
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Processing/proj_tidy

    v10 v11  
    1818~arsf/usr/bin/proj_tidy/
    1919
    20 == Basic guide to functions ==
     20== Basic guide to most important modules ==
    2121
    2222Some functions are taken care of in the script, some things which are more complicated or are easier in python are called as separate modules.
    2323
    24 Some things, like checking the webcam images take time and only need to be done once, so I've put these under a '-e' (extended) option, which only need to be ran once, when the flight arrives and is being checked for the first time.
     24Some things, like checking the webcam images take time and only need to be done once, so I've put these under a '-e' (extended) option, which only need to be ran once, when the flight arrives and is being checked for the first time. Fixing permissions takes time too, and this is done nightly on a cron, so I also moved this to it's own option '-m'.
    2525
    26 Fixing permissions takes time too, and this is done nightly on a cron, so I also moved this to it's own option '-m'.
     26It decides if any directories are missing using check_dirs_present module which uses the folder_structure library that anch originally wrote. This maintains the project structure (somewhere) in a central place, so in theory we only need to update one place when we change structure (e.g. add or move folders around). Note you'll still have to update proj_tidy's regex file for that year if things change, as files will now be in a different place. For any missing directories it will generate a 'mkdir' command, which will appear in "Suggested Commands" at the very bottom.
    2727
    28 It decides if directories are missing or not using using folder_structure module that anch originally wrote. This maintains the project structure (somewhere) in a central place, so in theory we only need to update one place when we change structure (e.g. move folders around). Note you'll still have to update proj_tidy's regex file, as files will now be in a different place. For missing directories it will generate a 'mkdir' command, which will appear in "Suggested Commands" at the very bottom (but before the suggested commands that deal with moving, generated below).
     28There are a number of modules/procedures which need to see all the files in the directory. proj_tidy therefore creates a list of all files at the start and writes to a temp file: /tmp/proj_tidy_$timeStamp. This file can then be parsed by any modules that require it.
    2929
    30 It makes a list of all files in the flight, filters out any expected ones using the regex file, and prints the rest as "unrecognised files".
     30check_mandatory_files takes the list of all files, and uses proj_tidy's yearly regex list to make sure all required files are present.
     31
     32check_unrecognised_files takes the list of all files in the flight, filters out any expected ones using the regex file, and prints the rest as "unrecognised files".
    3133
    3234The module that generates the suggested mv commands (which will be printed below) is one of the most complex and is located under ~arsf/usr/bin/proj_tidy/move_commands.sh. This has to be kept manually updated to match the structure that ops supply to us. It's broken down into one section for each specific directory and is in the form: Check if directory exists -> Check if directory has contents -> Move contents to correct place -> rmdir the directory. It's best start from the lowest level and move upwards (e.g. do leica/ipas/raw/ then leica/ipas/ then leica/) so you're not moving anything you then want to deal with further down the script. It also does some other more complicated things to do with azgcorr style stuff - this was important when converting the projects from the pre-2011 style to the current style and making sure old and new things were kept separated, but is less important now.[[BR]]