Changes between Version 10 and Version 11 of Processing/proj_tidy

Timestamp:

Oct 9, 2014, 4:07:36 PM (10 years ago)

Author:

knpa

Comment:

--

Processing/proj_tidy

@@ -18 +18 @@
 ~arsf/usr/bin/proj_tidy/
 
-== Basic guide to functions ==
+== Basic guide to most important modules ==
 
 Some functions are taken care of in the script, some things which are more complicated or are easier in python are called as separate modules.
 
-Some things, like checking the webcam images take time and only need to be done once, so I've put these under a '-e' (extended) option, which only need to be ran once, when the flight arrives and is being checked for the first time.
+Some things, like checking the webcam images take time and only need to be done once, so I've put these under a '-e' (extended) option, which only need to be ran once, when the flight arrives and is being checked for the first time. Fixing permissions takes time too, and this is done nightly on a cron, so I also moved this to it's own option '-m'.
 
-Fixing permissions takes time too, and this is done nightly on a cron, so I also moved this to it's own option '-m'.
+It decides if any directories are missing using check_dirs_present module which uses the folder_structure library that anch originally wrote. This maintains the project structure (somewhere) in a central place, so in theory we only need to update one place when we change structure (e.g. add or move folders around). Note you'll still have to update proj_tidy's regex file for that year if things change, as files will now be in a different place. For any missing directories it will generate a 'mkdir' command, which will appear in "Suggested Commands" at the very bottom.
 
-It decides if directories are missing or not using using folder_structure module that anch originally wrote. This maintains the project structure (somewhere) in a central place, so in theory we only need to update one place when we change structure (e.g. move folders around). Note you'll still have to update proj_tidy's regex file, as files will now be in a different place. For missing directories it will generate a 'mkdir' command, which will appear in "Suggested Commands" at the very bottom (but before the suggested commands that deal with moving, generated below).
+There are a number of modules/procedures which need to see all the files in the directory. proj_tidy therefore creates a list of all files at the start and writes to a temp file: /tmp/proj_tidy_$timeStamp. This file can then be parsed by any modules that require it.
 
-It makes a list of all files in the flight, filters out any expected ones using the regex file, and prints the rest as "unrecognised files".
+check_mandatory_files takes the list of all files, and uses proj_tidy's yearly regex list to make sure all required files are present.
+
+check_unrecognised_files takes the list of all files in the flight, filters out any expected ones using the regex file, and prints the rest as "unrecognised files".
 
 The module that generates the suggested mv commands (which will be printed below) is one of the most complex and is located under ~arsf/usr/bin/proj_tidy/move_commands.sh. This has to be kept manually updated to match the structure that ops supply to us. It's broken down into one section for each specific directory and is in the form: Check if directory exists -> Check if directory has contents -> Move contents to correct place -> rmdir the directory. It's best start from the lowest level and move upwards (e.g. do leica/ipas/raw/ then leica/ipas/ then leica/) so you're not moving anything you then want to deal with further down the script. It also does some other more complicated things to do with azgcorr style stuff - this was important when converting the projects from the pre-2011 style to the current style and making sure old and new things were kept separated, but is less important now.[[BR]]