Changes between Version 20 and Version 21 of Processing/proj_tidy

Timestamp:

Dec 1, 2014, 11:00:13 AM (9 years ago)

Author:

dap

Comment:

--

Processing/proj_tidy

@@ -44 +44 @@
 == Improvements/fixes ==
 * SERIOUS BUG - although it's set up to check pre-2011 delivies (as outlined above), it currently can't do this if that flight has been converted to the layout. Since ALL flights have now been converted to the new layout, this needs fixing before we can archive older flights. (dap, ask someone what about the change in 2011, reprocessing, and conversion of structures). 
-* Currently it determines if raw data for a sensor exists by looking if there is e.g. hyperspectral/fenix/ dir. This should be changed so it actually looks for data files e.g. FENIX*.raw
-  * This now works, except in cases where the raw images have been put in the wrong place.
-* Check if we have .hdr file and a .log file for every .raw file, and do something similar for other sensors. So basically check, if we do have a sensor as determined above, that all expected files are present.
-  * This has (mostly) been implemented for all the hyperspectral sensors and the owl sensor. This check isn't necessary for other sensors as they only have one type of raw file (.scn for LiDAR, .raw for camera).
-  * Note, there isn't currently a check to see that we have a hdr file for each raw file.
-* Check the database to get number of lines for a sensor, and check there are indeed this number
+* ~~Currently it determines if raw data for a sensor exists by looking if there is e.g. hyperspectral/fenix/ dir. This should be changed so it actually looks for data files e.g. FENIX*.raw~~
+* ~~Check if we have .hdr file and a .log file for every .raw file, and do something similar for other sensors. So basically check, if we do have a sensor as determined above, that all expected files are present.~~
+  * For the hawk and fenix hyperspectral sensors, there is a check to ensure that we have a .log and a .nav file for each .raw file in the raw directory.
+  * For the eagle sensor, there is a check to ensure that we have a .nav file for each .raw file but there is no need to check that we have a log file as these are kept in the hawk directory.
+  * For the owl sensor, there is a check to ensure we have a .nav file and a .hdr file for each .raw file but since there isn't always a .log file for each flight line (it's not compulsory), a check isn't necessary.
+  * For the camera and LiDAR sensor, there is no need to implement such a check as there is only one type of file for each of these sensors (.raw and .scn respectively).
+* [FUTURE] Check the database to get number of lines for a sensor, and check there are indeed this number
   * This can't be done yet, as there are often test lines flown by ops, which are not always entered in to the database.
 * If can't find raw data, don't search for files pertaining to those sensors
   * In the {{{Check for missing directories}}} section, hyperspectral/owl is looked for. This should be thermal/owl. This ties in with the point listed below when the project layout is built. This is due to incorrect key list in the folder_structure module.
   * Currently, {{{proj_tidy.missing_directories}}} calls {{{folder_structure.FolderStructure}}}'s constructor, which returns a full list of the directories that are expected to be in the project directory. This could be changed by either adding another function to proj_tidy.py that looks for regex instead or changing the FolderStructure constructor so that it can just return the relevant directories if necessary.
-* Check that the bandset in the hyperspectral header files is what we expect
+* ~~Check that the bandset in the hyperspectral header files is what we expect~~
   * This is done by checking if there are any .wls files in the hyperspectral sensor calibration directory (~arsf/calibration/<year>/<sensor>/) for that particular year. If it's there, it will count the number of lines in each file and add it to a Python list. It will then read in the header files and count the number of wavelengths in each one using the Python data_handler library. It will then check that the length of the header file is in the Python list of numbers of lines. If it's not, it will print a warning.
 * Some updates are needed to the regex files so it recognises some newer files
+  * I believe these are up to date. The only things that will need adding are the owl delivery directories and any owl files that are required in the main project directory. The folder_structure library will also need to be updated to include the owl directories so that they are built when unpacking.
+  * Where proj_tidy checks that all the compulsory files are present, checking for mandatory Owl files needs to be added.
 * The use folder_structure to decide what dirs are missing is creating folders we don't need in flights, like eagle and hawk dirs when there is no data for those. Should either remove these from folder_structure or (better) integrate this bit with the part that determines if sensors are present so it doesn't create these if they're not needed. It also currently puts owl in hyperspectral/owl when it should be in thermal/owl.
-* There are common errors that ops do that we should integrate into proj_tidy to make unpacking easier. For example, the RCD raws are often not put into the rcd/raw_images/ dir, so we could do this automatically.
+* ~~There are common errors that ops do that we should integrate into proj_tidy to make unpacking easier. For example, the RCD raws are often not put into the rcd/raw_images/ dir, so we could do this automatically.~~
+  * When proj_tidy checks which sensors are present, for the RCD, it looks in {{{$projpath/camera/rcd/raw_images}}} and if none are found in there, looks in {{{$projpath/camera/rcd}}}. 
+  * If they are then found in there AND the script is NOT being executed in just_checking mode, it will print an {{{mv}}} command to move the files to the correct place. 
+  * If the script IS being run in just_checking mode, it will warn the user that they are in the wrong place but won't suggest any mv commands.
+  * If the .raw files aren't found in either {{{$projpath/camera/rcd}}} or {{{$projpath/camera/rcd/raw_images}}}, it will assume there are no camera raw files present.