Version 78 (modified by dap, 10 years ago) (diff)

--

Delivery Checking

Once a dataset has been prepared for delivery, a second person should go over it and perform the checks below.

Note: if this is your first time doing a delivery check, please get a third (experienced) person to perform a second check when you're done

For Hyperspectral Data

  1. Verify we have the correct PI (check against application, call Gloucester if unsure)
  2. Check against the logsheet that all data that should be present is.
    • Number of files and sensible file sizes
  3. Check that all other folders and contents are present
    • Only the PDF read me file should be present (no .tex, .out, .aux, .log files)
    • Check that there are ASCII view vector files in the sensor FOV dirs for each sensor (multiple ones if different binning has been used)
    • Note that there should not be a flightlines/fodis directory for 2013 deliveries
  4. Check for typos, etc in the documentation.
  5. Ensure the text files are Windows compatible (use the 'file' command on the .txt files: they should be ASCII with CRLF terminators)
  6. Test out the readme level 3 processing command for all of the level 1 files, run check_apl_cmd -c <path to hyp_genreadme config file> in delivery directory, outputs tif files to tmp_cmd_check directory, check tif files in gtviewer
  7. Check each of the screenshots
  8. Run proj_tidy.sh -c -p <path_to_project>. Check if any of the problems listed apply to your delivery.
  9. Run check_bil_size.py -d <deliverylocation>. This will list any BIL files that don't match the size computed from the respective HDR file.
  10. All level 1 files should be looked at visually using fastQC.
    1. If any pixels appear to be constantly bad (c.f. 2009? dust on lens problems) then return to processor for re-processing calibration using a list of additional bad pixels to mask (see aplcal -qcfailures)
    2. In this case additional information should be added into read me to explain why they are being masked
  11. Check all of the lev3 mapped files open and look OK (don't need to go through all the bands)
    1. Make sure that all bands have been mapped (not just 3)
  12. Check the projection used is correct - especially the correct UTM zone for the data
  13. If the delivery is ready to deliver, update the dataset on the ​Processing status page to "Ready to deliver".
  14. If everything is ready to deliver, i.e. no more changing of ReadMe files, then in the sensor delivery directory run the script: zip_mapped.sh. This process should zip all of the mapped lines in the flightlines/mapped directory. Check the output.
  15. Add a comment on the ticket saying what problems there are / things that need resolving / things that you have resolved yourself / whether the data is ready to deliver.

For LIDAR Data

  1. If there is any full waveform data to be checked, start copying this over to a Windows machine with Wave Viewer installed (to avoid delays in the delivery check later).
  2. Verify we have the correct PI (check against application, call Gloucester if unsure)
  3. Check against the logsheet that all data that should be present is.
  4. Check that all other folders and contents are present.
    • Only the PDF read me file should be present (no .tex, .out, .aux, .log files)
  5. Check for typos, etc in the documentation.
  6. Ensure the text files are Windows compatible (use the 'file' command on the .txt files: they should be ASCII with CRLF terminators)
  7. Check all the LIDAR files open/load, look OK and fit together horizontally and vertically. Either:
    • Use lag to view the data
    • Or run lidar_intensity.sh?. You need to generate these yourself from the ascii files, you can't just look at already created DEM/Screenshots.
  8. Run the script check_ascii_lidar which will read in the ascii files and check all points have the correct number of records (9), report the min/max of the time/easting/northings and the number of points classified as noise.
  9. Look at a couple of lines to check that obvious noise has been classified.
  10. Check the DEM contents look OK
    • Check DEM in envi
    • check DEM header resolution is the same as in the Read_Me file - trim the ReadMe hdr resolution if in metres and unnecessary precision (e.g. 2.000124342 metres should be 2.0)
  11. Check the lidar separation from original ASTER data
    • run demcompare.py --lidar /ascii/folder/ -d lidar_patched_dem.dem -l UKBNG/UTM
    • measurements are in metres, check they aren't too big (>2 metres for UK data, >8 metres for UTM) on the masked statistics.
  12. Check the projection used is correct - especially the correct UTM zone for the data
  13. Check that the coverage of the DEM is sufficient for processing the hyperspectral data.
  14. Check the screenshots look OK
  15. Run proj_tidy.sh -c -p <path_to_project>. Check if any of the problems listed apply to your delivery.
  16. Add a comment on the ticket saying what problems there are / things that need resolving / things that you have resolved yourself / whether the data is ready to deliver
  17. If the delivery is ready to deliver, update the dataset on the ​Processing status page to "Ready to deliver".
    Full waveform deliveries only
  18. Check some of the full waveform LAS files. Either:
    • Use WaveViewer (under Leica on the windows machine), scroll through a few files and check that the wave form exists and the peaks follow the discreet points.
    • Test the files for waveform data using laszip -i lasfile.LAS -waveform -o /sensible/output/file.laz, problems with the wave form data will cause an error otherwise a .laz and .wdz file will be created.
  19. If fw_extractions folder is present: check the *_extractions.txt file - ensure all the listed folders/files are present. Check a couple of the ascii files to ensure they are readable. Check the *_extractions.jpg looks ok.

For Digital Photography Data

  1. Verify we have the correct PI (check against application, call Gloucester if unsure)
  2. Check against the readme that all data that should be present is.
  3. Check that all other folders and contents are present.
    • Only the PDF read me file should be present (no .tex, .out, .aux, .log files)
  4. Check for typos, etc in the documentation.
  5. View all of the thumbnail images. Look for any that are very over/under exposed or not relevant to the project. This could include: very bright images, very dark images, images of the instrument bay door, images not overlapping with any of the Eagle/Hawk data
  6. Check one (or more) of the photographs for tagging information: exiftool photographs/FILENAME
  7. View one of the photographs to check it opens OK.
  8. Check file sizes of photographs look reasonable (224M) and there are no _tmp photo files.
  9. Run proj_tidy.sh -c -p <path_to_project>. Check if any of the problems listed apply to your delivery.
  10. Open the kml file in googleearth and check it looks good and that thumbnails are displayed when the push pins are clicked.
    • If thumbnails are not visible open the kml file in a text editor. Check image source is the thumbnails directory and that image filenames are correct.
  11. Check filenames correspond. If more than one project has been flown check that julian day has the letter after it (e.g. 298b). This will be in photograph, thumbnail and eventfile directories. Also check filenames in the eventfile have the julian day letter included.
  12. Add a comment on the ticket saying what problems there are / things that need resolving / things that you have resolved yourself / whether the data is ready to deliver.
  13. If the delivery is ready to deliver, update the dataset on the ​Processing status page to "Ready to deliver".