Opened 5 years ago

Closed 3 years ago

#546 closed flight processing (fixed)

RG13/06, flight day 289/2014, Danum Valley

Reported by: dap Owned by:
Priority: alpha 5 Milestone:
Component: Processing: general Keywords:
Cc: Other processors: tec

Description (last modified by tec)

Data location: ~arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_289_Danum_Valley

Data arrived from ARSF via network transfer on 16/12/2014.

Scientific objective: Investigating the impact of humans on tropical forests.

Priority: alpha 5

PI: David Coomes

Full waveform LiDAR data are present
Thermal data are present

Some corrupt webcam images were deleted:
141016_004756_000.jpg
141016_004327_000.jpg
141016_013510_000.jpg
141016_005255_000.jpg
141016_005117_000.jpg
141016_015132_000.jpg

Sensors:

  • Fenix (requested)
  • Leica LIDAR (requested, delivered 27/04/2015)
  • Leica LIDAR FW (delivered 27/04/2015)
  • RCD (requested, delivered 03/03/2015)

Change History (58)

comment:1 Changed 5 years ago by tec

  • Other processors set to tec

Navigation Processing
Started navigation processing

SAFE_6th_ Basestation Data
Latitude4 43 30.56051
Longitude117 36 19.18563
El Height479.819m
Last edited 5 years ago by tec (previous) (diff)

comment:2 Changed 5 years ago by tec

Navigation Processing
Line 167 has nav of quality 2 for first 23 seconds of the line. Some of the lines have a second of quality 2 nav other than that its quite good.

Processed using IPAS Pro and grafnav, settings are below
Elevation Mask: 14.0 degrees
KAR Maximum dual frequency distance: 80.00 km
Code: 1.10m

comment:3 Changed 5 years ago by tec

RCD Processing
Starting RCD processing, converted images events file and converting raws to tifs now.

comment:4 Changed 5 years ago by tec

RCD Processing
Finished, awaiting delivery check.

comment:5 Changed 5 years ago by dap

Starting RCD Delivery Check

comment:6 Changed 5 years ago by dap

RCD Delivery Check complete

I would suggest adding a data quality remark about the blue 'tinge' that is around the edges of the photographs in to the read me as this is probably due to overexposure.

Once done, the dataset will be ready for delivery.

comment:7 Changed 5 years ago by tec

RCD
Finished ready for delivery.

comment:8 Changed 5 years ago by tec

LiDAR Processing
Starting LiDAR processing

comment:9 Changed 5 years ago by lah

Fenix Processing
Found SCTs

Flightline FENIX
1 0.98
2 0.98
3 1.01
4 0.95
5 1.02
6 0.94
7 1.02
8 0.98
9 0.97
10 0.97
11 1.00
12 1.06

comment:10 Changed 5 years ago by lah

Fenix
Delivery created, ready for checking.

comment:11 Changed 5 years ago by dap

Starting Fenix Delivery Check

comment:12 Changed 5 years ago by dac

Navigation Processing

Reprocessed basestation data.

Latitude4 43 30.56078
Longitude117 36 19.18509
El Height480.172 m

comment:13 Changed 5 years ago by dac

Navigation Processing

Couldn't get good enough solution in IPAS TC so processed in IPAS Pro (as was previously used for project).

comment:14 Changed 5 years ago by lah

Fenix Processing

  • Clipped .sol file as navigation has time going backwards error. Original stored as *backwards.sol.
  • Line 2 originally did not process, so added to delivery manually.

comment:15 Changed 5 years ago by lah

Fenix
New delivery created (20150220) with reprocessed nav data. Ready for checking.

comment:16 Changed 5 years ago by dap

Starting delivery check on reprocessed Fenix data delivery.

comment:17 Changed 5 years ago by dap

Fenix Delivery Check

  • Screenshots weren't named correctly - I renamed them.
  • Mapped bil and header file names for line 2 still had "_p_" in - renamed these files.

Just some minor problems left to fix in the read me and log sheet:

  • Data Quality Remarks section is a LaTeX section but Underflows and Overflows is a subsubsection - I would suggest changing the Underflows and Overflows section to a subsection.
  • The logsheet has the legacy empty page that we used to use for the Hawk and Eagle line information - probably no point in keeping it since it's just an empty table.

Please could you also delete the previous delivery if it is no longer needed.

Dataset RG13_06-289-hyperspectral-20150220 ready for delivery when the above issues have been addressed and the SWIR offset found as a result of the calibration has been investigated.

comment:18 Changed 5 years ago by lah

Fenix

  • Have removed surplus Eagle & Hawk pages from logsheet
  • Have changed Under/overflow from subsubsection to subsection in readme

comment:19 Changed 5 years ago by dac

RCD Delivery

RCD data sent to PI via FTP 03/03/2015

comment:20 follow-up: Changed 4 years ago by dac

LiDAR Processing

Found pitch and roll values. Starting classification.

Line Roll Pitch
LDR141016_000341_1 -0.006075 0.00002
LDR141016_001436_1 -0.006075 0.00002
LDR141016_002512_1 -0.006075 0.000021
LDR141016_003535_1 -0.006075 0.000022
LDR141016_004556_1 -0.006075 0.000022
LDR141016_005630_1 -0.006075 0.000022
LDR141016_010628_1 -0.006075 0.000023
LDR141016_011647_1 -0.006075 0.000023
LDR141016_012648_1 -0.006075 0.000023
LDR141016_013712_1 -0.006075 0.000023
LDR141016_014707_1 -0.006075 0.000023
LDR141016_020359_1 -0.006075 0.000023

comment:21 Changed 4 years ago by lah

Fenix
Have reprocessed with new calibration but some bad pixels remain. Added more pixels to file and reprocessing.

Last edited 4 years ago by lah (previous) (diff)

comment:22 Changed 4 years ago by dac

LiDAR Processing

Started processing full-waveform files in ALSPP. Handed over to tec.

comment:23 in reply to: ↑ 20 Changed 4 years ago by tec

LiDAR Processing

Line Roll Pitch Done FW Classified
LDR141016_000341_1 -0.006075 0.00002 Y Y
LDR141016_001436_1 -0.006075 0.00002 Y Y
LDR141016_002512_1 -0.006075 0.000021 Y Y
LDR141016_003535_1 -0.006075 0.000022 Y Y
LDR141016_004556_1 -0.006075 0.000022 Y Y
LDR141016_005630_1 -0.006075 0.000022 Y Y
LDR141016_010628_1 -0.006075 0.000023 Y Y
LDR141016_011647_1 -0.006075 0.000023 Y Y
LDR141016_012648_1 -0.006075 0.000023 Y Y
LDR141016_013712_1 -0.006075 0.000023 Y Y
LDR141016_014707_1 -0.006075 0.000023 Y Y
LDR141016_020359_1 -0.006075 0.000023 Y Y
Last edited 4 years ago by tec (previous) (diff)

comment:24 Changed 4 years ago by tec

LiDAR Processing
Running isolated points classification algorithm at 10 5 over all lines. Copying classification to FW.

Last edited 4 years ago by tec (previous) (diff)

comment:25 Changed 4 years ago by tec

LiDAR Processing
Think its ready for delivery check
Its missing the readme, but other than that it should be good.

Last edited 4 years ago by tec (previous) (diff)

comment:26 Changed 4 years ago by dap

Starting LiDAR delivery check

comment:27 Changed 4 years ago by dap

LiDAR Delivery Check

  • Logsheet
    • No need for the Eagle/Hawk page - this should be removed.
    • Might be useful for the PI to have the #XXXX line number?
  • Delivery Contents
    • Spurious temp file in ascii/ directory - probably left over from the DEM/screenshots generation? I haven't removed this as it seems to be an ASCII flight line, so unsure if needed or not.
    • Have renamed las1.2 directory to las1.0.
    • Read me document missing
    • DEM and DEM header incorrectly named - haven't renamed these as they will need to be regenerated when the LAS files have been reclassified.
    • DEM screenshot incorrectly named
  • Readme
    • Unable to check as it hasn't been generated - needs to be generated before delivery (as noted above and in comment:25)
  • LAS files
    • Line 11 is smaller in size and shorter in length - logsheet indicates that this flight line was aborted due to towering cumulus clouds, hence the shorter size so no problem here.
    • LAS 1.3 files open fine in Wave Viewer with no error messages produced.
  • Roll and Pitch Corrections
    • Good across all flight lines (though something should be noted in the read me that there is a lack of buildings making corrections difficult).
  • Classifications
    • 01 - 08 and 10 - 11 are OK
    • 11: Missed cloud at the south-westerly end of the line
    • 12: Too many points around the two big cloud patches have been falsely classified as noise.
    • DEM shows that cloud has been missed, so worth re-classifiying the above files.
  • Other notes:
    • LAS files don't have a projection when lasinfo is ran (Dan looking at this).
    • Haven't run demcompare.py yet - I will run this on the new DEMs after re-classification.

comment:28 Changed 4 years ago by lah

Fenix
Reprocessing again with bad pixels list from all flight lines (and 290's lines).

comment:29 Changed 4 years ago by lah

Fenix ready for DC
Same issue as with 290 creating readme - latex won't use thumbnail so used original mosaic screenshot

comment:30 Changed 4 years ago by lah

Fenix
Converted thumbnail using convert -density 300 -thumbnail 1024 fenix_mosaic.jpg fenix_mosaic.jpg and remade readme.

comment:31 Changed 4 years ago by tec

Fenix Delivery Check
Started DC
Done steps 1-5 on wiki, shall continue tomorrow

Last edited 4 years ago by tec (previous) (diff)

comment:32 Changed 4 years ago by tec

Fenix Delivery Check
Overflows and underflows are wrong I think, flight line 1 looks fine, line 2 has underflows from 530-535 and 590-621. I think the autoQC script's threshold needs to be tweaked.

Other than that it looks ok.

comment:33 Changed 4 years ago by tec

Fenix Delivery Check
Looks ok, ready to deliver

comment:34 Changed 4 years ago by tec

LiDAR
Finished, ready for DC.

comment:35 follow-up: Changed 4 years ago by dap

LiDAR Delivery Check

  • Logsheet
    • No need for the Eagle/Hawk page - this still needs to be removed (copying the one from the hyperspectral delivery would suffice).
  • Delivery Contents
    • Converted ASCII files to DOS line endings
  • Readme
    • Mention in the data quality remarks the lack of features for roll/pitch error correction.
    • Mention in the data quality remarks the cloud coverage
    • Elevation differences are in the wrong units - convert these to centimetres.
  • Classifications
    • Much better, all cloud seems to have been classified.
  • Other notes:
    • LAS files still don't have a projection when lasinfo is ran but this is not a reason to justify holding back the delivery. I have checked the projection used in the ALSPP config file, which shows UTM50N was used.
    • demcompare.py returned a result of greater than 12.
    • DEM screenshot doesn't open in eog but opens in Windows image viewer so the delivery is not worth holding back over this.

Just corrections to make to the read me and logsheet then it'll be ready to go.

comment:36 in reply to: ↑ 35 Changed 4 years ago by dac

Replying to dap:

  • DEM screenshot doesn't open in eog but opens in Windows image viewer so the delivery is not worth holding back over this.

Fixed screenshot using:

gdal_translate -of JPEG -outsize 80% 80% RG13_06-2014_289-dem.jpg /tmp/RG13_06-2014_289-dem.jpg

comment:37 Changed 4 years ago by tec

Delivery
Copied LiDAR to FTP arsf1

comment:38 Changed 4 years ago by dac

Fenix Processing

Changed status back to processing, as waiting on calibration before delivery.

comment:39 Changed 4 years ago by tec

  • Description modified (diff)

Delivery
LiDAR -> FTP 27/04/2015

comment:40 Changed 4 years ago by lah

Fenix
Remade delivery with new calibration data. Just needs approval on data quality report.

comment:41 Changed 4 years ago by dap

Fenix Delivery Check

Starting on delivery with new files.

comment:42 Changed 4 years ago by dap

Fenix DC complete

Looks fine, I made a few corrections:

  • Fixed typo in the name of the fenix mosaic screenshot
  • Replaced instances of 'Danum Valley' in all xml files using
    find . -name "*.xml" -exec sed -i 's/Danum Valley/SAFE Area/g' {} \;
    

from delivery directory

Please change the read me site location before delivery.

comment:43 Changed 4 years ago by dap

Running zip_mapped.sh

comment:44 Changed 4 years ago by lah

Fenix
Changed Readme and renamed logsheet to SAFE Area.
Other deliveries probably have Danum Valley in xml files...

comment:45 Changed 4 years ago by dap

Fenix

zip_mapped.sh has finished, now rsyncing to arsf1 on FTP server.

comment:46 Changed 4 years ago by dap

Fenix

rsync complete

comment:47 Changed 4 years ago by lah

PI notified of delivery 05/05/2015 (Fenix, Lidar & RCD)

comment:48 Changed 4 years ago by dac

Fenix

Replaced wavelengths in header files with ones from updated calibration.

comment:49 Changed 4 years ago by dac

Fenix

Processed files in APL - creating delivery.

comment:50 Changed 4 years ago by dac

Fenix

Delivery created. Ready for delivery check.

comment:51 Changed 4 years ago by lah

Fenix DC

  • renamed screenshots
  • just looking at the project.xml (and line xml) file, it's weird that it says "This is provided with the data:true" Why is the ":true" bit needed?
  • I believe the lidar dem was used to process this data. Is that ok?
  • Should probably mention cloud in data quality section of report, as was mentioned in previous delivery.
  • All spectra look good.
  • Under/overflows still fine.

Just a few things to check before delivering (with updated data quality report)

comment:52 Changed 4 years ago by dac

Fenix

  • The data:true isn't in the XML file, it is added by the stylesheet. It shouldn't be there - will make a note to fix. As the actual XML metadata is OK should be fine to deliver though.
  • Have added sentence on cloud cover from previous readme.
  • The files were processed with a LiDAR DSM by mistake - not sure why this was left in the config file (might have been me testing). Started reprocessing lines with ASTER for consistency and will replace mapped files and metadata in delivery.

comment:53 Changed 4 years ago by dac

Fenix

  • Reprocessed lines using ASTER DEM and replaced files in delivery.
  • Check spectra against 6S - look good.
  • Started zipping files.

comment:54 Changed 4 years ago by dac

Fenix

Delivered to PI via FTP (arsf1) 21/10/2015

comment:55 Changed 3 years ago by lah

Owl tidying

  • Removed processing/owl/* as only used in script testing, not operational processing.
  • Renamed David_Coomes_Safe_0135 to OWL289-14-7 to match the rest of the files. This line appears to have been aborted according to the logsheet so was probably previously ignored. There is also no header file so it cannot be processed. Timestamp also confirms that this was collected after line 6.

comment:56 Changed 3 years ago by dac

Archiving

Checked and ready for upload to NEODC.

comment:57 Changed 3 years ago by dac

Archiving

Uploaded to NEODC.

comment:58 Changed 3 years ago by asm

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.