RG13/06, flight day 289/2014, Danum Valley

[dap]

Data location: ~arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_289_Danum_Valley

Data arrived from ARSF via network transfer on 16/12/2014.

Scientific objective: Investigating the impact of humans on tropical forests.

Priority: alpha 5

PI: David Coomes

Full waveform LiDAR data are present
Thermal data are present

Some corrupt webcam images were deleted:
141016_004756_000.jpg
141016_004327_000.jpg
141016_013510_000.jpg
141016_005255_000.jpg
141016_005117_000.jpg
141016_015132_000.jpg

Sensors:

• Fenix (requested)
• Leica LIDAR (requested, delivered 27/04/2015)
• Leica LIDAR FW (delivered 27/04/2015)
• RCD (requested, delivered 03/03/2015)

[tec]

Navigation Processing
Started navigation processing

SAFE_6th_ Basestation Data
Latitude	4 43 30.56051
Longitude	117 36 19.18563
El Height	479.819m

[tec]

Navigation Processing
Line 167 has nav of quality 2 for first 23 seconds of the line. Some of the lines have a second of quality 2 nav other than that its quite good.

Processed using IPAS Pro and grafnav, settings are below
Elevation Mask: 14.0 degrees
KAR Maximum dual frequency distance: 80.00 km
Code: 1.10m

[tec]

RCD Processing
Starting RCD processing, converted images events file and converting raws to tifs now.

[tec]

RCD Processing
Finished, awaiting delivery check.

[dap]

Starting RCD Delivery Check

[dap]

RCD Delivery Check complete

I would suggest adding a data quality remark about the blue 'tinge' that is around the edges of the photographs in to the read me as this is probably due to overexposure.

Once done, the dataset will be ready for delivery.

[tec]

RCD
Finished ready for delivery.

[tec]

LiDAR Processing
Starting LiDAR processing

[lah]

Fenix Processing
Found SCTs

Flightline	FENIX
1	0.98
2	0.98
3	1.01
4	0.95
5	1.02
6	0.94
7	1.02
8	0.98
9	0.97
10	0.97
11	1.00
12	1.06

[lah]

Fenix
Delivery created, ready for checking.

[dap]

Starting Fenix Delivery Check

[dac]

Navigation Processing

Reprocessed basestation data.

Latitude	4 43 30.56078
Longitude	117 36 19.18509
El Height	480.172 m

[dac]

Navigation Processing

Couldn't get good enough solution in IPAS TC so processed in IPAS Pro (as was previously used for project).

[lah]

Fenix Processing

• Clipped .sol file as navigation has time going backwards error. Original stored as *backwards.sol.
• Line 2 originally did not process, so added to delivery manually.

[lah]

Fenix
New delivery created (20150220) with reprocessed nav data. Ready for checking.

[dap]

Starting delivery check on reprocessed Fenix data delivery.

[dap]

Fenix Delivery Check

• Screenshots weren't named correctly - I renamed them.
• Mapped bil and header file names for line 2 still had "_p_" in - renamed these files.

Just some minor problems left to fix in the read me and log sheet:

• Data Quality Remarks section is a LaTeX section but Underflows and Overflows is a subsubsection - I would suggest changing the Underflows and Overflows section to a subsection.
• The logsheet has the legacy empty page that we used to use for the Hawk and Eagle line information - probably no point in keeping it since it's just an empty table.

Please could you also delete the previous delivery if it is no longer needed.

Dataset RG13_06-289-hyperspectral-20150220 ready for delivery when the above issues have been addressed and the SWIR offset found as a result of the calibration has been investigated.

[lah]

Fenix

• Have removed surplus Eagle & Hawk pages from logsheet
• Have changed Under/overflow from subsubsection to subsection in readme

[dac]

RCD Delivery

RCD data sent to PI via FTP 03/03/2015

[dac]

LiDAR Processing

Found pitch and roll values. Starting classification.

Line	Roll	Pitch
LDR141016_000341_1	-0.006075	0.00002
LDR141016_001436_1	-0.006075	0.00002
LDR141016_002512_1	-0.006075	0.000021
LDR141016_003535_1	-0.006075	0.000022
LDR141016_004556_1	-0.006075	0.000022
LDR141016_005630_1	-0.006075	0.000022
LDR141016_010628_1	-0.006075	0.000023
LDR141016_011647_1	-0.006075	0.000023
LDR141016_012648_1	-0.006075	0.000023
LDR141016_013712_1	-0.006075	0.000023
LDR141016_014707_1	-0.006075	0.000023
LDR141016_020359_1	-0.006075	0.000023

[lah]

Fenix
Have reprocessed with new calibration but some bad pixels remain. Added more pixels to file and reprocessing.

[dac]

LiDAR Processing

Started processing full-waveform files in ALSPP. Handed over to tec.

[tec]

LiDAR Processing

Line	Roll	Pitch	Done FW	Classified
LDR141016_000341_1	-0.006075	0.00002	Y	Y
LDR141016_001436_1	-0.006075	0.00002	Y	Y
LDR141016_002512_1	-0.006075	0.000021	Y	Y
LDR141016_003535_1	-0.006075	0.000022	Y	Y
LDR141016_004556_1	-0.006075	0.000022	Y	Y
LDR141016_005630_1	-0.006075	0.000022	Y	Y
LDR141016_010628_1	-0.006075	0.000023	Y	Y
LDR141016_011647_1	-0.006075	0.000023	Y	Y
LDR141016_012648_1	-0.006075	0.000023	Y	Y
LDR141016_013712_1	-0.006075	0.000023	Y	Y
LDR141016_014707_1	-0.006075	0.000023	Y	Y
LDR141016_020359_1	-0.006075	0.000023	Y	Y

[tec]

LiDAR Processing
Running isolated points classification algorithm at 10 5 over all lines. Copying classification to FW.

[tec]

LiDAR Processing
Think its ready for delivery check
Its missing the readme, but other than that it should be good.

[dap]

Starting LiDAR delivery check

[dap]

LiDAR Delivery Check

• Logsheet
  • No need for the Eagle/Hawk page - this should be removed.
  • Might be useful for the PI to have the #XXXX line number?
• Delivery Contents
  • Spurious temp file in ascii/ directory - probably left over from the DEM/screenshots generation? I haven't removed this as it seems to be an ASCII flight line, so unsure if needed or not.
  • Have renamed las1.2 directory to las1.0.
  • Read me document missing
  • DEM and DEM header incorrectly named - haven't renamed these as they will need to be regenerated when the LAS files have been reclassified.
  • DEM screenshot incorrectly named
• Readme
  • Unable to check as it hasn't been generated - needs to be generated before delivery (as noted above and in comment:25)
• LAS files
  • Line 11 is smaller in size and shorter in length - logsheet indicates that this flight line was aborted due to towering cumulus clouds, hence the shorter size so no problem here.
  • LAS 1.3 files open fine in Wave Viewer with no error messages produced.
• Roll and Pitch Corrections
  • Good across all flight lines (though something should be noted in the read me that there is a lack of buildings making corrections difficult).
• Classifications
  • 01 - 08 and 10 - 11 are OK
  • 11: Missed cloud at the south-westerly end of the line
  • 12: Too many points around the two big cloud patches have been falsely classified as noise.
  • DEM shows that cloud has been missed, so worth re-classifiying the above files.
• Other notes:
  • LAS files don't have a projection when lasinfo is ran (Dan looking at this).
  • Haven't run demcompare.py yet - I will run this on the new DEMs after re-classification.

[lah]

Fenix
Reprocessing again with bad pixels list from all flight lines (and 290's lines).

[lah]

Fenix ready for DC
Same issue as with 290 creating readme - latex won't use thumbnail so used original mosaic screenshot

[lah]

Fenix
Converted thumbnail using convert -density 300 -thumbnail 1024 fenix_mosaic.jpg fenix_mosaic.jpg and remade readme.

[tec]

Fenix Delivery Check
Started DC
Done steps 1-5 on wiki, shall continue tomorrow

[tec]

Fenix Delivery Check
Overflows and underflows are wrong I think, flight line 1 looks fine, line 2 has underflows from 530-535 and 590-621. I think the autoQC script's threshold needs to be tweaked.

Other than that it looks ok.

[tec]

Fenix Delivery Check
Looks ok, ready to deliver

[tec]

LiDAR
Finished, ready for DC.

[dap]

LiDAR Delivery Check

• Logsheet
  • No need for the Eagle/Hawk page - this still needs to be removed (copying the one from the hyperspectral delivery would suffice).
• Delivery Contents
  • Converted ASCII files to DOS line endings
• Readme
  • Mention in the data quality remarks the lack of features for roll/pitch error correction.
  • Mention in the data quality remarks the cloud coverage
  • Elevation differences are in the wrong units - convert these to centimetres.
• Classifications
  • Much better, all cloud seems to have been classified.
• Other notes:
  • LAS files still don't have a projection when lasinfo is ran but this is not a reason to justify holding back the delivery. I have checked the projection used in the ALSPP config file, which shows UTM50N was used.
  • demcompare.py returned a result of greater than 12.
  • DEM screenshot doesn't open in eog but opens in Windows image viewer so the delivery is not worth holding back over this.

Just corrections to make to the read me and logsheet then it'll be ready to go.

[dac]

Replying to dap:

• DEM screenshot doesn't open in eog but opens in Windows image viewer so the delivery is not worth holding back over this.

Fixed screenshot using:

gdal_translate -of JPEG -outsize 80% 80% RG13_06-2014_289-dem.jpg /tmp/RG13_06-2014_289-dem.jpg

[tec]

Delivery
Copied LiDAR to FTP arsf1

[dac]

Fenix Processing

Changed status back to processing, as waiting on calibration before delivery.

[tec]

Delivery
LiDAR -> FTP 27/04/2015

[lah]

Fenix
Remade delivery with new calibration data. Just needs approval on data quality report.

[dap]

Fenix Delivery Check

Starting on delivery with new files.

[dap]

Fenix DC complete

Looks fine, I made a few corrections:

• Fixed typo in the name of the fenix mosaic screenshot
• Replaced instances of 'Danum Valley' in all xml files using

  find . -name "*.xml" -exec sed -i 's/Danum Valley/SAFE Area/g' {} \;
  

from delivery directory

Please change the read me site location before delivery.

[dap]

Running zip_mapped.sh

[lah]

Fenix
Changed Readme and renamed logsheet to SAFE Area.
Other deliveries probably have Danum Valley in xml files...

[dap]

Fenix

zip_mapped.sh has finished, now rsyncing to arsf1 on FTP server.

[dap]

Fenix

rsync complete

[lah]

PI notified of delivery 05/05/2015 (Fenix, Lidar & RCD)

[dac]

Fenix

Replaced wavelengths in header files with ones from updated calibration.

[dac]

Fenix

Processed files in APL - creating delivery.

[dac]

Fenix

Delivery created. Ready for delivery check.

[lah]

Fenix DC

• renamed screenshots
• just looking at the project.xml (and line xml) file, it's weird that it says "This is provided with the data:true" Why is the ":true" bit needed?
• I believe the lidar dem was used to process this data. Is that ok?
• Should probably mention cloud in data quality section of report, as was mentioned in previous delivery.
• All spectra look good.
• Under/overflows still fine.

Just a few things to check before delivering (with updated data quality report)

[dac]

Fenix

• The data:true isn't in the XML file, it is added by the stylesheet. It shouldn't be there - will make a note to fix. As the actual XML metadata is OK should be fine to deliver though.
• Have added sentence on cloud cover from previous readme.
• The files were processed with a LiDAR DSM by mistake - not sure why this was left in the config file (might have been me testing). Started reprocessing lines with ASTER for consistency and will replace mapped files and metadata in delivery.

[dac]

Fenix

• Reprocessed lines using ASTER DEM and replaced files in delivery.
• Check spectra against 6S - look good.
• Started zipping files.

[dac]

Fenix

Delivered to PI via FTP (arsf1) 21/10/2015

[lah]

Owl tidying

• Removed processing/owl/* as only used in script testing, not operational processing.
• Renamed David_Coomes_Safe_0135 to OWL289-14-7 to match the rest of the files. This line appears to have been aborted according to the logsheet so was probably previously ignored. There is also no header file so it cannot be processed. Timestamp also confirms that this was collected after line 6.

[dac]

Archiving

Checked and ready for upload to NEODC.

[dac]

Archiving

Uploaded to NEODC.