MA14/11, flight day 306/2014, Danum Valley

[dap]

Data location: ~arsf/arsf_data/2014/flight_data/malaysia/MA14_11-2014_306_Danum_Valley

Data arrived from ARSF via network transfer on 06/01/2015.

Scientific objective: estimate canopy height and structure from lidar and airborne SAR, to improve estimates of forest biomass, as well as use hyperspectral data to map tree species diversity.

Priority: Currently unfunded

PI: Mark Cutler, David Coomes (deliver processed data to both PIs)

OWL data were collected and are present.

This project directory contains one flightline from the aborted flight 306b. Be aware of this while processing. When downloaded from thelma, there were no basestation data included in the project.

Sensors:

• Fenix (requested)
• LiDAR (requested, highest priority)
• FW LiDAR (requested)
• RCD (not requested but flown anyway)

[dap]

David Coomes at the University of Cambridge should also be delivered the processed data from this flight as they shared this site between both projects. (See email below from Gary):

(email trimmed)
David Coomes data deliveries should also include those for MA14-11 (Mark Cutler) as they shared this site between both projects
(email trimmed)

[tec]

LiDAR
Started LiDAR

[tec]

Navigation
Lat 4 57 48.36361
Lon 117 48 10.32523
Ell 209.234
Antenna Height 1.46

[tec]

LiDAR

Line	Roll	Pitch	Notes	Classified	Classified FW
031124	-0.00615	-0.00235		Y
031826	-0.00615	-0.00235		Y
032456	-0.00595	-0.00235		Y
033143	-0.00635	-0.00215		Y
033830	-0.00595	-0.00235		Y
034515	-0.00595	-0.00295		Y
035209	-0.00615	-0.00195		Y
035835	-0.00595	-0.00315		Y
040534	-0.00575	-0.00255	Crossline	Y
041013	-0.00615	-0.00235	Line back at top (pair with 031124)	Y
042416	-0.00615	-0.00235	Single line	Y

[stgo]

Hyperspectral
Starting hyperspectral processing

[tec]

RCD
Started

[stgo]

Hyperspectral
SCT values:

Flightline	FENIX
1	0.90
2	1.00
3	1.00
4	1.00
5	0.90
6	1.01
7	0.80
8	1.20
9	0.90
10	0.97

[tec]

RCD
Images are rubbish, all are unusable so no delivery for this.

[stgo]

Hyperspectral
Ready for delivery check.

[asm]

Hyperspectral DC

Starting delivery check

[asm]

Found problem with underflow table in Readme file. Apparently the script is not filling the table properly. Found values manually:
Underflows:

-line 01: 504-524, 592-622
-line 02: 499-529, 600-622
-line 03: 497-536, 579-622
-line 04: 496-545, 576-622
-line 05: 424-440, 495-622
-line 06: 498-540, 580-622
-line 07: 420-439, 493-622
-line 08: 432-433(small),496-545, 576-622
-line 09: 428-439, 495-544, 576-622
-line 10: 424-438, 494-545, 577-622

Overflow: (Don't fully understand this section, someone please review)

-line 04: 505-525, 605-622
-line 08: 321-324

Everything else seems correct.
This is my first delivery checking, second checking should be needed.

[lah]

2nd Fenix DC

• data quality report is not most recent.(needs manually replacing)
• QC tables should not be in delivery
• There were *aux.xml files in the mapped folder from viewing with Tuiview. These always need removing at the end of delivery check.
• fodis directory needs removing
• some underflows are missing from readme table. I spot an error with the autoQC script! I'll look into this. (fenix overflows are unreliable in fastQC until the fenix binning problem is sorted. autoQC should be ok using values from the mask. Once reprocessed the overflows should show up in fastQC)
• scts look ok

I haven't made any changes as a new delivery will need to be created once reprocessed with fenix errors corrected.

[lah]

Replying to lah:

2nd Fenix DC

• data quality report is not most recent.(needs manually replacing)
• QC tables should not be in delivery
• There were *aux.xml files in the mapped folder from viewing with Tuiview. These always need removing at the end of delivery check.
• fodis directory needs removing
• some underflows are missing from readme table. I spot an error with the autoQC script! I'll look into this. (fenix overflows are unreliable in fastQC until the fenix binning problem is sorted. autoQC should be ok using values from the mask. Once reprocessed the overflows should show up in fastQC)
• scts look ok

I haven't made any changes as a new delivery will need to be created once reprocessed with fenix errors corrected.

ok reran autoQC and output is fine. Looks like you've just copied stuff into QC_under.txt

[tec]

LiDAR
Issue with DEM screenshot, waiting for dac to reappear

[lah]

Replying to lah:

Replying to lah:

2nd Fenix DC

• data quality report is not most recent.(needs manually replacing)
• QC tables should not be in delivery
• There were *aux.xml files in the mapped folder from viewing with Tuiview. These always need removing at the end of delivery check.
• fodis directory needs removing
• some underflows are missing from readme table. I spot an error with the autoQC script! I'll look into this. (fenix overflows are unreliable in fastQC until the fenix binning problem is sorted. autoQC should be ok using values from the mask. Once reprocessed the overflows should show up in fastQC)
• scts look ok

I haven't made any changes as a new delivery will need to be created once reprocessed with fenix errors corrected.

ok reran autoQC and output is fine. Looks like you've just copied stuff into QC_under.txt

found error in autoqc (this is called rather than the almost identical autoQC) and fixed, so readme over/under flow table creates correctly now.

[dac]

Replying to tec:

LiDAR
Issue with DEM screenshot, waiting for dac to reappear

Problem with screenshot is bounds of lidar are larger than hyperspectral, Recreating DEM with --lidar_bounds flag.

[dac]

LiDAR
Recreated DEMs and screenshot to lidar bounds.

[tec]

LiDAR
Thanks dac, DEM looks good now, LiDAR ready for DC

[asm]

Lidar DC
Started Delivery Check for Lidar

[asm]

Lidar DC
Not sure if flight line 001 should be delivered. There are only 10 flightlines on the logsheet and the summary says: "This project directory contains one flightline from the aborted flight 306b. Be aware of this while processing. When downloaded from thelma, there were no basestation data included in the project"

Done all the others steps for DC, everything else seems OK.

This is my first Lidar DC, second checking should be needed. But it should wait until check the strange flight line.

[asm]

Lidar DC
Getting rid of flightline 001 and re-creating delivery. Should need to re-create screenshots, logsheet and readme.

[asm]

Lidar DC
Need to add the second PI to the readme and logsheet.

[asm]

Lidar DC
Created new screenshots and readme-files. Everything is done except the new logsheet (waiting to solve problems with generateLogsheet.py)

[asm]

Lidar DC
Everything solved now. Ready for second DC.

[dap]

Beginning 2nd LiDAR delivery check

[dac]

LiDAR DC

Taking over second DC (needed as this was Aser's first DC) from Dale.

[dac]

LiDAR DC

Generally looks OK, couple of things to fix:

• There is still an empty table with elevation differences in. Just need to include sentance saying it couldn't be calculated
• There are three DEMs. Need to remove / rename 'MA14_11-2014_306-lidar_ASTER.dem'
• demcompare.py reports the mean is 5.2 m

[asm]

LiDAR
Fixed the ReadMe and deleted the DEM. Please re-check but it should be OK now.

[dac]

LiDAR DC

Looks good - ready to deliver.

[asm]

Dispatching Data to PI
Started. Delivery methods both FTP (for PI Coomes) and hard drive (for PI Cutler).

[asm]

LIDAR data has been delivered via hard drive USB. (3 August 2015)

[asm]

Dispatching Lidar Data to PI
Also set the delivery method for FTP and sent a mail to both PIs.
Finished Lidar Delivery

[stgo]

Started Hyperspectral reprocessing

[stgo]

Started Fenix reprocessing

Something weird happened on the grid with calibration file. APL reports that 13 bands were consistent with calibration, re running to identify if this was a one off thing from Friday night.

[stgo]

Fenix reprocessing

Ready for delivery check.

This delivery needs to be thoroughly checked for:

• correct lines included
• correct linenames from logsheet
• data quality comments
• spectra shifts

[asm]

Fenix reprocessing DC

Started
-Removed fodis directory.
-Added logsheet.
-All flightlines has correct name compared with logsheet and all lines from logsheet are included.

[asm]

Fenix reprocessing DC

Need to add to the readme file:
-Both PIs names: Mark Cutler and David Coomes
-Some data quality remarks about clouds on flightline 06 and 10.

[asm]

Fenix reprocessing DC

-Checked green vegetation spectra with Py6S. Good match for all
flightlines.
-Tested all tif files created from apl commands.

[lah]

Fenix DC - spectra
Checked spectra of a few lines and no major problems. There is a persistent spike in the mask region for this flight, not unlike many other flights. The spectral shift is within 5 nm compared to previous data (2014 169). SWIR detector looks fairly stable from line 1.

[lah]

Fenix DC
Starting final checks and corrections.

[lah]

Fenix DC

• Replaced data quality report with most recent
• renamed screenshots
• rechecked all bands of all lines in level 1 files using fastQC, and all look good.
• Recreated readme with 2 PIs and mentioned more cloud on line 10.

Zipping up mapped files and then will be ready to deliver.
Have deleted June delivery in processing folder.

[lah]

Fenix

Delivered to PIs on 28/10/15 by ftp (arsf15).

[dac]

Uploaded Fenix and LiDAR to FTP server (arsf13) for download by David Coomes - was cc'd in previous delivery but didn't download before slot was reused.

[dac]

Archiving

Started uploading to NEODC.

[dac]

Archiving

Data available from NEODC. Still under embargo so not yet accessible.