Opened 4 years ago

Closed 2 years ago

#563 closed flight processing (fixed)

MA14/11, flight day 306/2014, Danum Valley

Reported by: dap Owned by:
Priority: whenever Milestone:
Component: Processing: general Keywords:
Cc: Other processors:

Description (last modified by dap)

Data location: ~arsf/arsf_data/2014/flight_data/malaysia/MA14_11-2014_306_Danum_Valley

Data arrived from ARSF via network transfer on 06/01/2015.

Scientific objective: estimate canopy height and structure from lidar and airborne SAR, to improve estimates of forest biomass, as well as use hyperspectral data to map tree species diversity.

Priority: Currently unfunded

PI: Mark Cutler, David Coomes (deliver processed data to both PIs)

OWL data were collected and are present.

This project directory contains one flightline from the aborted flight 306b. Be aware of this while processing. When downloaded from thelma, there were no basestation data included in the project.

Sensors:

  • Fenix (requested)
  • LiDAR (requested, highest priority)
  • FW LiDAR (requested)
  • RCD (not requested but flown anyway)

Change History (47)

comment:1 Changed 4 years ago by dap

  • Description modified (diff)

comment:2 Changed 4 years ago by dap

David Coomes at the University of Cambridge should also be delivered the processed data from this flight as they shared this site between both projects. (See email below from Gary):


(email trimmed)
David Coomes data deliveries should also include those for MA14-11 (Mark Cutler) as they shared this site between both projects
(email trimmed)

Last edited 4 years ago by dap (previous) (diff)

comment:3 Changed 4 years ago by dap

  • Description modified (diff)

comment:4 Changed 4 years ago by tec

LiDAR
Started LiDAR

comment:5 Changed 4 years ago by tec

Navigation
Lat 4 57 48.36361
Lon 117 48 10.32523
Ell 209.234
Antenna Height 1.46

comment:6 Changed 4 years ago by tec

LiDAR

Line Roll Pitch Notes Classified Classified FW
031124 -0.00615 -0.00235 Y
031826 -0.00615 -0.00235 Y
032456 -0.00595 -0.00235 Y
033143 -0.00635 -0.00215 Y
033830 -0.00595 -0.00235 Y
034515 -0.00595 -0.00295 Y
035209 -0.00615 -0.00195 Y
035835 -0.00595 -0.00315 Y
040534 -0.00575 -0.00255 Crossline Y
041013 -0.00615 -0.00235 Line back at top (pair with 031124) Y
042416 -0.00615 -0.00235 Single line Y
Last edited 4 years ago by tec (previous) (diff)

comment:7 Changed 4 years ago by stgo

Hyperspectral
Starting hyperspectral processing

comment:8 Changed 4 years ago by tec

RCD
Started

comment:9 Changed 4 years ago by stgo

Hyperspectral
SCT values:

Flightline FENIX
1 0.90
2 1.00
3 1.00
4 1.00
5 0.90
6 1.01
7 0.80
8 1.20
9 0.90
10 0.97

comment:10 Changed 4 years ago by tec

RCD
Images are rubbish, all are unusable so no delivery for this.

Last edited 4 years ago by tec (previous) (diff)

comment:11 Changed 4 years ago by stgo

Hyperspectral
Ready for delivery check.

comment:12 Changed 4 years ago by asm

Hyperspectral DC

Starting delivery check

comment:13 Changed 4 years ago by asm

Found problem with underflow table in Readme file. Apparently the script is not filling the table properly. Found values manually:
Underflows:

-line 01: 504-524, 592-622
-line 02: 499-529, 600-622
-line 03: 497-536, 579-622
-line 04: 496-545, 576-622
-line 05: 424-440, 495-622
-line 06: 498-540, 580-622
-line 07: 420-439, 493-622
-line 08: 432-433(small),496-545, 576-622
-line 09: 428-439, 495-544, 576-622
-line 10: 424-438, 494-545, 577-622

Overflow: (Don't fully understand this section, someone please review)

-line 04: 505-525, 605-622
-line 08: 321-324

Everything else seems correct.
This is my first delivery checking, second checking should be needed.

comment:14 follow-up: Changed 4 years ago by lah

2nd Fenix DC

  • data quality report is not most recent.(needs manually replacing)
  • QC tables should not be in delivery
  • There were *aux.xml files in the mapped folder from viewing with Tuiview. These always need removing at the end of delivery check.
  • fodis directory needs removing
  • some underflows are missing from readme table. I spot an error with the autoQC script! I'll look into this. (fenix overflows are unreliable in fastQC until the fenix binning problem is sorted. autoQC should be ok using values from the mask. Once reprocessed the overflows should show up in fastQC)
  • scts look ok

I haven't made any changes as a new delivery will need to be created once reprocessed with fenix errors corrected.

comment:15 in reply to: ↑ 14 ; follow-up: Changed 4 years ago by lah

Replying to lah:

2nd Fenix DC

  • data quality report is not most recent.(needs manually replacing)
  • QC tables should not be in delivery
  • There were *aux.xml files in the mapped folder from viewing with Tuiview. These always need removing at the end of delivery check.
  • fodis directory needs removing
  • some underflows are missing from readme table. I spot an error with the autoQC script! I'll look into this. (fenix overflows are unreliable in fastQC until the fenix binning problem is sorted. autoQC should be ok using values from the mask. Once reprocessed the overflows should show up in fastQC)
  • scts look ok

I haven't made any changes as a new delivery will need to be created once reprocessed with fenix errors corrected.

ok reran autoQC and output is fine. Looks like you've just copied stuff into QC_under.txt

comment:16 follow-up: Changed 4 years ago by tec

LiDAR
Issue with DEM screenshot, waiting for dac to reappear

comment:17 in reply to: ↑ 15 Changed 4 years ago by lah

Replying to lah:

Replying to lah:

2nd Fenix DC

  • data quality report is not most recent.(needs manually replacing)
  • QC tables should not be in delivery
  • There were *aux.xml files in the mapped folder from viewing with Tuiview. These always need removing at the end of delivery check.
  • fodis directory needs removing
  • some underflows are missing from readme table. I spot an error with the autoQC script! I'll look into this. (fenix overflows are unreliable in fastQC until the fenix binning problem is sorted. autoQC should be ok using values from the mask. Once reprocessed the overflows should show up in fastQC)
  • scts look ok

I haven't made any changes as a new delivery will need to be created once reprocessed with fenix errors corrected.

ok reran autoQC and output is fine. Looks like you've just copied stuff into QC_under.txt

found error in autoqc (this is called rather than the almost identical autoQC) and fixed, so readme over/under flow table creates correctly now.

comment:18 in reply to: ↑ 16 Changed 4 years ago by dac

Replying to tec:

LiDAR
Issue with DEM screenshot, waiting for dac to reappear

Problem with screenshot is bounds of lidar are larger than hyperspectral, Recreating DEM with --lidar_bounds flag.

comment:19 Changed 4 years ago by dac

LiDAR
Recreated DEMs and screenshot to lidar bounds.

comment:20 Changed 4 years ago by tec

LiDAR
Thanks dac, DEM looks good now, LiDAR ready for DC

Last edited 4 years ago by tec (previous) (diff)

comment:21 Changed 4 years ago by asm

Lidar DC
Started Delivery Check for Lidar

comment:22 Changed 4 years ago by asm

Lidar DC
Not sure if flight line 001 should be delivered. There are only 10 flightlines on the logsheet and the summary says: "This project directory contains one flightline from the aborted flight 306b. Be aware of this while processing. When downloaded from thelma, there were no basestation data included in the project"

Done all the others steps for DC, everything else seems OK.

This is my first Lidar DC, second checking should be needed. But it should wait until check the strange flight line.

Last edited 4 years ago by asm (previous) (diff)

comment:23 Changed 4 years ago by asm

Lidar DC
Getting rid of flightline 001 and re-creating delivery. Should need to re-create screenshots, logsheet and readme.

Last edited 4 years ago by asm (previous) (diff)

comment:24 Changed 4 years ago by asm

Lidar DC
Need to add the second PI to the readme and logsheet.

Last edited 4 years ago by asm (previous) (diff)

comment:25 Changed 4 years ago by asm

Lidar DC
Created new screenshots and readme-files. Everything is done except the new logsheet (waiting to solve problems with generateLogsheet.py)

Last edited 4 years ago by asm (previous) (diff)

comment:26 Changed 4 years ago by asm

Lidar DC
Everything solved now. Ready for second DC.

comment:27 Changed 4 years ago by dap

Beginning 2nd LiDAR delivery check

comment:28 Changed 4 years ago by dac

LiDAR DC

Taking over second DC (needed as this was Aser's first DC) from Dale.

comment:29 Changed 4 years ago by dac

LiDAR DC

Generally looks OK, couple of things to fix:

  • There is still an empty table with elevation differences in. Just need to include sentance saying it couldn't be calculated
  • There are three DEMs. Need to remove / rename 'MA14_11-2014_306-lidar_ASTER.dem'
  • demcompare.py reports the mean is 5.2 m

comment:30 Changed 4 years ago by asm

LiDAR
Fixed the ReadMe and deleted the DEM. Please re-check but it should be OK now.

comment:31 Changed 4 years ago by dac

LiDAR DC

Looks good - ready to deliver.

comment:32 Changed 4 years ago by asm

Dispatching Data to PI
Started. Delivery methods both FTP (for PI Coomes) and hard drive (for PI Cutler).

comment:33 Changed 4 years ago by asm

LIDAR data has been delivered via hard drive USB. (3 August 2015)

comment:34 Changed 4 years ago by asm

Dispatching Lidar Data to PI
Also set the delivery method for FTP and sent a mail to both PIs.
Finished Lidar Delivery

comment:35 Changed 3 years ago by stgo

Started Hyperspectral reprocessing

comment:36 Changed 3 years ago by stgo

Started Fenix reprocessing

Something weird happened on the grid with calibration file. APL reports that 13 bands were consistent with calibration, re running to identify if this was a one off thing from Friday night.

comment:37 Changed 3 years ago by stgo

Fenix reprocessing

Ready for delivery check.

This delivery needs to be thoroughly checked for:

  • correct lines included
  • correct linenames from logsheet
  • data quality comments
  • spectra shifts

comment:38 Changed 3 years ago by asm

Fenix reprocessing DC

Started
-Removed fodis directory.
-Added logsheet.
-All flightlines has correct name compared with logsheet and all lines from logsheet are included.

comment:39 Changed 3 years ago by asm

Fenix reprocessing DC

Need to add to the readme file:
-Both PIs names: Mark Cutler and David Coomes
-Some data quality remarks about clouds on flightline 06 and 10.

Last edited 3 years ago by asm (previous) (diff)

comment:40 Changed 3 years ago by asm

Fenix reprocessing DC

-Checked green vegetation spectra with Py6S. Good match for all
flightlines.
-Tested all tif files created from apl commands.

comment:41 Changed 3 years ago by lah

Fenix DC - spectra
Checked spectra of a few lines and no major problems. There is a persistent spike in the mask region for this flight, not unlike many other flights. The spectral shift is within 5 nm compared to previous data (2014 169). SWIR detector looks fairly stable from line 1.

comment:42 Changed 3 years ago by lah

Fenix DC
Starting final checks and corrections.

comment:43 Changed 3 years ago by lah

Fenix DC

  • Replaced data quality report with most recent
  • renamed screenshots
  • rechecked all bands of all lines in level 1 files using fastQC, and all look good.
  • Recreated readme with 2 PIs and mentioned more cloud on line 10.

Zipping up mapped files and then will be ready to deliver.
Have deleted June delivery in processing folder.

comment:44 Changed 3 years ago by lah

Fenix

Delivered to PIs on 28/10/15 by ftp (arsf15).

comment:45 Changed 3 years ago by dac

Uploaded Fenix and LiDAR to FTP server (arsf13) for download by David Coomes - was cc'd in previous delivery but didn't download before slot was reused.

comment:46 Changed 3 years ago by dac

Archiving

Started uploading to NEODC.

comment:47 Changed 2 years ago by dac

  • Resolution set to fixed
  • Status changed from new to closed

Archiving

Data available from NEODC. Still under embargo so not yet accessible.

Note: See TracTickets for help on using tickets.