Opened 10 years ago
Closed 8 years ago
#563 closed flight processing (fixed)
MA14/11, flight day 306/2014, Danum Valley
Reported by: | dap | Owned by: | |
---|---|---|---|
Priority: | whenever | Milestone: | |
Component: | Processing: general | Keywords: | |
Cc: | Other processors: |
Description (last modified by dap)
Data location: ~arsf/arsf_data/2014/flight_data/malaysia/MA14_11-2014_306_Danum_Valley
Data arrived from ARSF via network transfer on 06/01/2015.
Scientific objective: estimate canopy height and structure from lidar and airborne SAR, to improve estimates of forest biomass, as well as use hyperspectral data to map tree species diversity.
Priority: Currently unfunded
PI: Mark Cutler, David Coomes (deliver processed data to both PIs)
OWL data were collected and are present.
This project directory contains one flightline from the aborted flight 306b. Be aware of this while processing. When downloaded from thelma, there were no basestation data included in the project.
Sensors:
- Fenix (requested)
- LiDAR (requested, highest priority)
- FW LiDAR (requested)
- RCD (not requested but flown anyway)
Change History (47)
comment:1 Changed 10 years ago by dap
- Description modified (diff)
comment:2 Changed 9 years ago by dap
comment:3 Changed 9 years ago by dap
- Description modified (diff)
comment:4 Changed 9 years ago by tec
LiDAR
Started LiDAR
comment:5 Changed 9 years ago by tec
Navigation
Lat 4 57 48.36361
Lon 117 48 10.32523
Ell 209.234
Antenna Height 1.46
comment:6 Changed 9 years ago by tec
LiDAR
Line | Roll | Pitch | Notes | Classified | Classified FW |
---|---|---|---|---|---|
031124 | -0.00615 | -0.00235 | Y | ||
031826 | -0.00615 | -0.00235 | Y | ||
032456 | -0.00595 | -0.00235 | Y | ||
033143 | -0.00635 | -0.00215 | Y | ||
033830 | -0.00595 | -0.00235 | Y | ||
034515 | -0.00595 | -0.00295 | Y | ||
035209 | -0.00615 | -0.00195 | Y | ||
035835 | -0.00595 | -0.00315 | Y | ||
040534 | -0.00575 | -0.00255 | Crossline | Y | |
041013 | -0.00615 | -0.00235 | Line back at top (pair with 031124) | Y | |
042416 | -0.00615 | -0.00235 | Single line | Y |
comment:7 Changed 9 years ago by stgo
Hyperspectral
Starting hyperspectral processing
comment:8 Changed 9 years ago by tec
RCD
Started
comment:9 Changed 9 years ago by stgo
Hyperspectral
SCT values:
Flightline | FENIX |
1 | 0.90 |
2 | 1.00 |
3 | 1.00 |
4 | 1.00 |
5 | 0.90 |
6 | 1.01 |
7 | 0.80 |
8 | 1.20 |
9 | 0.90 |
10 | 0.97 |
comment:10 Changed 9 years ago by tec
RCD
Images are rubbish, all are unusable so no delivery for this.
comment:11 Changed 9 years ago by stgo
Hyperspectral
Ready for delivery check.
comment:12 Changed 9 years ago by asm
Hyperspectral DC
Starting delivery check
comment:13 Changed 9 years ago by asm
Found problem with underflow table in Readme file. Apparently the script is not filling the table properly. Found values manually:
Underflows:
-line 01: 504-524, 592-622
-line 02: 499-529, 600-622
-line 03: 497-536, 579-622
-line 04: 496-545, 576-622
-line 05: 424-440, 495-622
-line 06: 498-540, 580-622
-line 07: 420-439, 493-622
-line 08: 432-433(small),496-545, 576-622
-line 09: 428-439, 495-544, 576-622
-line 10: 424-438, 494-545, 577-622
Overflow: (Don't fully understand this section, someone please review)
-line 04: 505-525, 605-622
-line 08: 321-324
Everything else seems correct.
This is my first delivery checking, second checking should be needed.
comment:14 follow-up: ↓ 15 Changed 9 years ago by lah
2nd Fenix DC
- data quality report is not most recent.(needs manually replacing)
- QC tables should not be in delivery
- There were *aux.xml files in the mapped folder from viewing with Tuiview. These always need removing at the end of delivery check.
- fodis directory needs removing
- some underflows are missing from readme table. I spot an error with the autoQC script! I'll look into this. (fenix overflows are unreliable in fastQC until the fenix binning problem is sorted. autoQC should be ok using values from the mask. Once reprocessed the overflows should show up in fastQC)
- scts look ok
I haven't made any changes as a new delivery will need to be created once reprocessed with fenix errors corrected.
comment:15 in reply to: ↑ 14 ; follow-up: ↓ 17 Changed 9 years ago by lah
Replying to lah:
2nd Fenix DC
- data quality report is not most recent.(needs manually replacing)
- QC tables should not be in delivery
- There were *aux.xml files in the mapped folder from viewing with Tuiview. These always need removing at the end of delivery check.
- fodis directory needs removing
- some underflows are missing from readme table. I spot an error with the autoQC script! I'll look into this. (fenix overflows are unreliable in fastQC until the fenix binning problem is sorted. autoQC should be ok using values from the mask. Once reprocessed the overflows should show up in fastQC)
- scts look ok
I haven't made any changes as a new delivery will need to be created once reprocessed with fenix errors corrected.
ok reran autoQC and output is fine. Looks like you've just copied stuff into QC_under.txt
comment:16 follow-up: ↓ 18 Changed 9 years ago by tec
LiDAR
Issue with DEM screenshot, waiting for dac to reappear
comment:17 in reply to: ↑ 15 Changed 9 years ago by lah
Replying to lah:
Replying to lah:
2nd Fenix DC
- data quality report is not most recent.(needs manually replacing)
- QC tables should not be in delivery
- There were *aux.xml files in the mapped folder from viewing with Tuiview. These always need removing at the end of delivery check.
- fodis directory needs removing
- some underflows are missing from readme table. I spot an error with the autoQC script! I'll look into this. (fenix overflows are unreliable in fastQC until the fenix binning problem is sorted. autoQC should be ok using values from the mask. Once reprocessed the overflows should show up in fastQC)
- scts look ok
I haven't made any changes as a new delivery will need to be created once reprocessed with fenix errors corrected.
ok reran autoQC and output is fine. Looks like you've just copied stuff into QC_under.txt
found error in autoqc (this is called rather than the almost identical autoQC) and fixed, so readme over/under flow table creates correctly now.
comment:18 in reply to: ↑ 16 Changed 9 years ago by dac
Replying to tec:
LiDAR
Issue with DEM screenshot, waiting for dac to reappear
Problem with screenshot is bounds of lidar are larger than hyperspectral, Recreating DEM with --lidar_bounds flag.
comment:19 Changed 9 years ago by dac
LiDAR
Recreated DEMs and screenshot to lidar bounds.
comment:20 Changed 9 years ago by tec
LiDAR
Thanks #DanClewley, DEM looks good now, LiDAR ready for DC
comment:21 Changed 9 years ago by asm
Lidar DC
Started Delivery Check for Lidar
comment:22 Changed 9 years ago by asm
Lidar DC
Not sure if flight line 001 should be delivered. There are only 10 flightlines on the logsheet and the summary says: "This project directory contains one flightline from the aborted flight 306b. Be aware of this while processing. When downloaded from thelma, there were no basestation data included in the project"
Done all the others steps for DC, everything else seems OK.
This is my first Lidar DC, second checking should be needed. But it should wait until check the strange flight line.
comment:23 Changed 9 years ago by asm
Lidar DC
Getting rid of flightline 001 and re-creating delivery. Should need to re-create screenshots, logsheet and readme.
comment:24 Changed 9 years ago by asm
Lidar DC
Need to add the second PI to the readme and logsheet.
comment:25 Changed 9 years ago by asm
Lidar DC
Created new screenshots and readme-files. Everything is done except the new logsheet (waiting to solve problems with generateLogsheet.py)
comment:26 Changed 9 years ago by asm
Lidar DC
Everything solved now. Ready for second DC.
comment:27 Changed 9 years ago by dap
Beginning 2nd LiDAR delivery check
comment:28 Changed 9 years ago by dac
LiDAR DC
Taking over second DC (needed as this was Aser's first DC) from Dale.
comment:29 Changed 9 years ago by dac
LiDAR DC
Generally looks OK, couple of things to fix:
- There is still an empty table with elevation differences in. Just need to include sentance saying it couldn't be calculated
- There are three DEMs. Need to remove / rename 'MA14_11-2014_306-lidar_ASTER.dem'
- demcompare.py reports the mean is 5.2 m
comment:30 Changed 9 years ago by asm
LiDAR
Fixed the ReadMe and deleted the DEM. Please re-check but it should be OK now.
comment:31 Changed 9 years ago by dac
LiDAR DC
Looks good - ready to deliver.
comment:32 Changed 9 years ago by asm
Dispatching Data to PI
Started. Delivery methods both FTP (for PI Coomes) and hard drive (for PI Cutler).
comment:33 Changed 9 years ago by asm
LIDAR data has been delivered via hard drive USB. (3 August 2015)
comment:34 Changed 9 years ago by asm
Dispatching Lidar Data to PI
Also set the delivery method for FTP and sent a mail to both PIs.
Finished Lidar Delivery
comment:35 Changed 9 years ago by stgo
Started Hyperspectral reprocessing
comment:36 Changed 9 years ago by stgo
Started Fenix reprocessing
Something weird happened on the grid with calibration file. APL reports that 13 bands were consistent with calibration, re running to identify if this was a one off thing from Friday night.
comment:37 Changed 9 years ago by stgo
Fenix reprocessing
Ready for delivery check.
This delivery needs to be thoroughly checked for:
- correct lines included
- correct linenames from logsheet
- data quality comments
- spectra shifts
comment:38 Changed 9 years ago by asm
Fenix reprocessing DC
Started
-Removed fodis directory.
-Added logsheet.
-All flightlines has correct name compared with logsheet and all lines from logsheet are included.
comment:39 Changed 9 years ago by asm
Fenix reprocessing DC
Need to add to the readme file:
-Both PIs names: Mark Cutler and David Coomes
-Some data quality remarks about clouds on flightline 06 and 10.
comment:40 Changed 9 years ago by asm
Fenix reprocessing DC
-Checked green vegetation spectra with Py6S. Good match for all
flightlines.
-Tested all tif files created from apl commands.
comment:41 Changed 9 years ago by lah
Fenix DC - spectra
Checked spectra of a few lines and no major problems. There is a persistent spike in the mask region for this flight, not unlike many other flights. The spectral shift is within 5 nm compared to previous data (2014 169). SWIR detector looks fairly stable from line 1.
comment:42 Changed 9 years ago by lah
Fenix DC
Starting final checks and corrections.
comment:43 Changed 9 years ago by lah
Fenix DC
- Replaced data quality report with most recent
- renamed screenshots
- rechecked all bands of all lines in level 1 files using fastQC, and all look good.
- Recreated readme with 2 PIs and mentioned more cloud on line 10.
Zipping up mapped files and then will be ready to deliver.
Have deleted June delivery in processing folder.
comment:44 Changed 9 years ago by lah
Fenix
Delivered to PIs on 28/10/15 by ftp (arsf15).
comment:45 Changed 9 years ago by dac
Uploaded Fenix and LiDAR to FTP server (arsf13) for download by David Coomes - was cc'd in previous delivery but didn't download before slot was reused.
comment:46 Changed 8 years ago by dac
Archiving
Started uploading to NEODC.
comment:47 Changed 8 years ago by dac
- Resolution set to fixed
- Status changed from new to closed
Archiving
Data available from NEODC. Still under embargo so not yet accessible.
David Coomes at the University of Cambridge should also be delivered the processed data from this flight as they shared this site between both projects. (See email below from Gary):
(email trimmed)
David Coomes data deliveries should also include those for MA14-11 (Mark Cutler) as they shared this site between both projects
(email trimmed)