MA14/21, flight day 283/2014, Danum Valley

[dap]

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/MA14_21-2014_283_Danum_Valley

Data arrived from ARSF via network transfer on 19/12/2014.

Scientific objective: Quantifying the accuracy of current estimates of forest biomass by combining data retrieved from state-of-the-art ground-based laser scanner with airborne data.

Priority: ?

PI: Mathias Disney

Delivery requested via FTP / Hard Disk

Note that this project directory contains the LiDAR and navigation data collected from the aborted flight 283a as well as the LiDAR and navigation data collected from flight 283b. The data relevant for this project are the ones collected in flight 283b. See the logsheet for more details.

Part of dataset was also sent via USB for testing during Malaysia campaign. This was mislabelled 243b. Processing is described in ​Ticket 544

Sensors:

• Fenix (Requested)
• Leica LIDAR (Requested)
• Leica LiDAR FW (?)
• RCD (Not Requested, flown anyway)

[tec]

Navigation Processing
Antenna height 1.40m
Lat 4 57 48.36552
Lon 117 48 10.32636
Ell 209.225

[tec]

Nav done, processed in ipas pro, all defaults seemed fine though it wouldn't finish processing in ipas tc for some reason.

[tec]

Fenix Processing
Starting, stuck on error TreeGridSize, waiting on mark1

[tec]

RCD Processing
Started converting tiffs

[lah]

Lidar Processing
First 4 lines don't process because they don't have corresponding nav data. Logsheet mentions that there was a problem and that these lines were reflown, so these probably don't need to be processed. Line 5 looks to have been aborted and is not worth processing (only 5MB compared to 200-250 MB for the other lines).

[lah]

Lidar Processing
Only tree canopies to check roll and pitch offsets, but boresight values seem fine (r: -0.006375, p: 0.00016318)

[tec]

RCD Processing
Removing images 008 011 012 013 014 021 022 026 027 029 036 037 038 039 040 041 042 044.

[lah]

Logsheet
Generated logsheet, but align in and out are before and after takeoff, so transit time is invalid.

[tec]

RCD Processing
Finished

[lah]

Lidar delivery created, ready for checking.

[tec]

LiDAR DC
Started

[tec]

LiDAR DC
Dem looks wrong, all lines are black and also looks far too big. Have asked Dac's opinion.
Demcompare gives 165, not good :'(
Pitch and roll look good.
Sol file was incorrectly named. -fixed
AGC is only in fw but that's acceptable, though ideally should be in both.

Other than the dem project is fine.

[dac]

When comparing the lidar DSM to ASTER the elevation values are too low, some areas I looked at were ~ 200 m lower, demcompare results reported by tec confirms this. Also checked against 3 arcsec SRTM, values are ~ 20 m different to ASTER but still a lot higher than lidar DSM.

Insufficient features to check horizontal accuracy (which could be the reason for the vertical offset) - suggest comparing to hyperspectral once processed.

[lah]

Lidar

Checked alspp settings and it appears that files might have been processed with horizontal datum ETRF89. Reprocessing with WGS84 to see if this solves the problem.

[tec]

Fenix
Line 8 doesnt line up at the end of the road, its most odd but I minimised the wobbles so dont think this is an sct problem.

[lah]

Lidar
Alas, reprocessed files dem has same problem. Lines appear identical to previous delivery, so they were probably processed in WGS84.

[dac]

RCD DC

Starting delivery check

[dac]

RCD DC
Looks good, couple of minor comments on the readme but other than that ready to deliver:

• Might want to mention that images weren't included for all lines (if normally do this)
• In the readme there is 'Some images contain cloud, they have been included as the images still contain useable data.' However, all the images look cloud free. There are some cloud shadows, so I would change to mention this.

[lah]

Lidar
Checked overlapping flight lines with 243b and the vertical offset is definitely in the flight lines (165m). Horizontal position matches 243b though. Could just put vertical offset into alspp, but doesn't explain why the offset is there.

[tec]

Replying to dac:

RCD DC
Looks good, couple of minor comments on the readme but other than that ready to deliver:

• Might want to mention that images weren't included for all lines (if normally do this)
• In the readme there is 'Some images contain cloud, they have been included as the images still contain useable data.' However, all the images look cloud free. There are some cloud shadows, so I would change to mention this.

Done, marking as ready to deliver

[tec]

Fenix

Flightline	FENIX
6	0.98
7	1.10
8	0.97
9	0.90
10	0.97
11	-0.22
12	0.90
13	0.99

[tec]

Fenix
Ready for delivery check

[dac]

Fenix DC

Starting delivery check

[dac]

Fenix DC

Delivery looks fine. However, the files look a bit weird. There is a bright / dark (depending on wavelength) strip down the centre of every line starting from band ~ 240 until the last band. This is present in the 1b and mapped data. It would be good to try and figure out what is causing this and mention in the readme.

Couple of comments.

• Empty fodis directory - removed
• There are two DEMs, one of which looks filtered. Remove the one which isn't required.

[tec]

Replying to dac:

Fenix DC

Delivery looks fine. However, the files look a bit weird. There is a bright / dark (depending on wavelength) strip down the centre of every line starting from band ~ 240 until the last band. This is present in the 1b and mapped data. It would be good to try and figure out what is causing this and mention in the readme.

Couple of comments.

• Empty fodis directory - removed
• There are two DEMs, one of which looks filtered. Remove the one which isn't required.

Removed DEM, mark1 is looking into the band thing

[dac]

LiDAR

Created DSM and compared to one generated from 243b data (both at 0.5 m resolution) using 'create_lidar_diff_image.py' with variance mask turned off (set to very high value).

Absolute mean diff 164.670 m (standard deviation 6.728)

Added this value onto every point in LAS 1.2 and LAS 1.3 files using laspy and copied to:

/users/rsg/dac/scratch_network/MA14_21-2014_283_Danum_Valley_lidar_offset

Updated files match up with 243b point clouds so suggest using these files for the delivery unless we can determine the actual reason for the offsets. Add explanation that an offset was applied to readme file.

[lah]

Lidar ready for DC

Remade delivery with updated files and commented in readme about offset. Original delivery has not yet been deleted, so can be used for investigation.

[dac]

LiDAR DC

Files with offset applied look better. Have added more detail to text in readme about offset.

Ready to delivery. Copying to FTP server now.

[dac]

LiDAR

Sent lidar data to PI (Mat Disney) 19/06/2015. Still investigating cause of offset, have checked likely causes (wrong datum, problems with navigation data etc.,) so need to investigate other reasons for offset.

Will deliver on hard drive and update on FTP (if needed) once this investigation has been concluded.

Marking project as finished as data has been delivered via FTP and investigation into offset isn't likely to significantly alter data, just identify if there is anything needed to prevent whatever caused the offset happening for future flights.

[dac]

LiDAR

PI Confirmed download of data 19/06/2015

[stgo]

Fenix Reprocessing

Started reprocessing

[dac]

Fenix

Area covered by these lines is also covered by two lines currently available from 243b.

[dac]

Fenix

As these data are processed they might as well be delivered, despite the stripe down the middle of the lines (likely caused by moisture on the sensor lens).

Have added note at top of the Readme.

Ready for delivery check.

[dac]

LiDAR Processing

Processed first navigation file to produce '_start_of_flight.sol' which covers first 4 lines. Using this able to process first 4 lines which couldn't previously be processed. Pitch and roll values used for other lines look good. Making delivery.

[asm]

Fenix Delivery Check

Started.

[asm]

Fenix Delivery Check

-Logsheet shows sortie b for this project but there is not a 283a, I suggest to delete it.
-hyp_genreadme and readme: Because it is considering sortie b, Dem appears as 283b, please remove.
-proj_tidy is complaining about logsheet name on delivery
-There are still some visible wobbles (on screenshot but much easier to spot using tuiview) on flightlines 11 and (specially) 12. That could explain comment posted on the ticket that flightline 08 does not have wobbles but does not overlaps. I suggest to re-process SCT values or add a note on the readme if those are the best values we can get.

[gej]

Fenix

Have found new SCT values for flightlines 11 and 12.
12 still has some substantial wobble but I think it is better than it was previously, the positioning against the other flightlines is also better.

New SCTS

Flightline	SCT
11	-0.34
12	0.84

[gej]

Fenix

Mapped files created and added to delivery, screenshots recreated.

Have kept a backup of the old mapped files and screenshots for 11 and 12 in processing/hyperspectral/flightlines/georeferencing/mapped/Old_Full_Mapped_11_12 incase it's decided that the old SCT values are better.

Ready for DC.

[asm]

Fenix DC

-Readme still has incorrect dem on aplcorr commands (but has solved from hyp_genreadme and commands have been tested). Should be 283 and not 283b
-Please add a note to data quality about wobbly lines.
-Have checked the spectra against py6s and looks OK except for a peak on the overlapping region that will be masked out.

Edit: dac suggested we will not deliver flightline 11 and 12 since they are repeated. They will be removed from this delivery

[dac]

LiDAR

Have recreated delivery with first 4 lines included and file numbers changed accordingly. Ready for delivery check.

[gej]

Fenix DC

Have removed flightlines 11 and 12 from delivery, recreated mosaic screenshot and readme to reflect this.

Ready for check again

[asm]

Fenix DC

-All looks fine now. I will suggest to add a note about not including those two flightlines on the readme.

Will zip files and once done will mark as ready to deliver.

[gej]

Lidar DC

Everything looks okay. utm dem has a mean of 8.8 so just over what suggested limits are, but latlong dem is 7.78, so I think it's okay.

Everything else looks fine.

Ready for delivery.

[dac]

LiDAR

About the difference between the LiDAR and ASTER DSM - given the height of the trees and possible errors in the ASTER data a difference of < 10 m is fine.

[dac]

Fenix and LiDAR data delivered to PI via FTP (arsf6) 29/01/2016.

[lah]

Archiving
There is Owl raw data, but no internal black body calibration files.

Uploaded to NEODC 20/06/16.

[asm]

JSON Files
Could not create JSON files for Lidar delivery, script failing. Will try to resolve before creating all JSON files.

[dac]

JSON creation

Script was failing due to LAS files having incorrect time. Modified script to set altitude as 'unknown' so files could be created.