Opened 5 years ago

Closed 3 years ago

Last modified 14 months ago

#557 closed flight processing (fixed)

MA14/21, flight day 283/2014, Danum Valley

Reported by: dap Owned by:
Priority: whenever Milestone: 2014 data processing completion
Component: Processing: general Keywords:
Cc: Other processors:

Description (last modified by dac)

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/MA14_21-2014_283_Danum_Valley

Data arrived from ARSF via network transfer on 19/12/2014.

Scientific objective: Quantifying the accuracy of current estimates of forest biomass by combining data retrieved from state-of-the-art ground-based laser scanner with airborne data.

Priority: ?

PI: Mathias Disney

Delivery requested via FTP / Hard Disk

Note that this project directory contains the LiDAR and navigation data collected from the aborted flight 283a as well as the LiDAR and navigation data collected from flight 283b. The data relevant for this project are the ones collected in flight 283b. See the logsheet for more details.

Part of dataset was also sent via USB for testing during Malaysia campaign. This was mislabelled 243b. Processing is described in Ticket 544

Sensors:

  • Fenix (Requested)
  • Leica LIDAR (Requested)
  • Leica LiDAR FW (?)
  • RCD (Not Requested, flown anyway)

Change History (51)

comment:1 Changed 5 years ago by tec

Navigation Processing
Antenna height 1.40m
Lat 4 57 48.36552
Lon 117 48 10.32636
Ell 209.225

Last edited 5 years ago by tec (previous) (diff)

comment:2 Changed 5 years ago by tec

Nav done, processed in ipas pro, all defaults seemed fine though it wouldn't finish processing in ipas tc for some reason.

comment:3 Changed 5 years ago by tec

Fenix Processing
Starting, stuck on error TreeGridSize, waiting on mark1

comment:4 Changed 5 years ago by tec

RCD Processing
Started converting tiffs

comment:5 Changed 5 years ago by lah

Lidar Processing
First 4 lines don't process because they don't have corresponding nav data. Logsheet mentions that there was a problem and that these lines were reflown, so these probably don't need to be processed. Line 5 looks to have been aborted and is not worth processing (only 5MB compared to 200-250 MB for the other lines).

comment:6 Changed 5 years ago by lah

Lidar Processing
Only tree canopies to check roll and pitch offsets, but boresight values seem fine (r: -0.006375, p: 0.00016318)

comment:7 Changed 5 years ago by tec

RCD Processing
Removing images 008 011 012 013 014 021 022 026 027 029 036 037 038 039 040 041 042 044.

comment:8 Changed 5 years ago by lah

Logsheet
Generated logsheet, but align in and out are before and after takeoff, so transit time is invalid.

comment:9 Changed 5 years ago by tec

RCD Processing
Finished

comment:10 Changed 5 years ago by lah

Lidar delivery created, ready for checking.

comment:11 Changed 5 years ago by tec

LiDAR DC
Started

comment:12 Changed 5 years ago by tec

LiDAR DC
Dem looks wrong, all lines are black and also looks far too big. Have asked Dac's opinion.
Demcompare gives 165, not good :'(
Pitch and roll look good.
Sol file was incorrectly named. -fixed
AGC is only in fw but that's acceptable, though ideally should be in both.

Other than the dem project is fine.

comment:13 Changed 5 years ago by dac

When comparing the lidar DSM to ASTER the elevation values are too low, some areas I looked at were ~ 200 m lower, demcompare results reported by tec confirms this. Also checked against 3 arcsec SRTM, values are ~ 20 m different to ASTER but still a lot higher than lidar DSM.

Insufficient features to check horizontal accuracy (which could be the reason for the vertical offset) - suggest comparing to hyperspectral once processed.

comment:14 Changed 5 years ago by lah

Lidar

Checked alspp settings and it appears that files might have been processed with horizontal datum ETRF89. Reprocessing with WGS84 to see if this solves the problem.

comment:15 Changed 5 years ago by tec

Fenix
Line 8 doesnt line up at the end of the road, its most odd but I minimised the wobbles so dont think this is an sct problem.

comment:16 Changed 5 years ago by lah

Lidar
Alas, reprocessed files dem has same problem. Lines appear identical to previous delivery, so they were probably processed in WGS84.

comment:17 Changed 5 years ago by dac

RCD DC

Starting delivery check

comment:18 follow-up: Changed 5 years ago by dac

Looks good, couple of minor comments on the readme but other than that ready to deliver:

  • Might want to mention that images weren't included for all lines (if normally do this)
  • In the readme there is 'Some images contain cloud, they have been included as the images still contain useable data.' However, all the images look cloud free. There are some cloud shadows, so I would change to mention this.
Version 0, edited 5 years ago by dac (next)

comment:19 Changed 5 years ago by lah

Lidar
Checked overlapping flight lines with 243b and the vertical offset is definitely in the flight lines (165m). Horizontal position matches 243b though. Could just put vertical offset into alspp, but doesn't explain why the offset is there.

comment:20 in reply to: ↑ 18 Changed 5 years ago by tec

Replying to dac:

RCD DC
Looks good, couple of minor comments on the readme but other than that ready to deliver:

  • Might want to mention that images weren't included for all lines (if normally do this)
  • In the readme there is 'Some images contain cloud, they have been included as the images still contain useable data.' However, all the images look cloud free. There are some cloud shadows, so I would change to mention this.

Done, marking as ready to deliver

comment:21 Changed 5 years ago by tec

Fenix

Flightline FENIX
6 0.98
7 1.10
8 0.97
9 0.90
10 0.97
11 -0.22
12 0.90
13 0.99

comment:22 Changed 5 years ago by tec

Fenix
Ready for delivery check

comment:23 Changed 5 years ago by dac

Fenix DC

Starting delivery check

comment:24 follow-up: Changed 5 years ago by dac

Fenix DC

Delivery looks fine. However, the files look a bit weird. There is a bright / dark (depending on wavelength) strip down the centre of every line starting from band ~ 240 until the last band. This is present in the 1b and mapped data. It would be good to try and figure out what is causing this and mention in the readme.

Couple of comments.

  • Empty fodis directory - removed
  • There are two DEMs, one of which looks filtered. Remove the one which isn't required.

comment:25 in reply to: ↑ 24 Changed 5 years ago by tec

Replying to dac:

Fenix DC

Delivery looks fine. However, the files look a bit weird. There is a bright / dark (depending on wavelength) strip down the centre of every line starting from band ~ 240 until the last band. This is present in the 1b and mapped data. It would be good to try and figure out what is causing this and mention in the readme.

Couple of comments.

  • Empty fodis directory - removed
  • There are two DEMs, one of which looks filtered. Remove the one which isn't required.

Removed DEM, mark1 is looking into the band thing

comment:26 Changed 5 years ago by dac

LiDAR

Created DSM and compared to one generated from 243b data (both at 0.5 m resolution) using 'create_lidar_diff_image.py' with variance mask turned off (set to very high value).

Absolute mean diff 164.670 m (standard deviation 6.728)

Added this value onto every point in LAS 1.2 and LAS 1.3 files using laspy and copied to:

/users/rsg/dac/scratch_network/MA14_21-2014_283_Danum_Valley_lidar_offset

Updated files match up with 243b point clouds so suggest using these files for the delivery unless we can determine the actual reason for the offsets. Add explanation that an offset was applied to readme file.

comment:27 Changed 5 years ago by dac

  • Description modified (diff)

comment:28 Changed 4 years ago by lah

Lidar ready for DC

Remade delivery with updated files and commented in readme about offset. Original delivery has not yet been deleted, so can be used for investigation.

comment:29 Changed 4 years ago by dac

LiDAR DC

Files with offset applied look better. Have added more detail to text in readme about offset.

Ready to delivery. Copying to FTP server now.

comment:30 Changed 4 years ago by dac

LiDAR

Sent lidar data to PI (Mat Disney) 19/06/2015. Still investigating cause of offset, have checked likely causes (wrong datum, problems with navigation data etc.,) so need to investigate other reasons for offset.

Will deliver on hard drive and update on FTP (if needed) once this investigation has been concluded.

Marking project as finished as data has been delivered via FTP and investigation into offset isn't likely to significantly alter data, just identify if there is anything needed to prevent whatever caused the offset happening for future flights.

comment:31 Changed 4 years ago by dac

LiDAR

PI Confirmed download of data 19/06/2015

comment:32 Changed 4 years ago by stgo

Fenix Reprocessing

Started reprocessing

comment:33 Changed 4 years ago by dac

Fenix

Area covered by these lines is also covered by two lines currently available from 243b.

comment:34 Changed 4 years ago by dac

Fenix

As these data are processed they might as well be delivered, despite the stripe down the middle of the lines (likely caused by moisture on the sensor lens).

Have added note at top of the Readme.

Ready for delivery check.

comment:35 Changed 4 years ago by dac

  • Description modified (diff)

comment:36 Changed 4 years ago by dac

LiDAR Processing

Processed first navigation file to produce '_start_of_flight.sol' which covers first 4 lines. Using this able to process first 4 lines which couldn't previously be processed. Pitch and roll values used for other lines look good. Making delivery.

comment:37 Changed 4 years ago by asm

Fenix Delivery Check

Started.

comment:38 Changed 4 years ago by asm

Fenix Delivery Check

-Logsheet shows sortie b for this project but there is not a 283a, I suggest to delete it.
-hyp_genreadme and readme: Because it is considering sortie b, Dem appears as 283b, please remove.
-proj_tidy is complaining about logsheet name on delivery
-There are still some visible wobbles (on screenshot but much easier to spot using tuiview) on flightlines 11 and (specially) 12. That could explain comment posted on the ticket that flightline 08 does not have wobbles but does not overlaps. I suggest to re-process SCT values or add a note on the readme if those are the best values we can get.

Last edited 4 years ago by asm (previous) (diff)

comment:39 Changed 4 years ago by gej

Fenix

Have found new SCT values for flightlines 11 and 12.
12 still has some substantial wobble but I think it is better than it was previously, the positioning against the other flightlines is also better.

New SCTS

FlightlineSCT
11-0.34
120.84

comment:40 Changed 4 years ago by gej

Fenix

Mapped files created and added to delivery, screenshots recreated.

Have kept a backup of the old mapped files and screenshots for 11 and 12 in processing/hyperspectral/flightlines/georeferencing/mapped/Old_Full_Mapped_11_12 incase it's decided that the old SCT values are better.

Ready for DC.

comment:41 Changed 4 years ago by asm

Fenix DC

-Readme still has incorrect dem on aplcorr commands (but has solved from hyp_genreadme and commands have been tested). Should be 283 and not 283b
-Please add a note to data quality about wobbly lines.
-Have checked the spectra against py6s and looks OK except for a peak on the overlapping region that will be masked out.

Edit: dac suggested we will not deliver flightline 11 and 12 since they are repeated. They will be removed from this delivery

Last edited 4 years ago by asm (previous) (diff)

comment:42 Changed 4 years ago by dac

LiDAR

Have recreated delivery with first 4 lines included and file numbers changed accordingly. Ready for delivery check.

comment:43 Changed 4 years ago by gej

Fenix DC

Have removed flightlines 11 and 12 from delivery, recreated mosaic screenshot and readme to reflect this.

Ready for check again

comment:44 Changed 4 years ago by asm

Fenix DC

-All looks fine now. I will suggest to add a note about not including those two flightlines on the readme.

Will zip files and once done will mark as ready to deliver.

comment:45 Changed 4 years ago by gej

Lidar DC

Everything looks okay. utm dem has a mean of 8.8 so just over what suggested limits are, but latlong dem is 7.78, so I think it's okay.

Everything else looks fine.

Ready for delivery.

comment:46 Changed 4 years ago by dac

LiDAR

About the difference between the LiDAR and ASTER DSM - given the height of the trees and possible errors in the ASTER data a difference of < 10 m is fine.

comment:47 Changed 4 years ago by dac

Fenix and LiDAR data delivered to PI via FTP (arsf6) 29/01/2016.

comment:48 Changed 3 years ago by lah

Archiving
There is Owl raw data, but no internal black body calibration files.

Uploaded to NEODC 20/06/16.

comment:49 Changed 3 years ago by asm

  • Resolution set to fixed
  • Status changed from new to closed

comment:50 Changed 23 months ago by asm

JSON Files
Could not create JSON files for Lidar delivery, script failing. Will try to resolve before creating all JSON files.

comment:51 Changed 14 months ago by dac

JSON creation

Script was failing due to LAS files having incorrect time. Modified script to set altitude as 'unknown' so files could be created.

Note: See TracTickets for help on using tickets.