Opened 6 years ago

Last modified 5 years ago

#513 new flight processing

BAS13/01, flight day 068a/2014, Tamar

Reported by: knpa Owned by:
Priority: alpha 4 medium Milestone:
Component: Processing: general Keywords:
Cc: Other processors:

Description

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/uk/BAS13_01-2014_068a_Tamar/

Data arrived from ARSF via network transfer on 10/03/2014

BAS13/01 Scientific objective: unknown

BAS13/01 Priority: ?

PI: France Gerard

Sensors:

Camera (Requested)
LiDAR (Requested)

Change History (49)

comment:1 Changed 5 years ago by knpa

France would like the flightlines divided into manageable chunks, 3 per flightline was suggested.

comment:2 Changed 5 years ago by tipo

RCD images are all somewhat to very overexposed. According to the log, exposure was fixed at 2 for the entire flight, rather than varying.

comment:3 Changed 5 years ago by benj

Gary has confirmed that the PI would like FW LiDAR data for this flight (if we have any).

comment:4 Changed 5 years ago by tipo

Processed LiDAR, will get steve to check it before creating the delivery, as they take a long time to create.

comment:5 Changed 5 years ago by tipo

Created LiDAR delivery, but apl fails to create the wgs84 dem. Possibly due to the large size of the flight? Trying lower resolution to see if that helps.

comment:6 Changed 5 years ago by tipo

Created RCD delivery. Given the poor quality of the images, I'm unsure if it's worth delivering, but it's there in case it's wanted.

comment:7 Changed 5 years ago by tipo

Created LiDAR delivery. Flightline files have been split into 3 parts with overlap as requested.

comment:8 Changed 5 years ago by stgo

Beginning RCD D/c

comment:9 Changed 5 years ago by stgo

D/c complete, photographs don't have full exif tagging. Also any explanation for the overexposure available for the readme?

comment:10 Changed 5 years ago by tipo

Images are now tagged, readme updated to give more info on the overexposure.

comment:11 Changed 5 years ago by stgo

Beginning RCD D/c

comment:12 Changed 5 years ago by stgo

RCD delivery check finished, re run readme script to get full exif from photos, otherwise ready to deliver.

Last edited 5 years ago by stgo (previous) (diff)

comment:13 Changed 5 years ago by stgo

Lidar DC in progress, couple of initial comments:

  • need a note somewhere about the format of file splitting (a b c)
  • need to make sure it's clear the elevation difference is between lines and not along lines.

comment:14 Changed 5 years ago by stgo

LiDAR DC (tentatively) complete.

Roll errors present between lines 5, 6 7 and 8. Will check others for more, seems to be about 16 cm in size. Particularly noticeable in part a/c of these lines in the town.

comment:15 Changed 5 years ago by stgo

RCD data dispatched on 04/06/2014

comment:16 Changed 5 years ago by mark1

Edited KML file in-place in RCD delivery to correct data paths for rcd thumbnails.

comment:17 Changed 5 years ago by stgo

Lines 105248 through to 121418 match up with the following roll values:

roll-correct = -0.004034393
pitch-correct = -0.000880469

1012 and 1026 are seperate from the main data body. I will check for any lines that overlap but they appear to have a considerable roll variation (40 cm).

Line 122811 has a roll variation of 15 cm from 121418. I have not checked the lines beyond this in the dataset.

Last edited 5 years ago by stgo (previous) (diff)

comment:18 Changed 5 years ago by dac

Found correct roll offset values.

Line Roll Pitch Heading
140309_101258 -0.003904393 -0.000880469 0.00045631
140309_102632 -0.003904393 -0.000880469 0.00045631
140309_103944 -0.003904393 -0.000880469 0.00045631
140309_105248 -0.003804393 -0.000880469 0.00045631
140309_110610 -0.003804393 -0.000880469 0.00045631
140309_111932 -0.003904393 -0.000880469 0.00045631
140309_113312 -0.003904393 -0.000880469 0.00045631
140309_114615 -0.003904393 -0.000880469 0.00045631
140309_115952 -0.004004393 -0.000880469 0.00045631
140309_121418 -0.004004393 -0.000880469 0.00045631
140309_122811 -0.004004393 -0.000880469 0.00045631
140309_124117 -0.004004393 -0.000880469 0.00045631
140309_125519 -0.004104393 -0.000880469 0.00045631
140309_130917 -0.004104393 -0.000880469 0.00045631

comment:19 Changed 5 years ago by dac

Only a single raw full-waveform file is available (140309_092407) which could not be processed.

AGL(m) given as -1 in ALSPP. Not possible to process full-waveform or corresponding discrete file.

Likely acquired prior to rest of data as all 14 lines of discrete return data on log sheet accounted for.

comment:20 Changed 5 years ago by dac

Classifying data

Last edited 5 years ago by dac (previous) (diff)

comment:21 Changed 5 years ago by dac

LiDAR Processing
Created delivery, ready for delivery check.

comment:22 Changed 5 years ago by lah

starting lidar DC

comment:23 Changed 5 years ago by lah

Lidar DC complete
Have removed .LAS from screenshot names and suggest rechecking a couple of things:

  • Roll values seem slightly off for a few lines, particularly 4-6. From 8 onwards are fine. Pitch is off too generally.

Roll errors

2-3 0.2
3-4 0.3
4-5 0.4
5-6 0.6
6-7 0.2
7-8 0.3
  • There is some missed noise in the classification. Above ground is probably acceptable, but there is a lot more below ground missed. Some points are classified as noise within the cluster if you want a way of finding these manually. Can you write a script to find the local ground level to remove these? Also can you check lines 7, 24 and 36 for haze. There are a lot of points that don't look like vegetation in a small group in each of these lines. Lines 25-30, 31-33 and 35 are fine as is.

The dem is so big it took me a while to find the flight data in it and ages for demcompare to run (4+ hours) giving mean unmasked -0.2.


comment:24 Changed 5 years ago by dac

LiDAR Processing

Refining roll values and finding pitch values for lines 1 - 8.

comment:25 Changed 5 years ago by dac

LiDAR Processing
Found pitch and roll value for first two lines:

roll-correct = -0.0039081257
pitch-correct = -0.0006873354

Different value required for third line:

comment:26 Changed 5 years ago by dac

Found pitch and roll values for third and fourth lines:

roll-correct = -0.0040081257
pitch-correct = -0.0006273354

comment:27 Changed 5 years ago by dac

Updated pitch and roll vales:

Line Roll Pitch
140309_101258 -0.0039081257 -0.0007273354
140309_102632 -0.0039081257 -0.0007073354
140309_103944 -0.0040081257 -0.0007073354
140309_105248 -0.0040081257 -0.0007073354
140309_110610 -0.0040081257 -0.0007073354
140309_111932 -0.0040581257 -0.0007073354
140309_113312 -0.0040581257 -0.0007473354
140309_114615 -0.0040081257 -0.0007473354
140309_115952 -0.0040081257 -0.0007473354
140309_121418 -0.0040081257 -0.0007673354
140309_122811 -0.0040081257 -0.0007673354
140309_124117 -0.0040081257 -0.0007673354
140309_125519 -0.0040081257 -0.0007673354
140309_130917 -0.0041081257 -0.0007673354

Will get lah to check before splitting, classifying and redoing delivery.

Last edited 5 years ago by dac (previous) (diff)

comment:28 Changed 5 years ago by lah

Checked pitch and roll values for each pair. Roll is within tolerance for all lines, though is slightly off at the bottom of lines 4-5. Pitch is approx 0.7m of for lines 1-5 and about 0.4m off for lines 5-9. Lines 9-14 are fine.

comment:29 Changed 5 years ago by dac

Updated pitch and roll values, starting classification.

comment:30 Changed 5 years ago by dac

LiDAR Processing
Finished classification, creating delivery.

comment:31 Changed 5 years ago by dac

LiDAR Processing

Delivery created, ready for checking.

comment:32 Changed 5 years ago by dap

Beginning LiDAR Delivery Check

comment:33 Changed 5 years ago by dap

LiDAR Delivery Check

Renamed logsheet using rename.sh -f 68a 068a in logsheet directory.

DEM compare results:

Difference statistics:
Min:		-53.1125415039062
Max:		104.707923583984
Sum:		-1692823.31105974
Mean:		-5.36858845318959
Median:		-5.00869
Absolute mean:	7.24126409432035
Std deviation:	7.67325032796051
Total cells:	2106484
Non-null cells:	315320


Unmasked statistics:
Min:		-53.1125415039062
Max:		104.707923583984
Sum:		-1690463.24793075
Mean:		-0.864203041227932
Median:		2.0752e-06
Absolute mean:	1.44709831096314
Std deviation:	3.86373121379511
Total cells:	2106484
Non-null cells:	1956095

Roll and pitch offsets:

  • Haven't checked all pairs of lines but roll and pitch seem to be offset for at least the following pairs
    • 49 and 48
    • 48 and 47
    • 47 and 46
    • 46 and 45

Classification hasn't yet been checked.

Everything checked up to now except roll and pitch errors seems fine.

Last edited 5 years ago by dap (previous) (diff)

comment:34 Changed 5 years ago by dap

Taken over LiDAR processing from dac, correcting roll and pitch error values. Will put new values in a new comment.

comment:35 Changed 5 years ago by dap

LiDAR Processing

New roll and pitch error values have now been found, and are as follows:

Line Roll (rad) Pitch (rad)
140309_101258-0.003978125-0.000997335
140309_102632-0.003868125-0.000707335
140309_103944-0.004018125-0.000987335
140309_105248-0.003858125-0.000707335
140309_110610-0.004008125-0.000947335
140309_111932-0.003958125-0.000687335
140309_113312-0.004008125-0.000907335
140309_114615-0.004008125-0.000747335
140309_115952-0.004008125-0.000807335
140309_121418-0.004008125-0.000797335
140309_122811-0.004008125-0.000787335
140309_124117-0.004008125-0.000847335
140309_125519-0.004038125-0.000777335
140309_130917-0.004108125-0.000817335

Note that for some flight lines, the best correction that could be found was ~15cm. Beginning classification.

Last edited 5 years ago by dap (previous) (diff)

comment:36 Changed 5 years ago by dap

LiDAR Processing

Classification complete. Flight lines have also been split using

split_las.py -d processing/als50/las-classified/ -o processing/als50/las-classified/split -p 3 --time 7

comment:37 Changed 5 years ago by dap

LiDAR Processing

Dan found problems with the LAS files when they were sorted (y values all 0). Will re-split the files and move them in to the delivery. Will then re-sort them.

comment:38 Changed 5 years ago by dap

LiDAR Processing

A delivery has now been created for these data, now generating a read me document.

comment:39 Changed 5 years ago by dap

LiDAR Processing

Read me document generated, delivery at BAS13_01-068a-lidar-20150114 ready for checking.

comment:40 Changed 5 years ago by lah

Lidar DC

  • Changed txt files to windows compatible
  • DEM results:

Difference statistics:
Min: -54.8722155761719
Max: 104.695563964844
Sum: -1700968.53721365
Mean: -5.39529270944385
Median: -5.02067
Absolute mean: 7.29369607734545
Std deviation: 7.74696294199489
Total cells: 2107794
Non-null cells: 315269

Unmasked statistics:
Min: -54.8722155761719
Max: 227.498000549316
Sum: -870256.564774547
Mean: -0.444394575886368
Median: -0.269921
Absolute mean: 3.22244584094512
Std deviation: 8.90760847517729
Total cells: 2107794
Non-null cells: 1958297

  • All other checks fine, just need to check offsets and classification when get LAG running.

comment:41 follow-up: Changed 5 years ago by lah

Lidar DC

  • Residual roll and pitch offsets are small (at most 0.1m except for lines 11 & 14).
  • Weird dip appears in 8 & 11, but could be real.

comment:42 Changed 5 years ago by lah

Lidar DC

  • Just looking at lines 1-6 there seems to be quite a bit of noise that isn't classified. Obvious bits are below ground and on lake. Might need to allow for bigger clusters of noise.

comment:43 in reply to: ↑ 41 Changed 5 years ago by dap

Replying to lah:

  • Weird dip appears in 8 & 11, but could be real.

I had noticed the dip while processing. Having looked at the kml overview, it looks as though it should be there - I think it's the overflow tower in the south-west corner of Roadford Lake.

comment:44 Changed 5 years ago by dap

LiDAR Processing

Giving files another look over, classifying missed noise.

comment:45 Changed 5 years ago by lah

Lidar DC

Classification now fine - most noise has been classified.

comment:46 Changed 5 years ago by dap

LiDAR Processing

Delivery has now been created with re-classified data. Ready for delivery check if necessary.

comment:47 Changed 5 years ago by dac

LiDAR DC

  • In readme states roll error has been corrected to around 15 cm but in the table no mean difference between lines exceeds 10 cm - suggest rewording.
  • In latlong DEM ASTER and lidar weren't patched correctly, have fixed bug in dem scripts and recreated DEM.

Once readme has been reworded ready to deliver.

comment:48 Changed 5 years ago by dap

LiDAR

Read me has been reworded, now dispatching the dataset by priority archiving at NEODC.

comment:49 Changed 5 years ago by dap

All raw and processed data have now been uploaded to NEODC, 05/02/2015.

Note: See TracTickets for help on using tickets.