Opened 10 years ago

Last modified 9 years ago

#511 new flight processing

BAS13/01, flight day 062/2014, Tamar

Reported by: knpa Owned by: benj
Priority: alpha 4 medium Milestone:
Component: Archiving Keywords:
Cc: Other processors: stgo, dap

Description (last modified by dap)

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/uk/BAS13_01-2014_062_Tamar

Data arrived from ARSF via network transfer on 10/03/2014

BAS13/01 Scientific objective: unknown

BAS13/01 Priority: ?

PI: France Gerard


LiDAR (Submitted to NEODC 13/11/2014)
FW LiDAR (Submitted to NEODC 13/11/2014)

Change History (46)

comment:1 Changed 10 years ago by knpa

  • Description modified (diff)

comment:2 Changed 10 years ago by knpa

France would like the flightlines divided into manageable chunks, 3 per flightline was suggested.

comment:3 Changed 10 years ago by stgo

Basestation ppp complete:
lat 51 03 24.94210
long -4 11 58.74401
eh 68.972

comment:4 Changed 10 years ago by benj

  • Description modified (diff)

Gary has confirmed that the PI would like FW LiDAR data for this flight

comment:5 Changed 10 years ago by stgo

LiDAR processing in progress, correcting for roll error.

comment:6 Changed 10 years ago by stgo

pitch and roll values found for lines 2243 and 2227:

roll -0.003969414
pitch 0.00125716

Testing against other lines.

edit:pitch and roll were the wrong way round

Last edited 10 years ago by stgo (previous) (diff)

comment:7 Changed 10 years ago by stgo

Pitch and roll values worked fine. Classifying points, large level of cloud in line 001050 onwards.

comment:8 Changed 10 years ago by stgo

Line 004533 has complete cloud cover, this will not be delivered.

comment:9 Changed 10 years ago by stgo

Having issues with waveform data on lines flown before midnight, LasHistoViewer claims they are not valid LAS 1.3 files. I am reprocessing one locally on the windows machine to see if this is a network error. I will split the files to check it is not a file size issue as they are considerably (>20 GB) larger than post midnight lines.

If I can't find answers will contact leica for next steps.

comment:10 Changed 10 years ago by stgo

I have processed using a fence in ALSPP to make a smaller file, this takes a long time with waveform processing (1 hour per line!) and creates usable files. This suggests it is definitely a file size issue rather than a processing one. I am waiting on the windows reprocessing to confirm.

Splitting using lastools is not an option as it is not compatible with LAS 1.3. For the moment best way to split the files will be fencing with ALSPP, this will be time consuming.

comment:11 Changed 10 years ago by stgo

Created delivery and updated with waveform files. Currently creating DEM files, first run produced sizes of 14 GB which are not very manageable. I am reproducing with half the tiles and a lower resolution.

comment:12 Changed 10 years ago by stgo

I have created a secondary nav file focusing purely on the data location for dem cretion, produces a 500 Mb file which is much more manageable. I would recommend this process for other flights but will try to script a way to use nav gpstimes instead of bounding box/ as well as bounding for dem generation.

Creation of the dem has also highlighted some particularly bad cloud I must have missed in first run for classification. Will repair before continuing.

Last edited 10 years ago by stgo (previous) (diff)

comment:13 Changed 10 years ago by stgo

Line 235554 has an extremely high level of cloud and what little land can be seen is covered in haze. Will discard.

comment:14 Changed 10 years ago by stgo

Line 2339 has not processed through the classification script, will find out why and repair.

comment:15 Changed 10 years ago by stgo

I've found variable roll errors in a couple of lines, will correct.

comment:16 Changed 10 years ago by stgo

Corrected roll errors:

222719: -0.004200413
224301: -0.003850413
001050: -0.003989414
001827: -0.003989414
002716: -0.003700413

225706 to 233904: -0.004049414

Now processing the full waveform data over this weekend.

comment:17 Changed 10 years ago by stgo

This dataset has been split, lines 0010 and 0018 (21, 22 and 23 in the delivery) were smaller than the others. 0010 has not been split. 0018 has been split into 2 rather than 3 parts.

Some lines have not been included because of complete cloud coverage (2355 0045) and it should be noted some of the other lines suffer from high cloud coverage.

Waveform files are already split and classified in las-classified/fw_class/{a..c} and a delivery has already been produced.

The dems and trj files have already been created, all that is needed is a readme and it will be good to go.

comment:18 Changed 10 years ago by dap

LiDAR Processing

Taken over processing from stgo. Continuing by filling in information in Readme config file and creating readme.

comment:19 Changed 10 years ago by dap

LiDAR Processing

Found problem with las1.0 files in delivery - part of two of the flightlines (224301 and 001827) is missing. Will reprocess these flightlines in alspp using roll and pitch values above.

edit: Two files were missing

Last edited 10 years ago by dap (previous) (diff)

comment:20 Changed 10 years ago by dap

LiDAR Processing

Re-classified point clouds and re-generated screenshots due to reprocessing of lines. Created read me for LiDAR data.

LiDAR data now ready for delivery.

comment:21 Changed 10 years ago by dap

Edit: wrong ticket

Last edited 10 years ago by dap (previous) (diff)

comment:22 Changed 10 years ago by knpa

Beginning LiDAR delivery check.

comment:23 Changed 10 years ago by knpa

Major fail on classification.

Just looking at the first line, there's too much missed noise. Ask me to show you where if you're not sure. There's also some false positives, these aren't that important but make sure they don't increase when you reclassify.

comment:24 Changed 10 years ago by dap

Found that one of the full waveform flightline segments for lines 2227 and 2243 is significantly smaller than the equivalent discrete flightline segment. Will reprocess the full waveform data for these two segments.

Last edited 10 years ago by dap (previous) (diff)

comment:25 Changed 10 years ago by dap

LiDAR data now ready for delivery check (BAS13_01-062-lidar-20140905).

comment:26 Changed 10 years ago by lah

Lidar Starting delivery check

comment:27 Changed 10 years ago by lah

Lidar DC
Classification needs improving - a lot of cloud is missed, particularly near ground (and under!) I am looking at line 17 because there are odd features in the dem screenshot which look as though cloud may be contributing to the dem when the script is supposed to only use ground values. Is it better just to classify ground as noise if there is so much cloud around it, which makes it hard to distinguish between cloud and trees?

Last edited 10 years ago by lah (previous) (diff)

comment:28 Changed 10 years ago by dap


Will recreate DEM so that noise is excluded

comment:29 Changed 10 years ago by lah

Lidar DC -FW
Looking at FW on windows machine. line 1 seems to be missing waveforms at start of file (approx 100000 records), but lines 2-4 seem fine. Only space on windows machine to check 5 at a time, so this could take a while...

Last edited 10 years ago by lah (previous) (diff)

comment:30 Changed 10 years ago by dap

Recreated DEMs, DEM screenshot and Read Me with new DEM image.

comment:31 Changed 10 years ago by lah

Lidar DC

  • Moved old files to processing directory
  • Corrected dem names to 062 & -bng
  • New dem is much better
  • Should logsheet have RCD & Grimm ticked?
  • changed dem hdr resolution to 2.0
  • Moved las1.0 to discrete_laser/las1.2
  • odd to to original line identifiers in ReadMe elevation table, but not a problem
  • screenshots look ok apart from cloud
  • Can't run demcompare [Errno 28} No space left on device
  • There are still roll offsets errors between 63 & 62. Pitch is bad for all (up to 0.5m offset)
  • classification as before, most lines are ok.
  • FW look ok in wave viewer, but files 1, 9, 10, 15, 21 and 22 missing waveforms for approx first 100000 records. There doesn't seem to be this problem before classification, so could be a problem with the fw classification process.

Have a think about these issues before resubmitting, particularly the missing waveforms.

comment:32 Changed 10 years ago by lah

Lidar DC
Also an error in zipping FW-13, FW-18 (but for point 255 not 56) , FW-20 (44)
WARNING: gap in waveform offsets.
WARNING: last offset plus size was 60 but new offset is -196 (for point 1)
ERROR: waveform offsets not in monotonically increasing order.
ERROR: last offset was -196 but new offset is 60 (for point 56)
ERROR: use option '-waveforms_with_map' to compress.

Last edited 10 years ago by lah (previous) (diff)

comment:33 Changed 10 years ago by dap

Ran -d dem/BAS13_01-2014_062-lidar_ASTER-bng.dem --lidar flightlines/discrete_laser/ascii/ -l UKBNG -n navigation/BAS13_01-2014_062.sol on a spare machine with sufficient space available in /tmp. Returned the following results:

Difference statistics:
Min:		-98.5019968261719
Max:		50.1992563476563
Sum:		-756112.419537204
Mean:		-4.99070928514893
Median:		-4.93109
Absolute mean:	7.04297277629906
Std deviation:	7.39447696749445
Total cells:	3088804
Non-null cells:	151504

Unmasked statistics:
Min:		-98.5019968261719
Max:		168.027332000732
Sum:		-763134.61983659
Mean:		-0.264170569333092
Median:		2.0752e-06
Absolute mean:	0.658307806970798
Std deviation:	2.38351970090949
Total cells:	3088804
Non-null cells:	2888795

comment:34 Changed 10 years ago by dap

Currently reprocessing file 01 (222719) with fence (max X 247000, max Y 113000, min X 237000, min Y 97800). Will then check las1.3 output in Wave Viewer to check that ALSPP is producing waveforms as expected. If so, will copy classification from the las1.2 file to the las1.3 file and reinspect the las1.3 file in Wave Viewer.

comment:35 Changed 10 years ago by dap

Reprocessed (unclassified) las1.3 file 01 (222719) has no waveforms for the first ~120,000 records when viewed in Wave Viewer.

comment:36 Changed 10 years ago by dap

Spoken to mark1 regarding missing records in waveform files - this is a known issue with las1.3 files. Will now investigate laszip issue with files 13, 18 and 20.

comment:37 Changed 10 years ago by dap

Edited logsheet so that RCD is checked but left GRIMM unchecked as the instrument crashed and all data was lost.

comment:38 Changed 9 years ago by dap

LiDAR Processing

Have investigated roll and pitch offset errors, new values are as follows:

Flightline Roll Error Pitch Error Heading Error
Last edited 9 years ago by dap (previous) (diff)

comment:39 Changed 9 years ago by dap

LiDAR Processing

tec checked new roll and pitch error values and suggested further corrections needed to be made to 232549 and 233904. I've edited my previous comment with the new values.

comment:40 Changed 9 years ago by dap

LiDAR Processing

All discrete files have been classified and copied to new delivery directory, currently converting the las1.2 files to ASCII and splitting the full waveform files.

comment:41 Changed 9 years ago by dap

LiDAR Processing

LiDAR data have been completely reprocessed since the last delivery check and a new delivery has been created. New roll and pitch offset error values were checked by tec.

Additional things to note:

  • I have moved previous processing from processing/als50 to processing/als50/stgo. I will remove this after the data have been delivered.
  • All processing for this delivery has been output to processing/als50/dap.
  • Files in this delivery were also kept in the processing/als50/dap directory.
  • alsproc config files were generated by me, dac and stgo but these resulted in strange wobbles at the ends of some of the lines. I have deleted the alsproc generated files but kept the config files and put them in processing/als50/alsproc_config/.
  • The majority of the files were split in to three "manageable chunks", apart from 001050 (left at its original length), 001827 (split in to two). Where splits were necessary, an overlap of 10 seconds was used.
  • Lines 2355 and 0045 were not processed (and therefore not included in the delivery) as they suffered from complete cloud coverage, containing no useful data.

Delivery at BAS13_01-062-lidar-20141105/ is now ready for delivery check.

Last edited 9 years ago by dap (previous) (diff)

comment:42 Changed 9 years ago by lah

Lidar Delivery Check complete

  • converted ascii files.
  • Roll & pitch good for all lines.
  • A few missed noise patches, but good enough overall. Changed dem header resolution to 2.0. demcompare produces similar results.
  • Some lines are missing waveforms at start (7, 10, 13, 20, 22) and some at end (9, 18, 21, 24) of line. Can't do much about ALSProc error, but should it be mentioned in read-me?
  • Minor typo in read-me (in to should be into).
  • Error zipping fw lines 3, 5 & 18 : "gap in waveform offsets".

If we can't do anything about the fw errors, then it's ready to deliver.

comment:43 Changed 9 years ago by dap


I've tidied up the processing directory by removing the following directories:

  • processing/als50/dac
  • processing/als50/stgo
  • processing/dac

I've also moved the content of processing/als50/dap to processing/als50 and removed processing/als50/dap and deleted all LaTeX .log, .aux and .out files from the top level procesing directory.

All files in las-unsplit/ and las-split/ are unclassified as the splitting was done before classification.

Currently changing the full waveform LiDAR data quality report to include statement about missing waveforms. I've changed the typos in the readme.

comment:44 Changed 9 years ago by dap


In case needed for future use, the fence coordinates used to split the lines were as follows:

Flightline Segment Min X Max X Min Y Max Y
222719a 238055 244243 97931 112659
222719b 243622 250002 83546 98679
222719c 249365 255513 68000 84346
224301a 249311 255391 69255 83872
224301b 243592 249902 83075 98182
224301c 237963 244180 97377 112333
225706a 237507 243619 98020 112536
225706b 243008 249340 83649 98730
225706c 248730 255024 69489 84440
231240a 248703 254710 69548 83858
231240b 243078 249312 83049 97967
231240c 237434 243693 97121 112092
232549a 237029 243173 97667 112100
232549b 242550 248856 83526 98140
232549c 248230 254375 69701 84259
233904a 248203 254258 69277 83621
233904b 242593 248807 82817 97807
233904c 236971 243165 96993 111889
001827a 230150 234619 73570 84227
001827b 233967 238414 64034 74365
002716a 234513 239056 63333 73898
002716b 230522 235114 73122 84042
002716c 226671 231110 83253 93730

I have also replaced doc/full_waveform_lidar.pdf in the latest delivery directory with the updated one, which mentions the missing waveforms, and removed the previous delivery directories.

I've tried reprocessing one of the files that produced the "gap in waveform offsets" error from laszip using ALSPP. The reprocessed file gave the same error when running laszip on it.

BAS13_01-062-lidar-20141105/ now ready for delivery.

comment:45 Changed 9 years ago by dap

Beginning archiving for delivery to NEODC.

comment:46 Changed 9 years ago by dap

  • Component changed from Processing: general to Archiving
  • Description modified (diff)
  • Other processors set to stgo, dap
  • Owner set to benj

LiDAR data at BAS13_01-062-lidar-20141105/ has now been submitted to NEODC, 13/11/2014.

Note: See TracTickets for help on using tickets.