BAS13/01, flight day 062/2014, Tamar

[knpa]

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/uk/BAS13_01-2014_062_Tamar

Data arrived from ARSF via network transfer on 10/03/2014

BAS13/01 Scientific objective: unknown

BAS13/01 Priority: ?

PI: France Gerard

Sensors:

LiDAR (Submitted to NEODC 13/11/2014)
FW LiDAR (Submitted to NEODC 13/11/2014)

[knpa]

France would like the flightlines divided into manageable chunks, 3 per flightline was suggested.

[stgo]

Basestation ppp complete:
lat 51 03 24.94210
long -4 11 58.74401
eh 68.972

[benj]

Gary has confirmed that the PI would like FW LiDAR data for this flight

[stgo]

LiDAR processing in progress, correcting for roll error.

[stgo]

pitch and roll values found for lines 2243 and 2227:

roll	-0.003969414
pitch	0.00125716

Testing against other lines.

edit:pitch and roll were the wrong way round

[stgo]

Pitch and roll values worked fine. Classifying points, large level of cloud in line 001050 onwards.

[stgo]

Line 004533 has complete cloud cover, this will not be delivered.

[stgo]

Having issues with waveform data on lines flown before midnight, LasHistoViewer claims they are not valid LAS 1.3 files. I am reprocessing one locally on the windows machine to see if this is a network error. I will split the files to check it is not a file size issue as they are considerably (>20 GB) larger than post midnight lines.

If I can't find answers will contact leica for next steps.

[stgo]

I have processed using a fence in ALSPP to make a smaller file, this takes a long time with waveform processing (1 hour per line!) and creates usable files. This suggests it is definitely a file size issue rather than a processing one. I am waiting on the windows reprocessing to confirm.

Splitting using lastools is not an option as it is not compatible with LAS 1.3. For the moment best way to split the files will be fencing with ALSPP, this will be time consuming.

[stgo]

Created delivery and updated with waveform files. Currently creating DEM files, first run produced sizes of 14 GB which are not very manageable. I am reproducing with half the tiles and a lower resolution.

[stgo]

I have created a secondary nav file focusing purely on the data location for dem cretion, produces a 500 Mb file which is much more manageable. I would recommend this process for other flights but will try to script a way to use nav gpstimes instead of bounding box/ as well as bounding for dem generation.

Creation of the dem has also highlighted some particularly bad cloud I must have missed in first run for classification. Will repair before continuing.

[stgo]

Line 235554 has an extremely high level of cloud and what little land can be seen is covered in haze. Will discard.

[stgo]

Line 2339 has not processed through the classification script, will find out why and repair.

[stgo]

I've found variable roll errors in a couple of lines, will correct.

[stgo]

Corrected roll errors:

222719: -0.004200413
224301: -0.003850413
001050: -0.003989414
001827: -0.003989414
002716: -0.003700413

225706 to 233904: -0.004049414

Now processing the full waveform data over this weekend.

[stgo]

This dataset has been split, lines 0010 and 0018 (21, 22 and 23 in the delivery) were smaller than the others. 0010 has not been split. 0018 has been split into 2 rather than 3 parts.

Some lines have not been included because of complete cloud coverage (2355 0045) and it should be noted some of the other lines suffer from high cloud coverage.

Waveform files are already split and classified in las-classified/fw_class/{a..c} and a delivery has already been produced.

The dems and trj files have already been created, all that is needed is a readme and it will be good to go.

[dap]

LiDAR Processing

Taken over processing from stgo. Continuing by filling in information in Readme config file and creating readme.

[dap]

LiDAR Processing

Found problem with las1.0 files in delivery - part of two of the flightlines (224301 and 001827) is missing. Will reprocess these flightlines in alspp using roll and pitch values above.

edit: Two files were missing

[dap]

LiDAR Processing

Re-classified point clouds and re-generated screenshots due to reprocessing of lines. Created read me for LiDAR data.

LiDAR data now ready for delivery.

[dap]

Edit: wrong ticket

[knpa]

Beginning LiDAR delivery check.

[knpa]

Major fail on classification.

Just looking at the first line, there's too much missed noise. Ask me to show you where if you're not sure. There's also some false positives, these aren't that important but make sure they don't increase when you reclassify.

[dap]

Found that one of the full waveform flightline segments for lines 2227 and 2243 is significantly smaller than the equivalent discrete flightline segment. Will reprocess the full waveform data for these two segments.

[dap]

LiDAR data now ready for delivery check (BAS13_01-062-lidar-20140905).

[lah]

Lidar Starting delivery check

[lah]

Lidar DC
Classification needs improving - a lot of cloud is missed, particularly near ground (and under!) I am looking at line 17 because there are odd features in the dem screenshot which look as though cloud may be contributing to the dem when the script is supposed to only use ground values. Is it better just to classify ground as noise if there is so much cloud around it, which makes it hard to distinguish between cloud and trees?

[dap]

LiDAR

Will recreate DEM so that noise is excluded

[lah]

Lidar DC -FW
Looking at FW on windows machine. line 1 seems to be missing waveforms at start of file (approx 100000 records), but lines 2-4 seem fine. Only space on windows machine to check 5 at a time, so this could take a while...

[dap]

Recreated DEMs, DEM screenshot and Read Me with new DEM image.

[lah]

Lidar DC

• Moved old files to processing directory
• Corrected dem names to 062 & -bng
• New dem is much better
• Should logsheet have RCD & Grimm ticked?
• changed dem hdr resolution to 2.0
• Moved las1.0 to discrete_laser/las1.2
• odd to to original line identifiers in ReadMe elevation table, but not a problem
• screenshots look ok apart from cloud
• Can't run demcompare [Errno 28} No space left on device
• There are still roll offsets errors between 63 & 62. Pitch is bad for all (up to 0.5m offset)
• classification as before, most lines are ok.
• FW look ok in wave viewer, but files 1, 9, 10, 15, 21 and 22 missing waveforms for approx first 100000 records. There doesn't seem to be this problem before classification, so could be a problem with the fw classification process.

Have a think about these issues before resubmitting, particularly the missing waveforms.

[lah]

Lidar DC
Also an error in zipping FW-13
WARNING: gap in waveform offsets.
WARNING: last offset plus size was 60 but new offset is -196 (for point 1)
ERROR: waveform offsets not in monotonically increasing order.
ERROR: last offset was -196 but new offset is 60 (for point 56)
ERROR: use option '-waveforms_with_map' to compress.

[dap]

Ran demcompare.py -d dem/BAS13_01-2014_062-lidar_ASTER-bng.dem --lidar flightlines/discrete_laser/ascii/ -l UKBNG -n navigation/BAS13_01-2014_062.sol on a spare machine with sufficient space available in /tmp. Returned the following results:

Difference statistics:
Min:		-98.5019968261719
Max:		50.1992563476563
Sum:		-756112.419537204
Mean:		-4.99070928514893
Median:		-4.93109
Absolute mean:	7.04297277629906
Std deviation:	7.39447696749445
Total cells:	3088804
Non-null cells:	151504


Unmasked statistics:
Min:		-98.5019968261719
Max:		168.027332000732
Sum:		-763134.61983659
Mean:		-0.264170569333092
Median:		2.0752e-06
Absolute mean:	0.658307806970798
Std deviation:	2.38351970090949
Total cells:	3088804
Non-null cells:	2888795

[dap]

Currently reprocessing file 01 (222719) with fence (max X 247000, max Y 113000, min X 237000, min Y 97800). Will then check las1.3 output in Wave Viewer to check that ALSPP is producing waveforms as expected. If so, will copy classification from the las1.2 file to the las1.3 file and reinspect the las1.3 file in Wave Viewer.

[dap]

Reprocessed (unclassified) las1.3 file 01 (222719) has no waveforms for the first ~120,000 records when viewed in Wave Viewer.

[dap]

Spoken to mark1 regarding missing records in waveform files - this is a known issue with las1.3 files. Will now investigate laszip issue with files 13, 18 and 20.

[dap]

Edited logsheet so that RCD is checked but left GRIMM unchecked as the instrument crashed and all data was lost.

[dap]

LiDAR Processing

Have investigated roll and pitch offset errors, new values are as follows:

Flightline	Roll Error	Pitch Error	Heading Error
222719	-0.004200413	0.00125716	0.000564840
224301	-0.003850413	0.00125716	0.000564840
225706	-0.004049414	0.00110716	0.000564840
231240	-0.004049414	0.00128716	0.000564840
232549	-0.003829414	0.00110716	0.000564840
233904	-0.004210413	0.00128716	0.000564840
001050	-0.003989414	0.00110716	0.000564840
001827	-0.003989414	0.00120716	0.000564840
002716	-0.003700413	0.00110716	0.000564840

[dap]

LiDAR Processing

tec checked new roll and pitch error values and suggested further corrections needed to be made to 232549 and 233904. I've edited my previous comment with the new values.

[dap]

LiDAR Processing

All discrete files have been classified and copied to new delivery directory, currently converting the las1.2 files to ASCII and splitting the full waveform files.

[dap]

LiDAR Processing

LiDAR data have been completely reprocessed since the last delivery check and a new delivery has been created. New roll and pitch offset error values were checked by tec.

Additional things to note:

• I have moved previous processing from processing/als50 to processing/als50/stgo. I will remove this after the data have been delivered.
• All processing for this delivery has been output to processing/als50/dap.
• Files in this delivery were also kept in the processing/als50/dap directory.
• alsproc config files were generated by me, dac and stgo but these resulted in strange wobbles at the ends of some of the lines. I have deleted the alsproc generated files but kept the config files and put them in processing/als50/alsproc_config/.
• The majority of the files were split in to three "manageable chunks", apart from 001050 (left at its original length), 001827 (split in to two). Where splits were necessary, an overlap of 10 seconds was used.
• Lines 2355 and 0045 were not processed (and therefore not included in the delivery) as they suffered from complete cloud coverage, containing no useful data.

Delivery at BAS13_01-062-lidar-20141105/ is now ready for delivery check.

[lah]

Lidar Delivery Check complete

• converted ascii files.
• Roll & pitch good for all lines.
• A few missed noise patches, but good enough overall. Changed dem header resolution to 2.0. demcompare produces similar results.
• Some lines are missing waveforms at start (7, 10, 13, 20, 22) and some at end (9, 18, 21, 24) of line. Can't do much about ALSProc error, but should it be mentioned in read-me?
• Minor typo in read-me (in to should be into).
• Error zipping fw lines 3, 5 & 18 : "gap in waveform offsets".

If we can't do anything about the fw errors, then it's ready to deliver.

[dap]

LiDAR

I've tidied up the processing directory by removing the following directories:

• processing/als50/dac
• processing/als50/stgo
• processing/dac

I've also moved the content of processing/als50/dap to processing/als50 and removed processing/als50/dap and deleted all LaTeX .log, .aux and .out files from the top level procesing directory.

All files in las-unsplit/ and las-split/ are unclassified as the splitting was done before classification.

Currently changing the full waveform LiDAR data quality report to include statement about missing waveforms. I've changed the typos in the readme.

[dap]

LiDAR

In case needed for future use, the fence coordinates used to split the lines were as follows:

Flightline Segment	Min X	Max X	Min Y	Max Y
222719a	238055	244243	97931	112659
222719b	243622	250002	83546	98679
222719c	249365	255513	68000	84346
224301a	249311	255391	69255	83872
224301b	243592	249902	83075	98182
224301c	237963	244180	97377	112333
225706a	237507	243619	98020	112536
225706b	243008	249340	83649	98730
225706c	248730	255024	69489	84440
231240a	248703	254710	69548	83858
231240b	243078	249312	83049	97967
231240c	237434	243693	97121	112092
232549a	237029	243173	97667	112100
232549b	242550	248856	83526	98140
232549c	248230	254375	69701	84259
233904a	248203	254258	69277	83621
233904b	242593	248807	82817	97807
233904c	236971	243165	96993	111889
001827a	230150	234619	73570	84227
001827b	233967	238414	64034	74365
002716a	234513	239056	63333	73898
002716b	230522	235114	73122	84042
002716c	226671	231110	83253	93730

I have also replaced doc/full_waveform_lidar.pdf in the latest delivery directory with the updated one, which mentions the missing waveforms, and removed the previous delivery directories.

I've tried reprocessing one of the files that produced the "gap in waveform offsets" error from laszip using ALSPP. The reprocessed file gave the same error when running laszip on it.

BAS13_01-062-lidar-20141105/ now ready for delivery.

[dap]

Beginning archiving for delivery to NEODC.

[dap]

LiDAR data at BAS13_01-062-lidar-20141105/ has now been submitted to NEODC, 13/11/2014.