RG12/09, flight day 088/2014, Wessex

[stgo]

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/uk/RG12_09-2014_088_Wessex

Data arrived from ARSF via network transfer on 31/03/2014

Scientific objective: Unknown

Priority: Alpha 5 - high

PI: France Gerrard

Sensors:

• Fenix (requested), Delivered 08/10/2014, Submitted to NEODC 21/04/2015
• Leica LIDAR (requested), Delivered 08/10/2014, Submitted to NEODC 21/04/2015
• RCD (requested), Delivered 08/10/2014, Submitted to NEODC 21/04/2015

[stgo]

Navigation processing started, basestation ppp results:

Lat	51 24 15.45178
Long	-1 30 50.16899
Ell. Height	183.434

[stgo]

Solution using ipas honeywell is poor, will continue trying to improve it but I'm going to process the alternate navigation at the same time to see if that is better.

[stgo]

Got 10 cm or less position accuracy on the flightline area using ipas honeywell.

[stgo]

Initial inspection of the flightlines indicates a uniform roll error. Testing the first 4 lines with new values.

[benj]

Starting Fenix processing

[benj]

There are fifteen lines in the logsheet but only ten Fenix data files (and line 10 has a raw file but nothing else). Instrument may have failed.

[benj]

...actually they are all there, just the file numbering is slightly confused (logsheet says the instrument crashed, I assume hence the slightly messed-up numbering and presumably the extra partial raw file).

[stgo]

Values used for lines 104713 and 110158:

Roll	-0.004001414
Pitch	0.00113716
Heading	0.00056484

[benj]

Fenix data do not contain dark frames. We had hoped to use dark frames from a different flight (similar to what we did for VOCALS in 2009) but looks like this is not a valid approach for Fenix at least. Checked dark frames from day 2014-152a (some of which have the same settings) and the dark frames vary by a factor of 3-4, so cannot say what the appropriate dark values to use for these flights would be. Marking Fenix blocked until we come up with another way to deal with that (if we can).

[stgo]

Currently checking to see if there is in fact a variable roll error or if I was just over confident with my values for 104713 and 110158.

[stgo]

Values for 104713 and 110158 are not applicable to other flightlines, 111834 on to 120318 use -0.003918414. will continue getting values.

[stgo]

Summary of roll values for lines:

104713	-0.004001414
110158	-0.004001414
111834	-0.003918414
113247	-0.003918414
114837	-0.003918414
120318	-0.003918414
121714	-0.003918414
123103	-0.003840414
124534	-0.003918414
130021	-0.003840414
131301	-0.003918414
132548	-0.003840414
133714	-0.004058414
134912	-0.003748414
140505	-0.003848414

Now classifying.

[benj]

Updated arrival date and location in ticket header - hadn't been entered to start with.

[stgo]

wrong ticket.

[stgo]

To split the lines as requested I have used the following x bounds in alspp processing:

Part	min x	max x
part a	414400	999999
part b	399300	415235
part c	0	400100

This has divided the waveform into much smaller chunks.

[benj]

Found the SCT offsets for Fenix:

Line	Offset (s)
1	0.95
2	0.93
3	0.95
4	0.96
5	0.95
6	0.95
7	0.96
8	0.94
9	-0.07
10	1.97
11	0.95
12	-0.05
13	0.94
14	0.95
15	0.95
16	0.95

Line 10 has no nav file - original file naming actually seems to have reset part way through line 10 (originally had 88-14-10.raw with 88-14.1-1.hdr and 88-14.1-1.log, file size for the header file matches the raw file).

[benj]

SCT offset for line 10 is 1.97s (will amend above comment to reflect this). Note that I think this is a dud line anyway - it isn't mentioned on the logsheet and basically only covers a curve between lines 9 and 11 anyway - the only straight bits are on the exit of line 9 and entry of line 11.

[stgo]

Beginning RCD processing, Lidar delivery still being created.

[stgo]

RCD processing complete, ready for DC. I've had to remove images 1 to 6 because of overexposure and a couple that were outbounds. Otherwise the data is fine.

[stgo]

Lidar delivery ready for DC.

[benj]

Creation of final flightlines to be delivered is stuck, latest version of APL is cutting a chunk off the end of the line and sticking it on next to the flightline (presumably further along the nav?). Previous version didn't do this. See attached screenshot (088_Wessex_georef_weirdness.png).

[benj]

Incorrect georeferencing illustration

[knpa]

Beginning RCD delivery check.

[knpa]

Check complete. No Problems found - RCD ready to deliver.

[knpa]

Beginning LiDAR delivery check.

[knpa]

Got to step nine and found significant problem, so stopping here.

• too many false positives on classification. If they get rid of class 7 they will loose half of the shrubbery. Need to be careful with isolated points function of classify_las. I usually classify those above and below the highest and lowest terrain points and use a very strict isolated points function.

Also found the following minor problems:

• Directory says las1.0 - don't we deliver 1.1 or 1.2 now? May have to update ReadMe too.
• No vectors included - have we emailed about this?
• There's a tif version of the intensity in the screenshots - I don't think this should be here
• Missing "full_waveform.pdf" from doc dir. I'm not sure if we actually need this, check with mark1 and take it off the file structure wiki page if don't
• Discrete laser ascii files do not have CRLF terminators. Check with mark1 if they need them and if they don't then add that as an exception to the DC wiki

Fix the above and submit for delivery check again.

[mark1]

Edited one of the specim nav files (-12) to remove a spurious letter in the frame number of one of the SPTSMP messages.

[dap]

Continuing Fenix processing from where benj left off, with SCT values as follows:

Flightline	FENIX
1	0.98
2	0.96
3	0.98
4	0.99
5	0.98
6	0.98
7	0.99
8	0.97
9	-0.04
10	2.00
11	0.98
12	-0.02
13	0.97
14	0.98
15	0.98
16	0.98

[dap]

Creating delivery for Fenix data.

[dap]

Fenix delivery created. The following steps need to be addressed before delivery checking (need to wait for return of stgo / benj):

• The percentage error value in the data quality remarks section needs to be amended (see large, bold reminder in the read me).
• Data needs to be split in to three parts, as with the LiDAR data, as requested by PI.
• SWIR bands need to be removed.

Also, an extra line was processed and wasn't in the logsheet. The read me file was amended to accommodate for this.

[dap]

SWIR bands have been removed. SWIR band files and original unsplit files have been moved to the relevant processing/hyperspectral/flightlines directories. The only .bil files that remain in the hyperspectral delivery are the VNIR files.

Files now need to be split in to three parts, as per PI's request. Read me may need to be changed to reflect this when done.

Percentage error value in readme is still being investigated but delivery is not to be delayed by this. This has therefore been noted in the readme.

[dap]

Reprocessing the lines in order to split them in to three parts. Will need to split VNIR and SWIR bands when this is complete (didn't realise splitting the lines in to three parts required reprocessing).

[stgo]

Lidar delivery recreated, now ready for delivery check.

[dap]

Found that the tifs generated when the files were split in to three parts appeared to have wobbles, so changed the config file so that the runscript_start_chain is nav2 and resubmitted to the grid. There is no need to split the SWIR bands off again since the already-split VNIR files were submitted to the grid.

[knpa]

Beginning LiDAR delivery check

[knpa]

Another failure, classification still has too many false positives. Just looking at the first line you can can see many hedgerows containing hundreds of false noise.

Did all other checks and got following minor errors:

• "Line" in e.g. Line 038 in "logsheet flight name" is unnecesary. The name is just 038.
• ReadMe says data is supplied as 1.0 point clouds when it is atually 1.2 (according to the directory name). It says this in a few diff places, including the ReadMe contents section.
• RG12_09-2014-088.(sol|sup) should be RG12_09-2014_088.(sol|sup) (i.e. _ instead of -)
• Screenshot dir contains a tif - should only contain jpgs
• "The data has been split during processing. Not that estimates of pointcloud sizes and noise data were made before splitting". I'd explain in more detail, e.g. lines were split into 3 equal parts to reduced file sizes. Also, do you mean "Note" rather than "Not". Also also, what does this second sentence mean? The per flight line statistics are for each part so were done AFTER splitting.

[stgo]

Recreated classified files with lower noise levels again.

Also completely remade readme, reflects comments above. Lidar ready for DC.

[knpa]

Beginning LiDAR delivery check again

[knpa]

LiDAR delivery check complete

All looks fine apart from the following:

• A couple of the FW lines seem strangely small. 05 and 25 are both ~1.5G when most other lines are ~7G. These lines are not smaller for the 1.2 or ascii versions
• There are individual screenshots for each line - we only supply the mosaic
• "The data has been split into 3 parts" should be "Each flightline has been split into three parts"
• unix2dos the ascii files
• Remove old LiDAR delivery if it was not sent out and not needed otherwise

[stgo]

I've corrected the readme and included a comment about the fullwaveform. I had already investigated this and concluded it was incomplete coverage of the data.

The per line screenshots are now included in the delivery. Currently unix2dos'ing the ascii

[stgo]

Finish unix2dos, lidar ready for recheck.

[stgo]

Lidar ready for delivery.

[dap]

Recreating hyperspectral delivery.

[dap]

Hyperspectral Processing

Found geotiff files to have wobbles, having looked at screenshots. Will process data again.

[dap]

Hyperspectral Processing

One of the timestamps in the raw .nav file for flight line -2 had an erroneous SPTSMP timestamp sentence (non-numeric). Fixed by changing $SPTSMP,207,4094+,1*38 to $SPTSMP,207,4094,1*38.

[dap]

Hyperspectral Processing

SWIR bands have now been stripped from the relevant bil files in the delivery directory and unsplit versions moved to the processing directories. Split files in the delivery directory have also been renamed to adhere to naming conventions so note that the bil files in the delivery directory do not contain SWIR bands.

[dap]

Hyperspectral Processing

Delivery at RG12_09-088-hyperspectral-20140919/ is now ready for delivery check.

Pay particular attention to the fact that the lines were split in to smaller, manageable chunks then had SWIR bands removed, which involved having to edit the bil header files and xml files (not part of the standard processing procedures).

[tec]

Hyperspectral Delivery Check
Starting DC

[tec]

3 Hyperspectral deliveries, checking the latest one.

• Converted sensor_FOV_vectors/fenix_viewvector_list.txt to windows line terminators
• Redo readme, commands in config refered to nextmap dems, I fixed config but you need to regenerate readme.
• Only 348 bands but SWIR cut so its fine
• Heavy noise from bands 1-8 all flights
• Data quality "While the data have been corrected" would "While the data has been corrected" sound better?

[dap]

Hyperspectral

• I edited the .tex file so the correct commands were in the file but had forgotten to change it in the config file.
• Added a data quality remark in the read me about noise in bands 1-8.
• Left "While the data have been corrected" as original.

[dap]

Fenix, LiDAR and RCD data sent to Dr Gerard, 08/10/2014

[tec]

Archiving
Started

[tec]

Archiving
Removed 2 old delivery in processing directory as the one delivered is newer.
Removed all las files in processing directory as they are far too large and it was a mess, they can be recreated anyway.
Removed flightlines directory as it can be recreated and it was 2.7TB so definitely not rsyncing that.
Finished