Opened 5 years ago

Closed 4 years ago

#520 closed flight processing (fixed)

RG12/09, flight day 088/2014, Wessex

Reported by: stgo Owned by: dap
Priority: alpha 5 Milestone: 2014 data processing completion
Component: Archiving Keywords:
Cc: Other processors:

Description (last modified by tec)

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/uk/RG12_09-2014_088_Wessex

Data arrived from ARSF via network transfer on 31/03/2014

Scientific objective: Unknown

Priority: Alpha 5 - high

PI: France Gerrard

Sensors:

  • Fenix (requested), Delivered 08/10/2014, Submitted to NEODC 21/04/2015
  • Leica LIDAR (requested), Delivered 08/10/2014, Submitted to NEODC 21/04/2015
  • RCD (requested), Delivered 08/10/2014, Submitted to NEODC 21/04/2015

Attachments (1)

088_Wessex_georef_weirdness.png (442.3 KB) - added by benj 5 years ago.
Incorrect georeferencing illustration

Download all attachments as: .zip

Change History (54)

comment:1 Changed 5 years ago by stgo

Navigation processing started, basestation ppp results:

Lat 51 24 15.45178
Long -1 30 50.16899
Ell. Height 183.434

comment:2 Changed 5 years ago by stgo

Solution using ipas honeywell is poor, will continue trying to improve it but I'm going to process the alternate navigation at the same time to see if that is better.

comment:3 Changed 5 years ago by stgo

Got 10 cm or less position accuracy on the flightline area using ipas honeywell.

comment:4 Changed 5 years ago by stgo

Initial inspection of the flightlines indicates a uniform roll error. Testing the first 4 lines with new values.

comment:5 Changed 5 years ago by benj

Starting Fenix processing

comment:6 Changed 5 years ago by benj

There are fifteen lines in the logsheet but only ten Fenix data files (and line 10 has a raw file but nothing else). Instrument may have failed.

comment:7 Changed 5 years ago by benj

...actually they are all there, just the file numbering is slightly confused (logsheet says the instrument crashed, I assume hence the slightly messed-up numbering and presumably the extra partial raw file).

comment:8 Changed 5 years ago by stgo

Values used for lines 104713 and 110158:

Roll -0.004001414
Pitch 0.00113716
Heading 0.00056484

comment:9 Changed 5 years ago by benj

Fenix data do not contain dark frames. We had hoped to use dark frames from a different flight (similar to what we did for VOCALS in 2009) but looks like this is not a valid approach for Fenix at least. Checked dark frames from day 2014-152a (some of which have the same settings) and the dark frames vary by a factor of 3-4, so cannot say what the appropriate dark values to use for these flights would be. Marking Fenix blocked until we come up with another way to deal with that (if we can).

comment:10 Changed 5 years ago by stgo

Currently checking to see if there is in fact a variable roll error or if I was just over confident with my values for 104713 and 110158.

comment:11 Changed 5 years ago by stgo

Values for 104713 and 110158 are not applicable to other flightlines, 111834 on to 120318 use -0.003918414. will continue getting values.

comment:12 Changed 5 years ago by stgo

Summary of roll values for lines:

104713 -0.004001414
110158 -0.004001414
111834 -0.003918414
113247 -0.003918414
114837 -0.003918414
120318 -0.003918414
121714 -0.003918414
123103 -0.003840414
124534 -0.003918414
130021 -0.003840414
131301 -0.003918414
132548 -0.003840414
133714 -0.004058414
134912 -0.003748414
140505 -0.003848414

Now classifying.

comment:13 Changed 5 years ago by benj

  • Description modified (diff)

Updated arrival date and location in ticket header - hadn't been entered to start with.

comment:14 Changed 5 years ago by stgo

wrong ticket.

Last edited 5 years ago by stgo (previous) (diff)

comment:15 Changed 5 years ago by stgo

To split the lines as requested I have used the following x bounds in alspp processing:

Part min x max x
part a 414400 999999
part b 399300 415235
part c 0 400100

This has divided the waveform into much smaller chunks.

comment:16 Changed 5 years ago by benj

Found the SCT offsets for Fenix:

Line Offset (s)
1 0.95
2 0.93
3 0.95
4 0.96
5 0.95
6 0.95
7 0.96
8 0.94
9 -0.07
10 1.97
11 0.95
12 -0.05
13 0.94
14 0.95
15 0.95
16 0.95

Line 10 has no nav file - original file naming actually seems to have reset part way through line 10 (originally had 88-14-10.raw with 88-14.1-1.hdr and 88-14.1-1.log, file size for the header file matches the raw file).

Last edited 5 years ago by benj (previous) (diff)

comment:17 Changed 5 years ago by benj

SCT offset for line 10 is 1.97s (will amend above comment to reflect this). Note that I think this is a dud line anyway - it isn't mentioned on the logsheet and basically only covers a curve between lines 9 and 11 anyway - the only straight bits are on the exit of line 9 and entry of line 11.

comment:18 Changed 5 years ago by stgo

Beginning RCD processing, Lidar delivery still being created.

comment:19 Changed 5 years ago by stgo

RCD processing complete, ready for DC. I've had to remove images 1 to 6 because of overexposure and a couple that were outbounds. Otherwise the data is fine.

comment:20 Changed 5 years ago by stgo

Lidar delivery ready for DC.

Last edited 5 years ago by stgo (previous) (diff)

comment:21 Changed 5 years ago by benj

Creation of final flightlines to be delivered is stuck, latest version of APL is cutting a chunk off the end of the line and sticking it on next to the flightline (presumably further along the nav?). Previous version didn't do this. See attached screenshot (088_Wessex_georef_weirdness.png).

Changed 5 years ago by benj

Incorrect georeferencing illustration

comment:22 Changed 5 years ago by knpa

Beginning RCD delivery check.

comment:23 Changed 5 years ago by knpa

Check complete. No Problems found - RCD ready to deliver.

comment:24 Changed 5 years ago by knpa

Beginning LiDAR delivery check.

comment:25 Changed 5 years ago by knpa

Got to step nine and found significant problem, so stopping here.

  • too many false positives on classification. If they get rid of class 7 they will loose half of the shrubbery. Need to be careful with isolated points function of classify_las. I usually classify those above and below the highest and lowest terrain points and use a very strict isolated points function.

Also found the following minor problems:

  • Directory says las1.0 - don't we deliver 1.1 or 1.2 now? May have to update ReadMe too.
  • No vectors included - have we emailed about this?
  • There's a tif version of the intensity in the screenshots - I don't think this should be here
  • Missing "full_waveform.pdf" from doc dir. I'm not sure if we actually need this, check with mark1 and take it off the file structure wiki page if don't
  • Discrete laser ascii files do not have CRLF terminators. Check with mark1 if they need them and if they don't then add that as an exception to the DC wiki

Fix the above and submit for delivery check again.

comment:26 Changed 5 years ago by mark1

Edited one of the specim nav files (-12) to remove a spurious letter in the frame number of one of the SPTSMP messages.

comment:27 Changed 5 years ago by dap

Continuing Fenix processing from where benj left off, with SCT values as follows:

Flightline FENIX
1 0.98
2 0.96
3 0.98
4 0.99
5 0.98
6 0.98
7 0.99
8 0.97
9 -0.04
10 2.00
11 0.98
12 -0.02
13 0.97
14 0.98
15 0.98
16 0.98
Last edited 5 years ago by dap (previous) (diff)

comment:28 Changed 5 years ago by dap

Creating delivery for Fenix data.

comment:29 Changed 5 years ago by dap

Fenix delivery created. The following steps need to be addressed before delivery checking (need to wait for return of stgo / benj):

  • The percentage error value in the data quality remarks section needs to be amended (see large, bold reminder in the read me).
  • Data needs to be split in to three parts, as with the LiDAR data, as requested by PI.
  • SWIR bands need to be removed.

Also, an extra line was processed and wasn't in the logsheet. The read me file was amended to accommodate for this.

comment:30 Changed 5 years ago by dap

  • Owner set to dap

comment:31 Changed 5 years ago by dap

SWIR bands have been removed. SWIR band files and original unsplit files have been moved to the relevant processing/hyperspectral/flightlines directories. The only .bil files that remain in the hyperspectral delivery are the VNIR files.

Files now need to be split in to three parts, as per PI's request. Read me may need to be changed to reflect this when done.

Percentage error value in readme is still being investigated but delivery is not to be delayed by this. This has therefore been noted in the readme.

comment:32 Changed 5 years ago by dap

Reprocessing the lines in order to split them in to three parts. Will need to split VNIR and SWIR bands when this is complete (didn't realise splitting the lines in to three parts required reprocessing).

comment:33 Changed 5 years ago by stgo

Lidar delivery recreated, now ready for delivery check.

comment:34 Changed 5 years ago by dap

Hyperspectral

Found that the tifs generated when the files were split in to three parts appeared to have wobbles, so changed the config file so that the runscript_start_chain is nav2 and resubmitted to the grid. There is no need to split the SWIR bands off again since the already-split VNIR files were submitted to the grid.

Last edited 5 years ago by dap (previous) (diff)

comment:35 Changed 5 years ago by knpa

Beginning LiDAR delivery check

comment:36 Changed 4 years ago by knpa

Another failure, classification still has too many false positives. Just looking at the first line you can can see many hedgerows containing hundreds of false noise.

Did all other checks and got following minor errors:

  • "Line" in e.g. Line 038 in "logsheet flight name" is unnecesary. The name is just 038.
  • ReadMe says data is supplied as 1.0 point clouds when it is atually 1.2 (according to the directory name). It says this in a few diff places, including the ReadMe contents section.
  • RG12_09-2014-088.(sol|sup) should be RG12_09-2014_088.(sol|sup) (i.e. _ instead of -)
  • Screenshot dir contains a tif - should only contain jpgs
  • "The data has been split during processing. Not that estimates of pointcloud sizes and noise data were made before splitting". I'd explain in more detail, e.g. lines were split into 3 equal parts to reduced file sizes. Also, do you mean "Note" rather than "Not". Also also, what does this second sentence mean? The per flight line statistics are for each part so were done AFTER splitting.

comment:37 Changed 4 years ago by stgo

Recreated classified files with lower noise levels again.

Also completely remade readme, reflects comments above. Lidar ready for DC.

comment:38 Changed 4 years ago by knpa

Beginning LiDAR delivery check again

comment:39 Changed 4 years ago by knpa

LiDAR delivery check complete

All looks fine apart from the following:

  • A couple of the FW lines seem strangely small. 05 and 25 are both ~1.5G when most other lines are ~7G. These lines are not smaller for the 1.2 or ascii versions
  • There are individual screenshots for each line - we only supply the mosaic
  • "The data has been split into 3 parts" should be "Each flightline has been split into three parts"
  • unix2dos the ascii files
  • Remove old LiDAR delivery if it was not sent out and not needed otherwise

comment:40 Changed 4 years ago by stgo

I've corrected the readme and included a comment about the fullwaveform. I had already investigated this and concluded it was incomplete coverage of the data.

The per line screenshots are now included in the delivery. Currently unix2dos'ing the ascii

comment:41 Changed 4 years ago by stgo

Finish unix2dos, lidar ready for recheck.

comment:42 Changed 4 years ago by stgo

Lidar ready for delivery.

comment:43 Changed 4 years ago by dap

Recreating hyperspectral delivery.

comment:44 Changed 4 years ago by dap

Hyperspectral Processing

Found geotiff files to have wobbles, having looked at screenshots. Will process data again.

comment:45 Changed 4 years ago by dap

Hyperspectral Processing

One of the timestamps in the raw .nav file for flight line -2 had an erroneous SPTSMP timestamp sentence (non-numeric). Fixed by changing $SPTSMP,207,4094+,1*38 to $SPTSMP,207,4094,1*38.

comment:46 Changed 4 years ago by dap

Hyperspectral Processing

SWIR bands have now been stripped from the relevant bil files in the delivery directory and unsplit versions moved to the processing directories. Split files in the delivery directory have also been renamed to adhere to naming conventions so note that the bil files in the delivery directory do not contain SWIR bands.

comment:47 Changed 4 years ago by dap

Hyperspectral Processing

Delivery at RG12_09-088-hyperspectral-20140919/ is now ready for delivery check.

Pay particular attention to the fact that the lines were split in to smaller, manageable chunks then had SWIR bands removed, which involved having to edit the bil header files and xml files (not part of the standard processing procedures).

comment:48 Changed 4 years ago by tec

Hyperspectral Delivery Check
Starting DC

Last edited 4 years ago by tec (previous) (diff)

comment:49 Changed 4 years ago by tec

3 Hyperspectral deliveries, checking the latest one.

  • Converted sensor_FOV_vectors/fenix_viewvector_list.txt to windows line terminators
  • Redo readme, commands in config refered to nextmap dems, I fixed config but you need to regenerate readme.
  • Only 348 bands but SWIR cut so its fine
  • Heavy noise from bands 1-8 all flights
  • Data quality "While the data have been corrected" would "While the data has been corrected" sound better?

comment:50 Changed 4 years ago by dap

Hyperspectral

  • I edited the .tex file so the correct commands were in the file but had forgotten to change it in the config file.
  • Added a data quality remark in the read me about noise in bands 1-8.
  • Left "While the data have been corrected" as original.

comment:51 Changed 4 years ago by dap

  • Component changed from Processing: general to Archiving
  • Description modified (diff)

Fenix, LiDAR and RCD data sent to Dr Gerard, 08/10/2014

comment:52 Changed 4 years ago by tec

Archiving
Started

comment:53 Changed 4 years ago by tec

  • Description modified (diff)
  • Resolution set to fixed
  • Status changed from new to closed

Archiving
Removed 2 old delivery in processing directory as the one delivered is newer.
Removed all las files in processing directory as they are far too large and it was a mess, they can be recreated anyway.
Removed flightlines directory as it can be recreated and it was 2.7TB so definitely not rsyncing that.
Finished

Note: See TracTickets for help on using tickets.