Opened 11 years ago
Closed 10 years ago
#520 closed flight processing (fixed)
RG12/09, flight day 088/2014, Wessex
Reported by: | stgo | Owned by: | dap |
---|---|---|---|
Priority: | alpha 5 | Milestone: | 2014 data processing completion |
Component: | Archiving | Keywords: | |
Cc: | Other processors: |
Description (last modified by tec)
Data location: /users/rsg/arsf/arsf_data/2014/flight_data/uk/RG12_09-2014_088_Wessex
Data arrived from ARSF via network transfer on 31/03/2014
Scientific objective: Unknown
Priority: Alpha 5 - high
PI: France Gerrard
Sensors:
- Fenix (requested), Delivered 08/10/2014, Submitted to NEODC 21/04/2015
- Leica LIDAR (requested), Delivered 08/10/2014, Submitted to NEODC 21/04/2015
- RCD (requested), Delivered 08/10/2014, Submitted to NEODC 21/04/2015
Attachments (1)
Change History (54)
comment:1 Changed 10 years ago by stgo
comment:2 Changed 10 years ago by stgo
Solution using ipas honeywell is poor, will continue trying to improve it but I'm going to process the alternate navigation at the same time to see if that is better.
comment:3 Changed 10 years ago by stgo
Got 10 cm or less position accuracy on the flightline area using ipas honeywell.
comment:4 Changed 10 years ago by stgo
Initial inspection of the flightlines indicates a uniform roll error. Testing the first 4 lines with new values.
comment:5 Changed 10 years ago by benj
Starting Fenix processing
comment:6 Changed 10 years ago by benj
There are fifteen lines in the logsheet but only ten Fenix data files (and line 10 has a raw file but nothing else). Instrument may have failed.
comment:7 Changed 10 years ago by benj
...actually they are all there, just the file numbering is slightly confused (logsheet says the instrument crashed, I assume hence the slightly messed-up numbering and presumably the extra partial raw file).
comment:8 Changed 10 years ago by stgo
Values used for lines 104713 and 110158:
Roll | -0.004001414 |
Pitch | 0.00113716 |
Heading | 0.00056484 |
comment:9 Changed 10 years ago by benj
Fenix data do not contain dark frames. We had hoped to use dark frames from a different flight (similar to what we did for VOCALS in 2009) but looks like this is not a valid approach for Fenix at least. Checked dark frames from day 2014-152a (some of which have the same settings) and the dark frames vary by a factor of 3-4, so cannot say what the appropriate dark values to use for these flights would be. Marking Fenix blocked until we come up with another way to deal with that (if we can).
comment:10 Changed 10 years ago by stgo
Currently checking to see if there is in fact a variable roll error or if I was just over confident with my values for 104713 and 110158.
comment:11 Changed 10 years ago by stgo
Values for 104713 and 110158 are not applicable to other flightlines, 111834 on to 120318 use -0.003918414. will continue getting values.
comment:12 Changed 10 years ago by stgo
Summary of roll values for lines:
104713 | -0.004001414 |
110158 | -0.004001414 |
111834 | -0.003918414 |
113247 | -0.003918414 |
114837 | -0.003918414 |
120318 | -0.003918414 |
121714 | -0.003918414 |
123103 | -0.003840414 |
124534 | -0.003918414 |
130021 | -0.003840414 |
131301 | -0.003918414 |
132548 | -0.003840414 |
133714 | -0.004058414 |
134912 | -0.003748414 |
140505 | -0.003848414 |
Now classifying.
comment:13 Changed 10 years ago by benj
- Description modified (diff)
Updated arrival date and location in ticket header - hadn't been entered to start with.
comment:14 Changed 10 years ago by stgo
wrong ticket.
comment:15 Changed 10 years ago by stgo
To split the lines as requested I have used the following x bounds in alspp processing:
Part | min x | max x |
part a | 414400 | 999999 |
part b | 399300 | 415235 |
part c | 0 | 400100 |
This has divided the waveform into much smaller chunks.
comment:16 Changed 10 years ago by benj
Found the SCT offsets for Fenix:
Line | Offset (s) |
1 | 0.95 |
2 | 0.93 |
3 | 0.95 |
4 | 0.96 |
5 | 0.95 |
6 | 0.95 |
7 | 0.96 |
8 | 0.94 |
9 | -0.07 |
10 | 1.97 |
11 | 0.95 |
12 | -0.05 |
13 | 0.94 |
14 | 0.95 |
15 | 0.95 |
16 | 0.95 |
Line 10 has no nav file - original file naming actually seems to have reset part way through line 10 (originally had 88-14-10.raw with 88-14.1-1.hdr and 88-14.1-1.log, file size for the header file matches the raw file).
comment:17 Changed 10 years ago by benj
SCT offset for line 10 is 1.97s (will amend above comment to reflect this). Note that I think this is a dud line anyway - it isn't mentioned on the logsheet and basically only covers a curve between lines 9 and 11 anyway - the only straight bits are on the exit of line 9 and entry of line 11.
comment:18 Changed 10 years ago by stgo
Beginning RCD processing, Lidar delivery still being created.
comment:19 Changed 10 years ago by stgo
RCD processing complete, ready for DC. I've had to remove images 1 to 6 because of overexposure and a couple that were outbounds. Otherwise the data is fine.
comment:20 Changed 10 years ago by stgo
Lidar delivery ready for DC.
comment:21 Changed 10 years ago by benj
Creation of final flightlines to be delivered is stuck, latest version of APL is cutting a chunk off the end of the line and sticking it on next to the flightline (presumably further along the nav?). Previous version didn't do this. See attached screenshot (088_Wessex_georef_weirdness.png).
comment:22 Changed 10 years ago by knpa
Beginning RCD delivery check.
comment:23 Changed 10 years ago by knpa
Check complete. No Problems found - RCD ready to deliver.
comment:24 Changed 10 years ago by knpa
Beginning LiDAR delivery check.
comment:25 Changed 10 years ago by knpa
Got to step nine and found significant problem, so stopping here.
- too many false positives on classification. If they get rid of class 7 they will loose half of the shrubbery. Need to be careful with isolated points function of classify_las. I usually classify those above and below the highest and lowest terrain points and use a very strict isolated points function.
Also found the following minor problems:
- Directory says las1.0 - don't we deliver 1.1 or 1.2 now? May have to update ReadMe too.
- No vectors included - have we emailed about this?
- There's a tif version of the intensity in the screenshots - I don't think this should be here
- Missing "full_waveform.pdf" from doc dir. I'm not sure if we actually need this, check with mark1 and take it off the file structure wiki page if don't
- Discrete laser ascii files do not have CRLF terminators. Check with mark1 if they need them and if they don't then add that as an exception to the DC wiki
Fix the above and submit for delivery check again.
comment:26 Changed 10 years ago by mark1
Edited one of the specim nav files (-12) to remove a spurious letter in the frame number of one of the SPTSMP messages.
comment:27 Changed 10 years ago by dap
Continuing Fenix processing from where benj left off, with SCT values as follows:
Flightline | FENIX |
1 | 0.98 |
2 | 0.96 |
3 | 0.98 |
4 | 0.99 |
5 | 0.98 |
6 | 0.98 |
7 | 0.99 |
8 | 0.97 |
9 | -0.04 |
10 | 2.00 |
11 | 0.98 |
12 | -0.02 |
13 | 0.97 |
14 | 0.98 |
15 | 0.98 |
16 | 0.98 |
comment:28 Changed 10 years ago by dap
Creating delivery for Fenix data.
comment:29 Changed 10 years ago by dap
Fenix delivery created. The following steps need to be addressed before delivery checking (need to wait for return of stgo / benj):
- The percentage error value in the data quality remarks section needs to be amended (see large, bold reminder in the read me).
- Data needs to be split in to three parts, as with the LiDAR data, as requested by PI.
- SWIR bands need to be removed.
Also, an extra line was processed and wasn't in the logsheet. The read me file was amended to accommodate for this.
comment:30 Changed 10 years ago by dap
- Owner set to dap
comment:31 Changed 10 years ago by dap
SWIR bands have been removed. SWIR band files and original unsplit files have been moved to the relevant processing/hyperspectral/flightlines directories. The only .bil files that remain in the hyperspectral delivery are the VNIR files.
Files now need to be split in to three parts, as per PI's request. Read me may need to be changed to reflect this when done.
Percentage error value in readme is still being investigated but delivery is not to be delayed by this. This has therefore been noted in the readme.
comment:32 Changed 10 years ago by dap
Reprocessing the lines in order to split them in to three parts. Will need to split VNIR and SWIR bands when this is complete (didn't realise splitting the lines in to three parts required reprocessing).
comment:33 Changed 10 years ago by stgo
Lidar delivery recreated, now ready for delivery check.
comment:34 Changed 10 years ago by dap
Hyperspectral
Found that the tifs generated when the files were split in to three parts appeared to have wobbles, so changed the config file so that the runscript_start_chain is nav2 and resubmitted to the grid. There is no need to split the SWIR bands off again since the already-split VNIR files were submitted to the grid.
comment:35 Changed 10 years ago by knpa
Beginning LiDAR delivery check
comment:36 Changed 10 years ago by knpa
Another failure, classification still has too many false positives. Just looking at the first line you can can see many hedgerows containing hundreds of false noise.
Did all other checks and got following minor errors:
- "Line" in e.g. Line 038 in "logsheet flight name" is unnecesary. The name is just 038.
- ReadMe says data is supplied as 1.0 point clouds when it is atually 1.2 (according to the directory name). It says this in a few diff places, including the ReadMe contents section.
- RG12_09-2014-088.(sol|sup) should be RG12_09-2014_088.(sol|sup) (i.e. _ instead of -)
- Screenshot dir contains a tif - should only contain jpgs
- "The data has been split during processing. Not that estimates of pointcloud sizes and noise data were made before splitting". I'd explain in more detail, e.g. lines were split into 3 equal parts to reduced file sizes. Also, do you mean "Note" rather than "Not". Also also, what does this second sentence mean? The per flight line statistics are for each part so were done AFTER splitting.
comment:37 Changed 10 years ago by stgo
Recreated classified files with lower noise levels again.
Also completely remade readme, reflects comments above. Lidar ready for DC.
comment:38 Changed 10 years ago by knpa
Beginning LiDAR delivery check again
comment:39 Changed 10 years ago by knpa
LiDAR delivery check complete
All looks fine apart from the following:
- A couple of the FW lines seem strangely small. 05 and 25 are both ~1.5G when most other lines are ~7G. These lines are not smaller for the 1.2 or ascii versions
- There are individual screenshots for each line - we only supply the mosaic
- "The data has been split into 3 parts" should be "Each flightline has been split into three parts"
- unix2dos the ascii files
- Remove old LiDAR delivery if it was not sent out and not needed otherwise
comment:40 Changed 10 years ago by stgo
I've corrected the readme and included a comment about the fullwaveform. I had already investigated this and concluded it was incomplete coverage of the data.
The per line screenshots are now included in the delivery. Currently unix2dos'ing the ascii
comment:41 Changed 10 years ago by stgo
Finish unix2dos, lidar ready for recheck.
comment:42 Changed 10 years ago by stgo
Lidar ready for delivery.
comment:43 Changed 10 years ago by dap
Recreating hyperspectral delivery.
comment:44 Changed 10 years ago by dap
Hyperspectral Processing
Found geotiff files to have wobbles, having looked at screenshots. Will process data again.
comment:45 Changed 10 years ago by dap
Hyperspectral Processing
One of the timestamps in the raw .nav file for flight line -2 had an erroneous SPTSMP timestamp sentence (non-numeric). Fixed by changing $SPTSMP,207,4094+,1*38 to $SPTSMP,207,4094,1*38.
comment:46 Changed 10 years ago by dap
Hyperspectral Processing
SWIR bands have now been stripped from the relevant bil files in the delivery directory and unsplit versions moved to the processing directories. Split files in the delivery directory have also been renamed to adhere to naming conventions so note that the bil files in the delivery directory do not contain SWIR bands.
comment:47 Changed 10 years ago by dap
Hyperspectral Processing
Delivery at RG12_09-088-hyperspectral-20140919/ is now ready for delivery check.
Pay particular attention to the fact that the lines were split in to smaller, manageable chunks then had SWIR bands removed, which involved having to edit the bil header files and xml files (not part of the standard processing procedures).
comment:48 Changed 10 years ago by tec
Hyperspectral Delivery Check
Starting DC
comment:49 Changed 10 years ago by tec
3 Hyperspectral deliveries, checking the latest one.
- Converted sensor_FOV_vectors/fenix_viewvector_list.txt to windows line terminators
- Redo readme, commands in config refered to nextmap dems, I fixed config but you need to regenerate readme.
- Only 348 bands but SWIR cut so its fine
- Heavy noise from bands 1-8 all flights
- Data quality "While the data have been corrected" would "While the data has been corrected" sound better?
comment:50 Changed 10 years ago by dap
Hyperspectral
- I edited the .tex file so the correct commands were in the file but had forgotten to change it in the config file.
- Added a data quality remark in the read me about noise in bands 1-8.
- Left "While the data have been corrected" as original.
comment:51 Changed 10 years ago by dap
- Component changed from Processing: general to Archiving
- Description modified (diff)
Fenix, LiDAR and RCD data sent to Dr Gerard, 08/10/2014
comment:52 Changed 10 years ago by tec
Archiving
Started
comment:53 Changed 10 years ago by tec
- Description modified (diff)
- Resolution set to fixed
- Status changed from new to closed
Archiving
Removed 2 old delivery in processing directory as the one delivered is newer.
Removed all las files in processing directory as they are far too large and it was a mess, they can be recreated anyway.
Removed flightlines directory as it can be recreated and it was 2.7TB so definitely not rsyncing that.
Finished
Navigation processing started, basestation ppp results: