#596 closed flight processing (fixed)
RANNIS15/28, flight day 241b/2015, Hekla
Reported by: | asm | Owned by: | |
---|---|---|---|
Priority: | immediate | Milestone: | 2015 data processing completion |
Component: | Processing: general | Keywords: | |
Cc: | Other processors: |
Description (last modified by dac)
Data location: ~arsf/arsf_data/2015/flight_data/iceland/RANNIS15_28-2015_241b_Hekla
Data arrived from ARSF via SATA disk 8 on 28/09/2015.
Priority: Unknown
PI: Gro Pedersen
CAFAM code: R028
Sensors:
- Fenix (requested)
- Owl (requested - currently unfunded)
- Leica LIDAR (requested)
- RCD (requested)
Change History (83)
comment:1 Changed 9 years ago by dac
- Milestone set to 2015 data processing completion
comment:2 Changed 9 years ago by asm
comment:3 Changed 9 years ago by dac
- Description modified (diff)
comment:4 Changed 9 years ago by asm
General Processing
Sent to PI ARSF data arrival notification for RANNIS15/28.
comment:5 Changed 9 years ago by asm
Navigation Processing, Basestation verification.
Needed to download base station information for this project. Used REYK
Lat: 64 08 19.62955
Lon: -21 57 19.75549
Elipsoidal Height: 93.238 (metres)
Antenna Height: 0.057 (metres)
comment:6 Changed 9 years ago by asm
Navigation Processing
Finished. Everything looks fine.
comment:7 Changed 9 years ago by asm
Navigation Processing
Also created sol file for camera.
comment:8 Changed 9 years ago by lah
RCD Processing
Missing ImageEvents file. Copied in extra from 241a and now processing fine.
comment:9 Changed 9 years ago by lah
RCD Processing
removed overexposed images:
29072835100098G3.jpg, 29072835100099G3.jpg, 29072835100107G3.jpg, 29072835100108G3.jpg,
29072835100228G3.jpg, 29072835100229G3.jpg, 29072835100291G3.jpg, 29072835100292G3.jpg,
29072835100301G3.jpg, 29072835100357G3.jpg, 29072835100358G3.jpg
and "not found" images:
29072835100204G3.tif, 29072835100205G3.tif, 29072835100206G3.tif
comment:10 Changed 9 years ago by asm
Lidar Processing
Started.
comment:11 Changed 9 years ago by asm
Lidar Processing
There is a lack of usable features and almost no overlap between flightlines except for two of them and a flightline that goes across the others. Used the overlapping flightlines to find the best pitch and roll value. Offset error seems to be consistent across flightlines and the error is within limits but the luck of features prevent us from finding finer tuning.
Will create the delivery with this values:
Roll: - 0.0014
Pitch: -0.0022458
comment:12 Changed 9 years ago by asm
Lidar Processing
Classification started.
comment:13 Changed 9 years ago by asm
Lidar Processing
Classification finished. Will not process FW data since it was not requested and only 3 flightlines were recorded since the sensor failed and was turned off.
comment:14 Changed 9 years ago by asm
Lidar Processing
Meassuring elevation difference when possible.
comment:15 Changed 9 years ago by asm
Lidar Processing
Delivery created. Ready for Delivery check.
comment:16 Changed 9 years ago by gej
Lidar DC
Roll and pitch for overlapping lines 5-6 is wrong, there is a house in the middle of the flightlines that you can check on. Other lines can't really tell so I would say get 5-6 right and then check the other ones are correct with the new roll and pitch.
Everything else looks okay to me.
comment:17 Changed 9 years ago by lah
RCD Ready for DC
Had to create a new delivery as kml file was missing RCD entries. Event file created ok this time, though had to change permissions to view it.
comment:18 Changed 9 years ago by asm
Lidar Processing
Found better offset values with the feature gej pointed out. All lines seem consistent:
Roll: - 0.000642
Pitch: -0.002741
Will start classification.
comment:19 Changed 9 years ago by asm
Lidar Processing
Finished classification and created delivery. Measured elevation difference and created readme file. Ready for Delivery Check.
comment:20 follow-up: ↓ 30 Changed 9 years ago by gej
Lidar delivery check
Readme states that elevation difference could be calculated for 01-02, 03-04, 05-06 and 11-12. There are only 3 elevation differences on the table.
comment:21 Changed 9 years ago by dac
RCD Delivery
Starting delivery check
comment:22 Changed 9 years ago by dac
RCD Delivery
All looks fine. Would be good to add a note in the readme about the cloud shadow in some images. Other than that ready to go.
comment:23 Changed 9 years ago by lah
RCD
Have mentioned cloud shadowing in readme, on bottom of page 5.
comment:24 Changed 9 years ago by dac
PI prefers delivery by hard drive
comment:25 Changed 9 years ago by gej
Fenix processing
Have started processing Fenix up to finding SCT offsets.
comment:26 Changed 9 years ago by gej
Fenix Processing
Found SCTs
Flightline | FENIX |
1 | 0.97 |
2 | 1.00 |
3 | 1.00 |
4 | 1.00 |
5 | 1.00 |
6 | 0.00 |
7 | 1.00 |
8 | 1.00 |
9 | 1.00 |
10 | 1.00 |
11 | 1.00 |
12 | 0.00 |
13 | 1.00 |
14 | 1.00 |
15 | 1.00 |
Waiting for new nav before mapping.
comment:27 Changed 9 years ago by dac
Marking as blocked - still waiting on GPS data from IMO.
comment:28 Changed 9 years ago by dac
RCD Processing
Converted raw files to tif.
comment:29 Changed 9 years ago by asm
Navigation processing
Navigation reprocessed with GPS data provided by Icelandic Met-office to compare data quality. Basestations are the same that 241a 2015
Best solution found using MJSK. Copying files and will rename old basestation and ipas_honeywell to old_nav_basestation (Please clean after processing, before archiving).
comment:30 in reply to: ↑ 20 Changed 9 years ago by asm
Replying to gej:
Lidar delivery check
Readme states that elevation difference could be calculated for 01-02, 03-04, 05-06 and 11-12. There are only 3 elevation differences on the table.
That was caused because there was a aborted line I did not notice. Will solve that for the new delivery.
comment:31 Changed 9 years ago by gej
Fenix Processing
SCTs for new navigation same as old.
Mapping files.
comment:32 Changed 9 years ago by asm
Lidar Processing
Found new offset using new navigation. All lines seem consistent:
Roll: -0.0006405
Pitch: -0.00253
Will start classification and create delivery.
comment:33 Changed 9 years ago by gej
Fenix Processing
Have checked SCTs with new navigation and all are the same.
Mapped files generated, creating delivery now.
comment:34 Changed 9 years ago by asm
Lidar Processing
Have created delivery and measured elevation difference. Ready for delivery check.
comment:35 Changed 9 years ago by dac
LiDAR Delivery Check
Started delivery check.
comment:36 Changed 9 years ago by dac
LiDAR Delivery Check
Putting on hold until processed with full range of scan angles.
comment:37 Changed 9 years ago by asm
LiDAR
Redone with full range of scan angles. Will classify and create delivery.
comment:38 Changed 9 years ago by asm
LiDAR
Re-created delivery, measured elevation difference and creating readme file. Awaitting delivery check
comment:39 Changed 9 years ago by stgo
LiDAR Delivery check
Something has happened to the ASCII Point Filter functions on the readme, the command is on a separate page to the text. This should be fixed and checked for other projects.
The same has happened for the copyright notice, shifting over on to the last page, we should try and make sure this stays together.
I disagree with the data quality remark "Flightlines on this project have a
lack of useful features which prevented from finding a more precise value for pitch and roll offsets." - this shouldn't be included.
There are still pitch and roll errors in the overlap regions between between the following lines:
01-02
02-05
03,04,07,12 (especially visible here)
comment:40 Changed 9 years ago by gej
Fenix delivery created, ready for delivery check.
comment:41 Changed 9 years ago by gej
RCD processing
Have re-tagged using new navigation, creating delivery now.
comment:42 Changed 9 years ago by gej
RCD processing
Delivery and read-me created, ready for delivery check
comment:43 Changed 9 years ago by asm
RCD delivery check
DC started.
-Need to remove old delivery
-Eventfile did not have read permissions for group.(-rw-------. !) Could not open it (less) with other user apart form airborne.
comment:44 Changed 9 years ago by asm
RCD delivery check
Anything else looks fine. Ready to deliver.
comment:45 Changed 9 years ago by dac
Fenix Delivery Check
Starting delivery check.
comment:46 Changed 9 years ago by dac
Fenix Delivery Check
All looks fine. However, according to the handwritten logsheet some of the lines (8a, 9a, 12a and 10a) were reflown due to problems during the first acquisition. Only the second (correct) versions should be delivered and the other files removed.
Suggest just remove repeat lines and add note to readme.
comment:47 Changed 9 years ago by asm
Lidar Processing
Did not realise that with wider angle range, there were usable features to find better pitch and roll values. Found offsets and also checked their value with stgo. Consistent trough all lines.
Roll: -0.000805
Pitch: -0.00251
Will classify and create delivery.
comment:48 Changed 9 years ago by gej
Fenix processing
Have removed flightlines 8a,9a,12a and 10a from delivery. Recreated mosaic jpeg and added a note to readme that 8a,9a,10a and 12a have been removed due to being reflown.
Ready for DC
comment:49 Changed 9 years ago by dac
Changed readme text to make it clear lines were repeated due to a problem with the sensor:
"Due to problems with the Fenix sensor lines 008a 009a 012a and 010a were repeated for this flight, only the second acquisition has been included in the delivery"
Have started zipping files, marking as ready to deliver.
comment:50 Changed 9 years ago by asm
Lidar Processing
Pitch and roll errors are consistent trough all lines except flighlitne 01 and 02. Will solve that before creating delivery.
comment:51 Changed 9 years ago by asm
Lidar Processing
Flightline 01:
Roll: -0.000805
Pitch: -0.0027
Flightline 02:
Roll: -0.000805
Pitch: -0.0020
Remaining flightlines:
Roll: -0.000805
Pitch: -0.00251
Will classify and create delivery.
comment:52 Changed 9 years ago by asm
Lidar Processing
Delivery and readme created. Ready for delivery check
comment:53 Changed 9 years ago by stgo
Lidar Delivery check
Starting check
comment:54 Changed 9 years ago by stgo
Lidar delivery check
Pagination errors still existed in the readme, we need to either move sections on to their own pages or reduce the smallskips between sections to fix this.
I can't run dem_compare at the moment, will look into this and hopefully have a figure this afternoon. Otherwise this delivery is ready to go.
comment:55 Changed 9 years ago by stgo
Lidar delivery check
demcompare results:
Mean: -0.418515014075755
Median: -0.111104
Absolute mean: 2.12190412497265
Std deviation: 3.78019477066741
These are very positive. Once the readme is fixed the lidar is ready to go!
comment:56 Changed 9 years ago by stgo
Lidar delivery check
Erm, actually no. The overlap measurements table is definitely out of date, you should update it to contain measurements between all lines that have an overlap region (which is all of them).
comment:57 Changed 9 years ago by asm
Lidar
It is not out of date, they were not consecutive. As dac wrote on 242 ticket but was intended for this project: Should get overlap between more lines. The readme does state only sequential flightlines were used but could go by line ID rather than filename and get more - from the screenshot it is clear there are a lot of overlapping lines.
Will measure some more elevation difference.
comment:58 Changed 9 years ago by dac
Also checked (and wrote on 242 ticket).
- Should get overlap between more lines. The readme does state only sequential flightlines were used but could go by line ID rather than filename and get more - from the screenshot it is clear there are a lot of overlapping lines.
- Pitch and roll look good.
comment:59 Changed 9 years ago by asm
Lidar
Have measured more elevation differences even for not consecutive flightlines and replaced old readme file. Will mark as ready to deliver and will deliver via hard drive as soon as possible.
comment:60 Changed 9 years ago by asm
Delivery
RCD, Lidar and Fenix data dispatched via hard drive on 23/02/2016.
comment:61 Changed 6 years ago by dac
Archiving
Data uploaded to CEDA at request of PI.
comment:62 Changed 6 years ago by dac
- Resolution set to fixed
- Status changed from new to closed
Archiving
Data available from CEDA. Marking as closed.
comment:63 Changed 5 years ago by wja
comment:64 Changed 5 years ago by wja
comment:65 Changed 5 years ago by wja
Please ignore previous two comments. They belong (and have been moved to) ticket 595.
comment:66 Changed 5 years ago by wja
Owl Processing
- 14 flightlines
- 13 calibration files
- Calibration files in separate directories to raw data
- Calibration files with the closest timestamps to raw data have been symlinked into raw data directories
- All lines have been stitched
- All lines only contain one timestamp - start time at the bottom of the header
Level1b data has been processed. Continuing to process mapped lines.
comment:67 Changed 5 years ago by wja
Owl Processing
'GPS Start Time' in stitched line headers more likely resemble stop times. APL config has been generated using start and stop times from Fenix lines.
Currently processing without using real time navigation.
comment:68 Changed 5 years ago by wja
Owl Processing
Level1b lines have been split in sections of 10000 lines. An APL config file has been created with split lines
Start and stop times have been calculated from the original headers.
Some lines were failing due to the DEM being smaller than the nav files. A new ASTER DEM has been created with a larger extent to resolve this
- Flightlines which are sucessfully processing: 1 (part 3), 2, 3 ,4 ,5 ,6, 7, 8 (parts 0, 1 & 2)
- Flightlines which have not processed: 1 (parts 1 & 2), 5 (parts 0 & 1), 8 (part 3), 9, 10, 11, 12, 13, 14
Unprocessed flightlines all are producing an error because times are outside of the nav file range. Investigating...
comment:69 Changed 5 years ago by wja
Owl Processing
APL seems to have taken the start times from the processed level1b lines. These start times are 1 hour and 1 second ahead of the raw data end times (which explains why the later lines failed to process and the SCT values of the other lines were huge). This is now resolved.
Finding SCT values of all lines.
comment:70 Changed 5 years ago by wja
Owl Processing
Flightline 1 contains no features in any bands and therefore will be omitted from further processing. Processing has been attempted with calibration files T1637 and T1645.
Owl SCT values:
Flightline | SCT |
1 | N/A |
2 | 2.96 |
3 | 3.36 |
4 | 2.90 |
5 | 2.34 |
6 | 3.08 |
7 | 3.82 |
8 | 3.00 |
9 | 2.74 |
10 | 3.06 |
11 | 2.54 |
12 | 2.70 |
13 | 2.78 |
14 | 2.96 |
Creating delivery.
comment:71 Changed 5 years ago by wja
Owl Processing
Delivery created, currently editing ReadMe.
comment:72 Changed 5 years ago by wja
Owl Delivery
ReadMe edited. Awaiting delivery check.
comment:73 Changed 5 years ago by asm
Owl Delivery Check
Started.
comment:74 Changed 5 years ago by asm
Owl Delivery Check
-Just looking through the screenshots, it looks like there are problems with SCTs:
-owl 06
-owl 07
-owl 10
-owl 14 part 0?
-No xml files (wja already on it)
-Logsheet:
-Line Ids do not match with the previous delivered logsheet. This complicates a lot to match Fenix already delivered and new owl data. I suggest to deliver the old style logsheet and just add the owl data (File, FPS, IT) on it for consistency
-Readme:
-Logsheet vs flightline name: This is not a table and does not match with the fenix logsheet. Again, I think the logsheet names should follow old style logsheet format
-gdal_translate command: The bands and areas are all 0. Needs an working command as example.
-aplcorr command: Goes out of the page
-aplmask command: Wrong filenames
-After aplmap: also goes to another line
comment:75 Changed 5 years ago by wja
Owl Delivery
Reprocessing with the following SCT values:
Flightline | SCT |
1 | N/A |
2 | 2.96 |
3 | 2.32 |
4 | 2.90 |
5 | 2.34 |
6 | 2.15 |
7 | 2.84 |
8 | 3.00 |
9 | 2.74 |
10 | 2.07 |
11 | 2.54 |
12 | 2.70 |
13 | 2.78 |
14 | 2.96 |
comment:76 Changed 5 years ago by wja
Owl Delivery
Delivery has been updated to remain consistant with hyperspectral delivery.
- Flightlines have not been split.
- Screenshots have been updated.
- All nessessary level1, mapped, navigation and line information is present and named appropriately.
- New logsheet and ReadMes have been generated with flightline IDs that match hyperspectral delivery.
- APL commands have been updated.
- project_information *xml had to be generated manually, stop times were need to be input into level1 headers.
Awaiting Delivery Check
comment:77 Changed 5 years ago by asm
Owl Delivery Check
Changes look good:
-xml files created correctly
-No SCT errors. Lines are not split but they are not too heavy so there is no need
-Readme problems have been fixed
-APL commands work fine.
-All others checks were made with positive results
Will zip the files and mark project as ready to be delivered once done.
comment:78 Changed 5 years ago by asm
Owl Delivery Check
All files have been zipped correctly, marking project as ready to be delivered.
comment:79 Changed 5 years ago by wja
Owl Delivered
Data placed on FTP slot 11.
Notification e-mail sent.
Delivery finalised.
comment:80 Changed 5 years ago by wja
Owl Archiving
Started archiving.
comment:81 Changed 5 years ago by wja
Owl Archiving
Added to PostGIS.
Starting rsync to CEDA (15:18 31/05/2019).
comment:82 Changed 5 years ago by wja
Owl Archiving
rsync to CEDA complete (3/6/19).
E-mail sent to confirm it has been received.
comment:83 Changed 5 years ago by wja
Owl Archiving
CEDA has confirmed recepit of the reprocessed Owl data.
All data is now archived.
Flight Details
Logsheet has been created
-The logsheet shows that flightline 009a has Fenix VNIR with Binning to 4:2
-Also says that flightlines 008, 009 were aborted (but there is Fenix data from those, please check when processing.)
-There is only Full-waveform data for flightlines: 005, 006, 008, 009a.
-Landing taking off and engine times are approximate.