Opened 8 years ago

Closed 6 years ago

Last modified 5 years ago

#596 closed flight processing (fixed)

RANNIS15/28, flight day 241b/2015, Hekla

Reported by: asm Owned by:
Priority: immediate Milestone: 2015 data processing completion
Component: Processing: general Keywords:
Cc: Other processors:

Description (last modified by dac)

Data location: ~arsf/arsf_data/2015/flight_data/iceland/RANNIS15_28-2015_241b_Hekla

Data arrived from ARSF via SATA disk 8 on 28/09/2015.

Priority: Unknown

PI: Gro Pedersen

CAFAM code: R028

Sensors:

  • Fenix (requested)
  • Owl (requested - currently unfunded)
  • Leica LIDAR (requested)
  • RCD (requested)

Change History (83)

comment:1 Changed 8 years ago by dac

  • Milestone set to 2015 data processing completion

comment:2 Changed 8 years ago by asm

Flight Details
Logsheet has been created

-The logsheet shows that flightline 009a has Fenix VNIR with Binning to 4:2
-Also says that flightlines 008, 009 were aborted (but there is Fenix data from those, please check when processing.)
-There is only Full-waveform data for flightlines: 005, 006, 008, 009a.
-Landing taking off and engine times are approximate.

Last edited 8 years ago by asm (previous) (diff)

comment:3 Changed 8 years ago by dac

  • Description modified (diff)

comment:4 Changed 8 years ago by asm

General Processing
Sent to PI ARSF data arrival notification for RANNIS15/28.

comment:5 Changed 8 years ago by asm

Navigation Processing, Basestation verification.

Needed to download base station information for this project. Used REYK

Lat: 64 08 19.62955
Lon: -21 57 19.75549
Elipsoidal Height: 93.238 (metres)
Antenna Height: 0.057 (metres)

comment:6 Changed 8 years ago by asm

Navigation Processing

Finished. Everything looks fine.

comment:7 Changed 8 years ago by asm

Navigation Processing

Also created sol file for camera.

comment:8 Changed 8 years ago by lah

RCD Processing
Missing ImageEvents file. Copied in extra from 241a and now processing fine.

comment:9 Changed 8 years ago by lah

RCD Processing
removed overexposed images:
29072835100098G3.jpg, 29072835100099G3.jpg, 29072835100107G3.jpg, 29072835100108G3.jpg,
29072835100228G3.jpg, 29072835100229G3.jpg, 29072835100291G3.jpg, 29072835100292G3.jpg,
29072835100301G3.jpg, 29072835100357G3.jpg, 29072835100358G3.jpg

and "not found" images:
29072835100204G3.tif, 29072835100205G3.tif, 29072835100206G3.tif

comment:10 Changed 8 years ago by asm

Lidar Processing

Started.

comment:11 Changed 8 years ago by asm

Lidar Processing

There is a lack of usable features and almost no overlap between flightlines except for two of them and a flightline that goes across the others. Used the overlapping flightlines to find the best pitch and roll value. Offset error seems to be consistent across flightlines and the error is within limits but the lack of features prevent us from finding finer tuning.

Will create the delivery with this values:
Roll: - 0.0014
Pitch: -0.0022458

Last edited 8 years ago by asm (previous) (diff)

comment:12 Changed 8 years ago by asm

Lidar Processing

Classification started.

comment:13 Changed 8 years ago by asm

Lidar Processing

Classification finished. Will not process FW data since it was not requested and only 3 flightlines were recorded since the sensor failed and was turned off.

comment:14 Changed 8 years ago by asm

Lidar Processing

Meassuring elevation difference when possible.

comment:15 Changed 8 years ago by asm

Lidar Processing

Delivery created. Ready for Delivery check.

comment:16 Changed 8 years ago by gej

Lidar DC

Roll and pitch for overlapping lines 5-6 is wrong, there is a house in the middle of the flightlines that you can check on. Other lines can't really tell so I would say get 5-6 right and then check the other ones are correct with the new roll and pitch.

Everything else looks okay to me.

comment:17 Changed 8 years ago by lah

RCD Ready for DC

Had to create a new delivery as kml file was missing RCD entries. Event file created ok this time, though had to change permissions to view it.

comment:18 Changed 8 years ago by asm

Lidar Processing

Found better offset values with the feature gej pointed out. All lines seem consistent:
Roll: - 0.000642
Pitch: -0.002741

Will start classification.

comment:19 Changed 8 years ago by asm

Lidar Processing

Finished classification and created delivery. Measured elevation difference and created readme file. Ready for Delivery Check.

comment:20 follow-up: Changed 8 years ago by gej

Lidar delivery check

Readme states that elevation difference could be calculated for 01-02, 03-04, 05-06 and 11-12. There are only 3 elevation differences on the table.

comment:21 Changed 8 years ago by dac

RCD Delivery

Starting delivery check

comment:22 Changed 8 years ago by dac

RCD Delivery

All looks fine. Would be good to add a note in the readme about the cloud shadow in some images. Other than that ready to go.

comment:23 Changed 8 years ago by lah

RCD

Have mentioned cloud shadowing in readme, on bottom of page 5.

comment:24 Changed 8 years ago by dac

PI prefers delivery by hard drive

Last edited 8 years ago by dac (previous) (diff)

comment:25 Changed 8 years ago by gej

Fenix processing

Have started processing Fenix up to finding SCT offsets.

comment:26 Changed 8 years ago by gej

Fenix Processing

Found SCTs

Flightline FENIX
1 0.97
2 1.00
3 1.00
4 1.00
5 1.00
6 0.00
7 1.00
8 1.00
9 1.00
10 1.00
11 1.00
12 0.00
13 1.00
14 1.00
15 1.00

Waiting for new nav before mapping.

Last edited 8 years ago by gej (previous) (diff)

comment:27 Changed 8 years ago by dac

Marking as blocked - still waiting on GPS data from IMO.

comment:28 Changed 8 years ago by dac

RCD Processing

Converted raw files to tif.

comment:29 Changed 8 years ago by asm

Navigation processing

Navigation reprocessed with GPS data provided by Icelandic Met-office to compare data quality. Basestations are the same that 241a 2015

Best solution found using MJSK. Copying files and will rename old basestation and ipas_honeywell to old_nav_basestation (Please clean after processing, before archiving).

comment:30 in reply to: ↑ 20 Changed 8 years ago by asm

Replying to gej:

Lidar delivery check

Readme states that elevation difference could be calculated for 01-02, 03-04, 05-06 and 11-12. There are only 3 elevation differences on the table.

That was caused because there was a aborted line I did not notice. Will solve that for the new delivery.

Last edited 8 years ago by asm (previous) (diff)

comment:31 Changed 8 years ago by gej

Fenix Processing

SCTs for new navigation same as old.
Mapping files.

comment:32 Changed 8 years ago by asm

Lidar Processing

Found new offset using new navigation. All lines seem consistent:
Roll: -0.0006405
Pitch: -0.00253

Will start classification and create delivery.

comment:33 Changed 8 years ago by gej

Fenix Processing

Have checked SCTs with new navigation and all are the same.
Mapped files generated, creating delivery now.

comment:34 Changed 8 years ago by asm

Lidar Processing

Have created delivery and measured elevation difference. Ready for delivery check.

comment:35 Changed 8 years ago by dac

LiDAR Delivery Check

Started delivery check.

comment:36 Changed 8 years ago by dac

LiDAR Delivery Check

Putting on hold until processed with full range of scan angles.

comment:37 Changed 8 years ago by asm

LiDAR

Redone with full range of scan angles. Will classify and create delivery.

comment:38 Changed 8 years ago by asm

LiDAR

Re-created delivery, measured elevation difference and creating readme file. Awaitting delivery check

comment:39 Changed 8 years ago by stgo

LiDAR Delivery check

Something has happened to the ASCII Point Filter functions on the readme, the command is on a separate page to the text. This should be fixed and checked for other projects.

The same has happened for the copyright notice, shifting over on to the last page, we should try and make sure this stays together.

I disagree with the data quality remark "Flightlines on this project have a
lack of useful features which prevented from finding a more precise value for pitch and roll offsets." - this shouldn't be included.

There are still pitch and roll errors in the overlap regions between between the following lines:

01-02
02-05
03,04,07,12 (especially visible here)

comment:40 Changed 8 years ago by gej

Fenix delivery created, ready for delivery check.

comment:41 Changed 8 years ago by gej

RCD processing
Have re-tagged using new navigation, creating delivery now.

comment:42 Changed 8 years ago by gej

RCD processing

Delivery and read-me created, ready for delivery check

comment:43 Changed 8 years ago by asm

RCD delivery check

DC started.
-Need to remove old delivery
-Eventfile did not have read permissions for group.(-rw-------. !) Could not open it (less) with other user apart form airborne.

comment:44 Changed 8 years ago by asm

RCD delivery check

Anything else looks fine. Ready to deliver.

comment:45 Changed 8 years ago by dac

Fenix Delivery Check

Starting delivery check.

comment:46 Changed 8 years ago by dac

Fenix Delivery Check

All looks fine. However, according to the handwritten logsheet some of the lines (8a, 9a, 12a and 10a) were reflown due to problems during the first acquisition. Only the second (correct) versions should be delivered and the other files removed.

Suggest just remove repeat lines and add note to readme.

comment:47 Changed 8 years ago by asm

Lidar Processing

Did not realise that with wider angle range, there were usable features to find better pitch and roll values. Found offsets and also checked their value with stgo. Consistent trough all lines.
Roll: -0.000805
Pitch: -0.00251

Will classify and create delivery.

comment:48 Changed 8 years ago by gej

Fenix processing

Have removed flightlines 8a,9a,12a and 10a from delivery. Recreated mosaic jpeg and added a note to readme that 8a,9a,10a and 12a have been removed due to being reflown.

Ready for DC

comment:49 Changed 8 years ago by dac

Changed readme text to make it clear lines were repeated due to a problem with the sensor:

"Due to problems with the Fenix sensor lines 008a 009a 012a and 010a were repeated for this flight, only the second acquisition has been included in the delivery"

Have started zipping files, marking as ready to deliver.

comment:50 Changed 8 years ago by asm

Lidar Processing

Pitch and roll errors are consistent trough all lines except flighlitne 01 and 02. Will solve that before creating delivery.

comment:51 Changed 8 years ago by asm

Lidar Processing

Flightline 01:
Roll: -0.000805
Pitch: -0.0027

Flightline 02:
Roll: -0.000805
Pitch: -0.0020

Remaining flightlines:
Roll: -0.000805
Pitch: -0.00251

Will classify and create delivery.

comment:52 Changed 8 years ago by asm

Lidar Processing

Delivery and readme created. Ready for delivery check

comment:53 Changed 8 years ago by stgo

Lidar Delivery check
Starting check

comment:54 Changed 8 years ago by stgo

Lidar delivery check

Pagination errors still existed in the readme, we need to either move sections on to their own pages or reduce the smallskips between sections to fix this.

I can't run dem_compare at the moment, will look into this and hopefully have a figure this afternoon. Otherwise this delivery is ready to go.

comment:55 Changed 8 years ago by stgo

Lidar delivery check

demcompare results:
Mean: -0.418515014075755
Median: -0.111104
Absolute mean: 2.12190412497265
Std deviation: 3.78019477066741

These are very positive. Once the readme is fixed the lidar is ready to go!

comment:56 Changed 8 years ago by stgo

Lidar delivery check

Erm, actually no. The overlap measurements table is definitely out of date, you should update it to contain measurements between all lines that have an overlap region (which is all of them).

comment:57 Changed 8 years ago by asm

Lidar

It is not out of date, they were not consecutive. As dac wrote on 242 ticket but was intended for this project: Should get overlap between more lines. The readme does state only sequential flightlines were used but could go by line ID rather than filename and get more - from the screenshot it is clear there are a lot of overlapping lines.

Will measure some more elevation difference.

comment:58 Changed 8 years ago by dac

Also checked (and wrote on 242 ticket).

  • Should get overlap between more lines. The readme does state only sequential flightlines were used but could go by line ID rather than filename and get more - from the screenshot it is clear there are a lot of overlapping lines.
  • Pitch and roll look good.

comment:59 Changed 8 years ago by asm

Lidar

Have measured more elevation differences even for not consecutive flightlines and replaced old readme file. Will mark as ready to deliver and will deliver via hard drive as soon as possible.

comment:60 Changed 8 years ago by asm

Delivery

RCD, Lidar and Fenix data dispatched via hard drive on 23/02/2016.

Last edited 8 years ago by asm (previous) (diff)

comment:61 Changed 6 years ago by dac

Archiving

Data uploaded to CEDA at request of PI.

comment:62 Changed 6 years ago by dac

  • Resolution set to fixed
  • Status changed from new to closed

Archiving

Data available from CEDA. Marking as closed.

comment:63 Changed 5 years ago by wja

Last edited 5 years ago by wja (previous) (diff)

comment:64 Changed 5 years ago by wja

Last edited 5 years ago by wja (previous) (diff)

comment:65 Changed 5 years ago by wja

Please ignore previous two comments. They belong (and have been moved to) ticket 595.

comment:66 Changed 5 years ago by wja

Owl Processing

  • 14 flightlines
  • 13 calibration files
  • Calibration files in separate directories to raw data
  • Calibration files with the closest timestamps to raw data have been symlinked into raw data directories
  • All lines have been stitched
  • All lines only contain one timestamp - start time at the bottom of the header

Level1b data has been processed. Continuing to process mapped lines.

Last edited 5 years ago by wja (previous) (diff)

comment:67 Changed 5 years ago by wja

Owl Processing
'GPS Start Time' in stitched line headers more likely resemble stop times. APL config has been generated using start and stop times from Fenix lines.
Currently processing without using real time navigation.

Last edited 5 years ago by wja (previous) (diff)

comment:68 Changed 5 years ago by wja

Owl Processing
Level1b lines have been split in sections of 10000 lines. An APL config file has been created with split lines
Start and stop times have been calculated from the original headers.
Some lines were failing due to the DEM being smaller than the nav files. A new ASTER DEM has been created with a larger extent to resolve this

  • Flightlines which are sucessfully processing: 1 (part 3), 2, 3 ,4 ,5 ,6, 7, 8 (parts 0, 1 & 2)
  • Flightlines which have not processed: 1 (parts 1 & 2), 5 (parts 0 & 1), 8 (part 3), 9, 10, 11, 12, 13, 14

Unprocessed flightlines all are producing an error because times are outside of the nav file range. Investigating...

comment:69 Changed 5 years ago by wja

Owl Processing
APL seems to have taken the start times from the processed level1b lines. These start times are 1 hour and 1 second ahead of the raw data end times (which explains why the later lines failed to process and the SCT values of the other lines were huge). This is now resolved.
Finding SCT values of all lines.

Last edited 5 years ago by wja (previous) (diff)

comment:70 Changed 5 years ago by wja

Owl Processing
Flightline 1 contains no features in any bands and therefore will be omitted from further processing. Processing has been attempted with calibration files T1637 and T1645.
Owl SCT values:

FlightlineSCT
1N/A
22.96
33.36
42.90
52.34
63.08
73.82
83.00
92.74
103.06
112.54
122.70
132.78
142.96

Creating delivery.

comment:71 Changed 5 years ago by wja

Owl Processing
Delivery created, currently editing ReadMe.

comment:72 Changed 5 years ago by wja

Owl Delivery
ReadMe edited. Awaiting delivery check.

comment:73 Changed 5 years ago by asm

Owl Delivery Check

Started.

comment:74 Changed 5 years ago by asm

Owl Delivery Check

-Just looking through the screenshots, it looks like there are problems with SCTs:

-owl 06
-owl 07
-owl 10
-owl 14 part 0?

-No xml files (wja already on it)
-Logsheet:

-Line Ids do not match with the previous delivered logsheet. This complicates a lot to match Fenix already delivered and new owl data. I suggest to deliver the old style logsheet and just add the owl data (File, FPS, IT) on it for consistency

-Readme:

-Logsheet vs flightline name: This is not a table and does not match with the fenix logsheet. Again, I think the logsheet names should follow old style logsheet format
-gdal_translate command: The bands and areas are all 0. Needs an working command as example.
-aplcorr command: Goes out of the page
-aplmask command: Wrong filenames
-After aplmap: also goes to another line

comment:75 Changed 5 years ago by wja

Owl Delivery
Reprocessing with the following SCT values:

FlightlineSCT
1N/A
22.96
32.32
42.90
52.34
62.15
72.84
83.00
92.74
102.07
112.54
122.70
132.78
142.96

comment:76 Changed 5 years ago by wja

Owl Delivery
Delivery has been updated to remain consistant with hyperspectral delivery.

  • Flightlines have not been split.
  • Screenshots have been updated.
  • All nessessary level1, mapped, navigation and line information is present and named appropriately.
  • New logsheet and ReadMes have been generated with flightline IDs that match hyperspectral delivery.
  • APL commands have been updated.
  • project_information *xml had to be generated manually, stop times were need to be input into level1 headers.

Awaiting Delivery Check

comment:77 Changed 5 years ago by asm

Owl Delivery Check

Changes look good:
-xml files created correctly
-No SCT errors. Lines are not split but they are not too heavy so there is no need
-Readme problems have been fixed
-APL commands work fine.
-All others checks were made with positive results

Will zip the files and mark project as ready to be delivered once done.

comment:78 Changed 5 years ago by asm

Owl Delivery Check

All files have been zipped correctly, marking project as ready to be delivered.

comment:79 Changed 5 years ago by wja

Owl Delivered
Data placed on FTP slot 11.
Notification e-mail sent.
Delivery finalised.

comment:80 Changed 5 years ago by wja

Owl Archiving
Started archiving.

comment:81 Changed 5 years ago by wja

Owl Archiving
Added to PostGIS.
Starting rsync to CEDA (15:18 31/05/2019).

comment:82 Changed 5 years ago by wja

Owl Archiving
rsync to CEDA complete (3/6/19).
E-mail sent to confirm it has been received.

comment:83 Changed 5 years ago by wja

Owl Archiving
CEDA has confirmed recepit of the reprocessed Owl data.
All data is now archived.

Note: See TracTickets for help on using tickets.