Opened 8 years ago

Closed 6 years ago

Last modified 5 years ago

#595 closed flight processing (fixed)

RANNIS15/28, flight day 241a/2015, Hekla.

Reported by: asm Owned by:
Priority: immediate Milestone: 2015 data processing completion
Component: Processing: general Keywords:
Cc: Other processors:

Description (last modified by dac)

Data location: ~arsf/arsf_data/2015/flight_data/iceland/RANNIS15_28-2015_241a_Hekla

Data arrived from ARSF via SATA disk 8 on 28/09/2015.

Priority: Unknown

PI: Gro Pedersen

CAFAM code: R028

Sensors:

  • Fenix (requested)
  • Owl (requested - currently unfunded)
  • Leica LIDAR (requested)
  • RCD (requested)

Change History (90)

comment:1 Changed 8 years ago by dac

  • Milestone set to 2015 data processing completion

comment:2 Changed 8 years ago by asm

  • Description modified (diff)

comment:3 Changed 8 years ago by dac

  • Description modified (diff)

comment:4 Changed 8 years ago by asm

General Processing
-Logsheet generated.
-Landing and taking off times are approximate.

comment:5 Changed 8 years ago by asm

General Processing
Sent to PI ARSF data arrival notification for RANNIS15/28.

comment:6 Changed 8 years ago by asm

Navigation Processing,

Started. Needed to download base station information for this project. Used REYK

comment:7 Changed 8 years ago by asm

Basestation verification
Needed to download base station information for this project. Used REYK

Lat: 64 08 19.63063
Lon: -21 57 19.7628
Elipsoidal Height: 93.223 (metres)
Antenna Height: 0.057 (metres)

comment:8 Changed 8 years ago by asm

Navigation Processing

Could not find a suitable solution on Ipas TC. Will try using Ips Pro.

comment:9 Changed 8 years ago by gej

RCD processing

Have produced tif files, waiting for navigation.

comment:10 Changed 8 years ago by asm

Navigation Processing

Navigation was done using Ipas Pro. It is located on a different folder on ipas_honeywell since it will be tested during processing chain (specially Lidar). If no further problems appear, navigation is finished.

comment:11 Changed 8 years ago by lah

Lidar Processing
Processed with 20150829_090311_ipaspro.sol. No obvious problems yet.

comment:12 Changed 8 years ago by lah

Lidar Processing
Found pitch and roll. Seems stable for all lines, though later lines don't have much overlap. No buildings, so had to use rock formations.

Roll = -0.0007
Pitch = -0.0025

comment:13 Changed 8 years ago by lah

Lidar Processing
All classified. Volcanic ash cloud required substantial manual classification.

comment:14 Changed 8 years ago by lah

Lidar Processing
Delivery made. Ready for checking. Alspp would not process first line, so it is omitted. FW is included even though not requested.

A screenshot from this flight has been uploaded to twitter with PIs approval.

comment:15 Changed 8 years ago by asm

Lidar DC

Started
-Readme is not present at delivery but was created and had a look at it on processing directory. Flightline 028 recorded in logsheet is not on delivery, a note should be added to the readme.
-proj_tidy is complaning about every single file name but I can't see anything wrong, I suspect it is caused by a longer than usual project_code.

comment:16 follow-up: Changed 8 years ago by lah

Lidar

Have moved readme into delivery. Missing line 28 is mentioned at the end of page 2 with the line info. Should it be somewhere else? Not sure why alspp wouldn't process it though.

Pretty sure I previously ran proj tidy before the delivery and didn't get any naming problems. Looks like RANNIS is to long for the regex (2-5 chars allowed), so fails everything in the delivery directory.

comment:17 in reply to: ↑ 16 Changed 8 years ago by asm

Replying to lah:

Lidar

Have moved readme into delivery. Missing line 28 is mentioned at the end of page 2 with the line info. Should it be somewhere else? Not sure why alspp wouldn't process it though.

Everything looks good now. My mistake, I did not see those additional lines before, I usually write it on the data quality section, that is why I've missed it but I think is better this way.

comment:18 Changed 8 years ago by asm

Lidar DC

Navigation is currently being re-doing after some issues found on Ipas Pro. Will check again if line 28 could be processed with the new navigation.

comment:19 Changed 8 years ago by stgo

Navigation processing

I've reprocessed in Ipas TC with ppp, separation of 2 cm so will be using this over the Ipas pro solution. Gej is copying off the windows machine now.

comment:20 Changed 8 years ago by gej

Navigation Processing

Files have been copied to posatt directory, in ipas_Honeywell.

RCD processing
Event file recreated with new sol files. No Problems anymore (was previously giving nav values of 0 for the first 50 images).

Tagging now.

comment:21 Changed 8 years ago by stgo

Navigation processing

That suggests the Lidar file that wouldn't process should be fine now

comment:22 Changed 8 years ago by asm

Fenix Processing

Started. Generated config file and found that Owl header files are missing all starting and ending GPS times and points. We will have to discuss how to process the Owl data.

Last edited 8 years ago by asm (previous) (diff)

comment:23 Changed 8 years ago by lah

Lidar Processing
Line 028 has been processed with new sol and aligns well with adjacent line. Is now classified, so just needs adding to delivery.

Should we include both sol files in delivery?

comment:24 Changed 8 years ago by stgo

Lidar Processing

The PPP result is considerably better quality than the differential so I don't think we should be sending the differential. It also doesn't cover the full survey site (hence the problems processing the first lidar line), have the other lines been reprocessed with the new nav file?

comment:25 Changed 8 years ago by lah

Lidar Processing

Needs reprocessing as original nav quality not good enough.

comment:26 Changed 8 years ago by asm

Fenix Processing

Found Sct values.

Flightline FENIX
1 0.88
2 1.10
3 0.91
4 1.08
5 0.94
6 1.03
7 0.93
8 1.04
9 1.02
10 1.03
11 0.94
12 1.01
13 1.04
14 0.96
15 0.97
16 0.98
17 1.04
18 1.01

Will split lines since they are fairly long.

comment:27 Changed 8 years ago by lah

Lidar Processing
Have reprocessed data with new sol file, copied in classifications and made new delivery

Ready for delivery check.

comment:28 Changed 8 years ago by gej

RCD ready for delivery check

comment:29 Changed 8 years ago by asm

Fenix Processing

Fenix delivery created included readme file. Flightlines have been split and are still around 40Gb a note should be added to the notification mail in case they are having problems with them. Ready for Delivery Check.

comment:30 Changed 8 years ago by dac

Fenix Delivery Check

Starting delivery check

comment:31 Changed 8 years ago by asm

Fenix Processing

Notice that the script for creating screenshots is failing to name split flightlines and were not named according to its part. Have solved the issue manually.

comment:32 Changed 8 years ago by dac

Fenix Delivery Check

  • Missing ASCII view vectors
  • On page 7 of readme undefined LaTeX reference (see ?? section)
  • Config file / APL commands contain the wrong name for the DEM
  • Screenshots still have SCT values in name.
  • proj_tidy reports 100s of errors (likely due to name)
  • Checked spectral with Py6S - look fine.
  • Logsheet has Al HOwlad as the pilot - suggest fix capitalisation
  • Part numbering starts at 0 not 1 - could cause confusion. This is the same as 174 but I hadn't realised the script did this - starting at 1 might be better.
  • Need to remove fodis directory.


Other than this all look OK.

comment:33 Changed 8 years ago by asm

Fenix Processing

All issues have been checked. A couple of comments:
-Readme undefined LaTeX reference - It was referenced before being defined, it took me sometime to realise that nothing was wrong and it had to be compiled at least twice.
-Missing ASCII view vectors - The file "fenix_fov_fullccd_vectors.bil" is there; please, let me know if is missing something else.

comment:34 Changed 8 years ago by dac

Fenix

Also need ASCII version of view vectors. Created using get_ascii_fov_for_level1.py the script wasn't finding the files from the directory so had to specify an individual line.

check_apl_cmd works now and generated files look fine.

Ready to deliver

comment:35 Changed 8 years ago by dac

Fenix Delivery

Started zipping mapped files.

comment:36 Changed 8 years ago by dac

RCD Delivery Check

Starting delivery check

comment:37 Changed 8 years ago by dac

RCD Delivery Check

No Read me. Everything else looks OK. Will continue when readme is present.

comment:38 Changed 8 years ago by dac

LiDAR Delivery Check

Starting delivery check.

comment:39 Changed 8 years ago by gej

RCD Delivery

Have generated readme and moved to delivery directory.

Ready to carry on DC.

comment:40 Changed 8 years ago by dac

LiDAR Delivery Check

Couple of minor points found and fixed:

  • In overlap table file number not line name used (have fixed)
  • The pilots name is wrong in the logsheet - it should be Al Howland not Al Howlad (I think). Have corrected and in hyperspectral delivery as well.
  • demcompare.py wouldn't run. Checked patched DEM instead, LiDAR and ASTER line up well.

Ready to deliver.

comment:41 Changed 8 years ago by dac

RCD Delivery

Readme look fine. Ready to deliver.

comment:42 Changed 8 years ago by dac

Delivery

Send RCD, Fenix and LiDAR to Gro Pederson (University of Iceland) on a hard drive (21/12/2015). Also placed on FTP server (arsf4) for early access.

comment:43 Changed 8 years ago by lah

OWL

Used this flight for the missing dark frame tests and example imagery for the Owl report. Only used 5 lines, so project still requires proper unpacking. scts found are in table below, based on GPS times in Fenix headers (owl and fenix line numbers are different due to the way data has been stored - asm is working on automated unpacking to deal with this):

Flightline (Fenix) Flightline (Owl) OWL
2 5 501.83
3 8 -3.08
4 10 -2.98
5 12 -3.10
8 18 9.35

Owl line 5 is much shorter than fenix line 2 so probably failed to start at the same time and explains the large sct.

comment:44 Changed 8 years ago by asm

Navigation processing

Navigation reprocessed with GPS data provided by Icelandic Met-office to compare data quality.
Basestation verification for FEDG:
Latitude 64 01 30.93075
Lon -19 41 22.05467
Ell Height 503.758
Ant Height 1.00 m, to L1-PC (TRM411249.00, MeasDist 0.929)
Which corresponds on decimal degrees to the values sent by Met-office: -19.68946006 64.02525821 503.73149445

Basestation verification for HESA:
Latitude 64 02 48.62706
Lon -19 33 38.08790
Ell Height 529.112
Ant Height 1.090 m, to L1-PC (TRM411249.00, MeasDist 1.019)
Also match data recieved: -19.56057992 64.04684073 529.08620580 HESA deltaz=0.244

Basestation verification for MJSK:
Latitude 63 55 58.59960
Lon -19 40 20.52995
Ell Height 770.561
Ant Height 1.050 m, to L1-PC (TRM411249.00, MeasDist 0.979)

Best solution found using FEDG and HESA. Copied files and renamed old basestation and ipas_honeywell to old_nav_basestation (Please clean after processing, before archiving).

comment:45 Changed 8 years ago by asm

RCD Processing

Tagging pictures with the new navigation. There are two different ImageEvents1 and PhotoId1, the correct one is:
150829-013924

comment:46 Changed 8 years ago by asm

RCD Processing

Have created a new delivery folder with the new event file. Tagging all files again. New event file has 8 pictures out of bonds that will be removed on this delivery.

Last edited 8 years ago by asm (previous) (diff)

comment:47 Changed 8 years ago by asm

RCD Processing

Finished tagging, creating delivery. Noticed that 8 pictures outside bonds where also outside bonds in the first event and therefore there is no problem.

comment:48 Changed 8 years ago by asm

RCD Processing

Delivery created and readme file added. Ready for Delivery Check.

comment:49 Changed 8 years ago by gej

RCD delivery check

No Problems with delivery,

Ready to deliver

comment:50 Changed 8 years ago by asm

Delivery

RCD data dispatched via hard drive on 23/02/2016.

comment:51 Changed 8 years ago by lah

Hyperspectral reprocessing

Started reprocessing with new navigation. There were a lot of files still in the hyperspectral processing directory, dated after the previous delivery so I've temporarily moved them to hyperspectral_old as I don't know what they're for. Please delete or move elsewhere if still in use.

comment:52 Changed 8 years ago by lah

Lidar reprocessing

Started reprocessing lidar with new nav. Also corrected problems from boresight file which were used in previous delivery:

  • Filter range bank B deselected
  • max angle set to 45 instead of 7

comment:53 Changed 8 years ago by dac

Hyperspectral

The hyperspectral directory lah moved to 'hyperspectral_old' contained gej's tests comparing the old and new navigation data. No longer needed so have removed.

comment:54 Changed 8 years ago by lah

Lidar reprocessing

New files created. Reclassifying as more points than previously.

comment:55 Changed 8 years ago by lah

Hyperspectral reprocessing

New delivery created with new nav and old scts. Ready for DC.

comment:56 Changed 8 years ago by lah

Lidar reprocessing

New delivery created with new nav and new manual classification. Quickly checked the first few elevation differences and they still seem reasonable.

Ready for DC

comment:57 Changed 8 years ago by stgo

Lidar delivery check

Beginning check

Last edited 8 years ago by stgo (previous) (diff)

comment:58 Changed 8 years ago by asm

Hyperspectral Delivery Check

Everything looks fine.
Spectra was checked using Py6s and it could not always find vegetation pixel but spectra is OK.
Will zip files and should be ready for delivery tomorrow morning.

comment:59 follow-up: Changed 8 years ago by stgo

Lidar delivery check

Everything looks fine, waiting on demcompare to confirm.

Proj tidy really hates the hyperspectral delivery, we should probably update it to accept the "part_xxx" files

comment:60 in reply to: ↑ 59 Changed 8 years ago by asm

Replying to stgo:

Lidar delivery check

Everything looks fine, waiting on demcompare to confirm.

Proj tidy really hates the hyperspectral delivery, we should probably update it to accept the "part_xxx" files

And larger project names.

comment:61 Changed 8 years ago by stgo

Lidar delivery check

demcompare output:
Min: -759.149876953125
Max: 78.0656688232422
Sum: -3121742.19369611
Mean: -7.44021286604393
Median: -7.57575
Absolute mean: 9.56823590658078
Std deviation: 11.5387215208943
Total cells: 6486792
Non-null cells: 419577

Mean height difference looks fine for the masked area, ready for delivery.

comment:62 Changed 8 years ago by asm

Hyperspectral Delivery Check

Mapped files have been zipped. Project is ready to go.

comment:63 Changed 8 years ago by asm

Delivery

Fenix and LiDAR reprocessed with new navigation dispatched via hard drive on 03/03/2016. No more reprocessing pending.

Last edited 8 years ago by asm (previous) (diff)

comment:64 Changed 8 years ago by lah

OWL Reprocessing

Started reprocessing 5 flightlines with improved navigation data using existing scts. Manually renamed raw files as previous copies had been deleted and added GPS times into renamed raw header files.

comment:65 Changed 8 years ago by lah

OWL

Created a partial delivery with the 5 flightlines. A couple of things have failed (project info & ascii FOV vectors), but may not be required for sample data.

The duplicated raw files need deleting when this is 'delivered'.

comment:66 Changed 8 years ago by lah

OWL

Forced creation of ascii FOV vectors by bypassing getting sensor from level 1 header ID (lines 276 - 286 of get_ascii_fov_for_level1.py). Should only be xml info missing now.

comment:67 Changed 8 years ago by lah

OWL

Alas, the delivery scripts overwrote the level 1 data with the mask data so it will have to be reprocessed.

comment:68 Changed 8 years ago by lah

OWL

Created new delivery with all problems resolved. Ready for a delivery check.

comment:69 Changed 8 years ago by dac

Owl delivery check

Starting delivery check.

comment:70 Changed 8 years ago by dac

Owl delivery check

Ran through procedure for hyperspectral data and ran into the following issues:

  • Wasn't sure if check_apl_cmd works so didn't run this. Did test APL commands from readme and worked OK though.
  • Proj_tidy has problems with project name so didn't run
  • 1b files are very faint in fastQC and produce error as sensor isn't recognised. Look OK in TuiView though.

Only problem found is the XML metadata files list Carl as the pilot and Tom as the instrument operator (different to logsheet). This is the same as the Fenix data so not worth blocking the project for but will need fixing in templates.

Ready to deliver.

comment:71 Changed 8 years ago by lah

Owl Delivery

  • Changed readme GDAL translate command to use owl bands and a size within flightline.
  • Also added file descriptions to delivery contents table

Only other difference is that metadata files don't have processing chain / software info since apl was run only in meta stage to create the xml files after data were already processed.

comment:72 Changed 8 years ago by lah

OWL Delivery

Uploaded to ftp arsf4 on 7/4/16 and notification sent to Gro, Susan and Graham.

Dispatched hard disk to Gro on 11/4/16.

comment:73 Changed 8 years ago by dac

RCD Processing

Reprocessed RCD images to locate missing files.

comment:74 Changed 8 years ago by dac

RCD Processing

Looks like images were removed due to overexposure and a red tint (not recorded in ticket). Will work out out which files were previously missing and generate new delivery with these.

comment:75 Changed 8 years ago by asm

RCD Processing

The red tint was caused by a problem with calibration. Once solved revealed pictures were not delivered due to overexposure only (still not recorded on ticket).

Delivered on 12/May/2016 using ftp 4

comment:76 Changed 8 years ago by lah

OWL renaming

  • Ran new renaming script on original owl raw files
  • Replaced some of these with the modified raw files used in processing (which have now been manually renamed to the correct number)
  • Identified T1 & T2 used in processing by new name before deleting copy
Processed line New raw file Calibration file used in processing
5 1 1113
8 3 1120
10 4 1133
12 5 1146
18 8 1225

i.e. raw files 1,3,4,5, and 8 have been modified, all others have been automatically timestamped but have no dark frames appended.

comment:77 Changed 6 years ago by dac

Archiving

Data uploaded to CEDA at request of PI.

comment:78 Changed 6 years ago by dac

  • Resolution set to fixed
  • Status changed from new to closed

Archiving

Data archived at CEDA. Closing ticket.

comment:79 Changed 5 years ago by wja

Owl Processing
Starting Owl processing.

comment:80 Changed 5 years ago by wja

Owl Processing

  • Calibration files are in separate directories to line data. These have been symlinked to the nearest flightline capture directory (by timestamps) and a config file has been created to match each flightline to a calibration file.
  • Line 5 has no timestamp, it has been matched to the calibration file for flightline 4.
  • Darkframes missing on lines 2, 6, 7, 9, 10, 11, 12 and no separate darkframe files are present. These flightlines have been stitched to their closest calibration file. Flightlines which do not need stitching have been symlinked into the stitched directory for processing.

Level1b data processed.
The start and end times from hardcopy logsheet have been added to the stitched headers where they are missing (lines 2, 6, 7, 9, 10, 11 & 12).
Error being thrown regarding the timestamp of flightline 2 when trying to generate processing config file. Investigating.

Last edited 5 years ago by wja (previous) (diff)

comment:81 Changed 5 years ago by wja

Owl Processing
Config generation has not been successful with lines that have been stitched (2, 6, 7, 9, 10, 11, 12).
Have noticed that the start times of stiched headers were one hour ahead, which I have corrected. However, the following needs to added at the bottom of stitched headers manually.

GPS Stop Time = UTC TIME: hh:mm:ss.ssss

comment:82 Changed 5 years ago by wja

Owl Processing
Stop times calculated using owl_hdr_fix.py.
Flightline 2 has been omitted from further processing as it is considerably short (only 669 lines).
SCT Values found:

FlightlineSCT
1503.80
2N/A
3-1.12
4-0.96
5-1.10
62.08
72.44
811.25
93.20
102.60
112.44
122.96

XMLs are being produced by processing full lines and changing the 'sensorid' in the level1b headers to match that found in the hyperspectral parameters CSV.
Need to process split lines for delivery.

comment:83 Changed 5 years ago by wja

Owl Processing
Reprocessed to level3 with the closest calibration times for each lines using the flightline times from owl_hdr_fix.py and accounting for the calibration file times being 1 hour ahead:

LineCal File
1T1108
2T1120
3T1120
4T1133
5T1146
6T1200
7T1212
8T1225
9T1239
10T1252
11T1305
12T1318

All lines apart from line 1 have been split into 3 parts to be similar to the delivered hyperspectral data.
Ready for Delivery Check

comment:84 Changed 5 years ago by asm

Owl Delivery Check

Started.

comment:85 Changed 5 years ago by asm

Owl Delivery Check

-Renamed screenshots from "mapped_utm27n.bil.jpg" to ".jpg"
-Screenshots for overlapping parts look a bit odd due to the stretch but that shouldn't be aproblem
-Realised that Fenix line 02 overlaps with owl line 01 (a bit inconvenient but this is OK). Then owl line 02 is not delivered and line 03 overlaps with fenix 03 (and so on). This format is consistent with other deliveries.
-APL commands work fine.
-All others checks were made with positive results

Mapped files have been zipped and will mark project as ready to be delivered

comment:86 Changed 5 years ago by wja

Owl Delivered
Data placed on FTP slot 11.
Notification e-mail sent.
Delivery finalised.

comment:87 Changed 5 years ago by wja

Archiving Owl
Started archiving.

comment:88 Changed 5 years ago by wja

Owl Archiving
Added to PostGIS.
Starting rsync to CEDA (15:18 31/05/2019).

comment:89 Changed 5 years ago by wja

Owl Archiving
rsync to CEDA complete (3/6/19).
E-mail sent to confirm it has been received.

comment:90 Changed 5 years ago by wja

Owl Archiving
CEDA has confirmed recepit of the reprocessed Owl data.
All data is now archived.

Note: See TracTickets for help on using tickets.