Opened 2 years ago

Last modified 19 months ago

#595 new flight processing

RANNIS15/28, flight day 241a/2015, Hekla.

Reported by: asm Owned by:
Priority: immediate Milestone: 2015 data processing completion
Component: Processing: general Keywords:
Cc: Other processors:

Description (last modified by dac)

Data location: ~arsf/arsf_data/2015/flight_data/iceland/RANNIS15_28-2015_241a_Hekla

Data arrived from ARSF via SATA disk 8 on 28/09/2015.

Priority: Unknown

PI: Gro Pedersen

CAFAM code: R028

Sensors:

  • Fenix (requested)
  • Owl (requested - currently unfunded)
  • Leica LIDAR (requested)
  • RCD (requested)

Change History (76)

comment:1 Changed 2 years ago by dac

  • Milestone set to 2015 data processing completion

comment:2 Changed 2 years ago by asm

  • Description modified (diff)

comment:3 Changed 2 years ago by dac

  • Description modified (diff)

comment:4 Changed 2 years ago by asm

General Processing
-Logsheet generated.
-Landing and taking off times are approximate.

comment:5 Changed 2 years ago by asm

General Processing
Sent to PI ARSF data arrival notification for RANNIS15/28.

comment:6 Changed 2 years ago by asm

Navigation Processing,

Started. Needed to download base station information for this project. Used REYK

comment:7 Changed 2 years ago by asm

Basestation verification
Needed to download base station information for this project. Used REYK

Lat: 64 08 19.63063
Lon: -21 57 19.7628
Elipsoidal Height: 93.223 (metres)
Antenna Height: 0.057 (metres)

comment:8 Changed 2 years ago by asm

Navigation Processing

Could not find a suitable solution on Ipas TC. Will try using Ips Pro.

comment:9 Changed 2 years ago by gej

RCD processing

Have produced tif files, waiting for navigation.

comment:10 Changed 2 years ago by asm

Navigation Processing

Navigation was done using Ipas Pro. It is located on a different folder on ipas_honeywell since it will be tested during processing chain (specially Lidar). If no further problems appear, navigation is finished.

comment:11 Changed 2 years ago by lah

Lidar Processing
Processed with 20150829_090311_ipaspro.sol. No obvious problems yet.

comment:12 Changed 2 years ago by lah

Lidar Processing
Found pitch and roll. Seems stable for all lines, though later lines don't have much overlap. No buildings, so had to use rock formations.

Roll = -0.0007
Pitch = -0.0025

comment:13 Changed 2 years ago by lah

Lidar Processing
All classified. Volcanic ash cloud required substantial manual classification.

comment:14 Changed 2 years ago by lah

Lidar Processing
Delivery made. Ready for checking. Alspp would not process first line, so it is omitted. FW is included even though not requested.

A screenshot from this flight has been uploaded to twitter with PIs approval.

comment:15 Changed 2 years ago by asm

Lidar DC

Started
-Readme is not present at delivery but was created and had a look at it on processing directory. Flightline 028 recorded in logsheet is not on delivery, a note should be added to the readme.
-proj_tidy is complaning about every single file name but I can't see anything wrong, I suspect it is caused by a longer than usual project_code.

comment:16 follow-up: Changed 2 years ago by lah

Lidar

Have moved readme into delivery. Missing line 28 is mentioned at the end of page 2 with the line info. Should it be somewhere else? Not sure why alspp wouldn't process it though.

Pretty sure I previously ran proj tidy before the delivery and didn't get any naming problems. Looks like RANNIS is to long for the regex (2-5 chars allowed), so fails everything in the delivery directory.

comment:17 in reply to: ↑ 16 Changed 2 years ago by asm

Replying to lah:

Lidar

Have moved readme into delivery. Missing line 28 is mentioned at the end of page 2 with the line info. Should it be somewhere else? Not sure why alspp wouldn't process it though.

Everything looks good now. My mistake, I did not see those additional lines before, I usually write it on the data quality section, that is why I've missed it but I think is better this way.

comment:18 Changed 2 years ago by asm

Lidar DC

Navigation is currently being re-doing after some issues found on Ipas Pro. Will check again if line 28 could be processed with the new navigation.

comment:19 Changed 2 years ago by stgo

Navigation processing

I've reprocessed in Ipas TC with ppp, separation of 2 cm so will be using this over the Ipas pro solution. Gej is copying off the windows machine now.

comment:20 Changed 2 years ago by gej

Navigation Processing

Files have been copied to posatt directory, in ipas_Honeywell.

RCD processing
Event file recreated with new sol files. No Problems anymore (was previously giving nav values of 0 for the first 50 images).

Tagging now.

comment:21 Changed 2 years ago by stgo

Navigation processing

That suggests the Lidar file that wouldn't process should be fine now

comment:22 Changed 2 years ago by asm

Fenix Processing

Started. Generated config file and found that Owl header files are missing all starting and ending GPS times and points. We will have to discuss how to process the Owl data.

Last edited 2 years ago by asm (previous) (diff)

comment:23 Changed 2 years ago by lah

Lidar Processing
Line 028 has been processed with new sol and aligns well with adjacent line. Is now classified, so just needs adding to delivery.

Should we include both sol files in delivery?

comment:24 Changed 2 years ago by stgo

Lidar Processing

The PPP result is considerably better quality than the differential so I don't think we should be sending the differential. It also doesn't cover the full survey site (hence the problems processing the first lidar line), have the other lines been reprocessed with the new nav file?

comment:25 Changed 2 years ago by lah

Lidar Processing

Needs reprocessing as original nav quality not good enough.

comment:26 Changed 2 years ago by asm

Fenix Processing

Found Sct values.

Flightline FENIX
1 0.88
2 1.10
3 0.91
4 1.08
5 0.94
6 1.03
7 0.93
8 1.04
9 1.02
10 1.03
11 0.94
12 1.01
13 1.04
14 0.96
15 0.97
16 0.98
17 1.04
18 1.01

Will split lines since they are fairly long.

comment:27 Changed 2 years ago by lah

Lidar Processing
Have reprocessed data with new sol file, copied in classifications and made new delivery

Ready for delivery check.

comment:28 Changed 2 years ago by gej

RCD ready for delivery check

comment:29 Changed 2 years ago by asm

Fenix Processing

Fenix delivery created included readme file. Flightlines have been split and are still around 40Gb a note should be added to the notification mail in case they are having problems with them. Ready for Delivery Check.

comment:30 Changed 2 years ago by dac

Fenix Delivery Check

Starting delivery check

comment:31 Changed 2 years ago by asm

Fenix Processing

Notice that the script for creating screenshots is failing to name split flightlines and were not named according to its part. Have solved the issue manually.

comment:32 Changed 2 years ago by dac

Fenix Delivery Check

  • Missing ASCII view vectors
  • On page 7 of readme undefined LaTeX reference (see ?? section)
  • Config file / APL commands contain the wrong name for the DEM
  • Screenshots still have SCT values in name.
  • proj_tidy reports 100s of errors (likely due to name)
  • Checked spectral with Py6S - look fine.
  • Logsheet has Al HOwlad as the pilot - suggest fix capitalisation
  • Part numbering starts at 0 not 1 - could cause confusion. This is the same as 174 but I hadn't realised the script did this - starting at 1 might be better.
  • Need to remove fodis directory.


Other than this all look OK.

comment:33 Changed 2 years ago by asm

Fenix Processing

All issues have been checked. A couple of comments:
-Readme undefined LaTeX reference - It was referenced before being defined, it took me sometime to realise that nothing was wrong and it had to be compiled at least twice.
-Missing ASCII view vectors - The file "fenix_fov_fullccd_vectors.bil" is there; please, let me know if is missing something else.

comment:34 Changed 2 years ago by dac

Fenix

Also need ASCII version of view vectors. Created using get_ascii_fov_for_level1.py the script wasn't finding the files from the directory so had to specify an individual line.

check_apl_cmd works now and generated files look fine.

Ready to deliver

comment:35 Changed 2 years ago by dac

Fenix Delivery

Started zipping mapped files.

comment:36 Changed 2 years ago by dac

RCD Delivery Check

Starting delivery check

comment:37 Changed 2 years ago by dac

RCD Delivery Check

No Read me. Everything else looks OK. Will continue when readme is present.

comment:38 Changed 2 years ago by dac

LiDAR Delivery Check

Starting delivery check.

comment:39 Changed 2 years ago by gej

RCD Delivery

Have generated readme and moved to delivery directory.

Ready to carry on DC.

comment:40 Changed 2 years ago by dac

LiDAR Delivery Check

Couple of minor points found and fixed:

  • In overlap table file number not line name used (have fixed)
  • The pilots name is wrong in the logsheet - it should be Al Howland not Al Howlad (I think). Have corrected and in hyperspectral delivery as well.
  • demcompare.py wouldn't run. Checked patched DEM instead, LiDAR and ASTER line up well.

Ready to deliver.

comment:41 Changed 2 years ago by dac

RCD Delivery

Readme look fine. Ready to deliver.

comment:42 Changed 2 years ago by dac

Delivery

Send RCD, Fenix and LiDAR to Gro Pederson (University of Iceland) on a hard drive (21/12/2015). Also placed on FTP server (arsf4) for early access.

comment:43 Changed 2 years ago by lah

OWL

Used this flight for the missing dark frame tests and example imagery for the Owl report. Only used 5 lines, so project still requires proper unpacking. scts found are in table below, based on GPS times in Fenix headers (owl and fenix line numbers are different due to the way data has been stored - asm is working on automated unpacking to deal with this):

Flightline (Fenix) Flightline (Owl) OWL
2 5 501.83
3 8 -3.08
4 10 -2.98
5 12 -3.10
8 18 9.35

Owl line 5 is much shorter than fenix line 2 so probably failed to start at the same time and explains the large sct.

comment:44 Changed 23 months ago by asm

Navigation processing

Navigation reprocessed with GPS data provided by Icelandic Met-office to compare data quality.
Basestation verification for FEDG:
Latitude 64 01 30.93075
Lon -19 41 22.05467
Ell Height 503.758
Ant Height 1.00 m, to L1-PC (TRM411249.00, MeasDist 0.929)
Which corresponds on decimal degrees to the values sent by Met-office: -19.68946006 64.02525821 503.73149445

Basestation verification for HESA:
Latitude 64 02 48.62706
Lon -19 33 38.08790
Ell Height 529.112
Ant Height 1.090 m, to L1-PC (TRM411249.00, MeasDist 1.019)
Also match data recieved: -19.56057992 64.04684073 529.08620580 HESA deltaz=0.244

Basestation verification for MJSK:
Latitude 63 55 58.59960
Lon -19 40 20.52995
Ell Height 770.561
Ant Height 1.050 m, to L1-PC (TRM411249.00, MeasDist 0.979)

Best solution found using FEDG and HESA. Copied files and renamed old basestation and ipas_honeywell to old_nav_basestation (Please clean after processing, before archiving).

comment:45 Changed 22 months ago by asm

RCD Processing

Tagging pictures with the new navigation. There are two different ImageEvents1 and PhotoId1, the correct one is:
150829-013924

comment:46 Changed 22 months ago by asm

RCD Processing

Have created a new delivery folder with the new event file. Tagging all files again. New event file has 8 pictures out of bonds that will be removed on this delivery.

Last edited 22 months ago by asm (previous) (diff)

comment:47 Changed 22 months ago by asm

RCD Processing

Finished tagging, creating delivery. Noticed that 8 pictures outside bonds where also outside bonds in the first event and therefore there is no problem.

comment:48 Changed 22 months ago by asm

RCD Processing

Delivery created and readme file added. Ready for Delivery Check.

comment:49 Changed 22 months ago by gej

RCD delivery check

No Problems with delivery,

Ready to deliver

comment:50 Changed 22 months ago by asm

Delivery

RCD data dispatched via hard drive on 23/02/2016.

comment:51 Changed 22 months ago by lah

Hyperspectral reprocessing

Started reprocessing with new navigation. There were a lot of files still in the hyperspectral processing directory, dated after the previous delivery so I've temporarily moved them to hyperspectral_old as I don't know what they're for. Please delete or move elsewhere if still in use.

comment:52 Changed 22 months ago by lah

Lidar reprocessing

Started reprocessing lidar with new nav. Also corrected problems from boresight file which were used in previous delivery:

  • Filter range bank B deselected
  • max angle set to 45 instead of 7

comment:53 Changed 22 months ago by dac

Hyperspectral

The hyperspectral directory lah moved to 'hyperspectral_old' contained gej's tests comparing the old and new navigation data. No longer needed so have removed.

comment:54 Changed 22 months ago by lah

Lidar reprocessing

New files created. Reclassifying as more points than previously.

comment:55 Changed 22 months ago by lah

Hyperspectral reprocessing

New delivery created with new nav and old scts. Ready for DC.

comment:56 Changed 22 months ago by lah

Lidar reprocessing

New delivery created with new nav and new manual classification. Quickly checked the first few elevation differences and they still seem reasonable.

Ready for DC

comment:57 Changed 22 months ago by stgo

Lidar delivery check

Beginning check

Last edited 22 months ago by stgo (previous) (diff)

comment:58 Changed 22 months ago by asm

Hyperspectral Delivery Check

Everything looks fine.
Spectra was checked using Py6s and it could not always find vegetation pixel but spectra is OK.
Will zip files and should be ready for delivery tomorrow morning.

comment:59 follow-up: Changed 22 months ago by stgo

Lidar delivery check

Everything looks fine, waiting on demcompare to confirm.

Proj tidy really hates the hyperspectral delivery, we should probably update it to accept the "part_xxx" files

comment:60 in reply to: ↑ 59 Changed 22 months ago by asm

Replying to stgo:

Lidar delivery check

Everything looks fine, waiting on demcompare to confirm.

Proj tidy really hates the hyperspectral delivery, we should probably update it to accept the "part_xxx" files

And larger project names.

comment:61 Changed 22 months ago by stgo

Lidar delivery check

demcompare output:
Min: -759.149876953125
Max: 78.0656688232422
Sum: -3121742.19369611
Mean: -7.44021286604393
Median: -7.57575
Absolute mean: 9.56823590658078
Std deviation: 11.5387215208943
Total cells: 6486792
Non-null cells: 419577

Mean height difference looks fine for the masked area, ready for delivery.

comment:62 Changed 22 months ago by asm

Hyperspectral Delivery Check

Mapped files have been zipped. Project is ready to go.

comment:63 Changed 22 months ago by asm

Delivery

Fenix and LiDAR reprocessed with new navigation dispatched via hard drive on 03/03/2016. No more reprocessing pending.

Last edited 22 months ago by asm (previous) (diff)

comment:64 Changed 22 months ago by lah

OWL Reprocessing

Started reprocessing 5 flightlines with improved navigation data using existing scts. Manually renamed raw files as previous copies had been deleted and added GPS times into renamed raw header files.

comment:65 Changed 22 months ago by lah

OWL

Created a partial delivery with the 5 flightlines. A couple of things have failed (project info & ascii FOV vectors), but may not be required for sample data.

The duplicated raw files need deleting when this is 'delivered'.

comment:66 Changed 21 months ago by lah

OWL

Forced creation of ascii FOV vectors by bypassing getting sensor from level 1 header ID (lines 276 - 286 of get_ascii_fov_for_level1.py). Should only be xml info missing now.

comment:67 Changed 21 months ago by lah

OWL

Alas, the delivery scripts overwrote the level 1 data with the mask data so it will have to be reprocessed.

comment:68 Changed 21 months ago by lah

OWL

Created new delivery with all problems resolved. Ready for a delivery check.

comment:69 Changed 21 months ago by dac

Owl delivery check

Starting delivery check.

comment:70 Changed 21 months ago by dac

Owl delivery check

Ran through procedure for hyperspectral data and ran into the following issues:

  • Wasn't sure if check_apl_cmd works so didn't run this. Did test APL commands from readme and worked OK though.
  • Proj_tidy has problems with project name so didn't run
  • 1b files are very faint in fastQC and produce error as sensor isn't recognised. Look OK in TuiView though.

Only problem found is the XML metadata files list Carl as the pilot and Tom as the instrument operator (different to logsheet). This is the same as the Fenix data so not worth blocking the project for but will need fixing in templates.

Ready to deliver.

comment:71 Changed 21 months ago by lah

Owl Delivery

  • Changed readme GDAL translate command to use owl bands and a size within flightline.
  • Also added file descriptions to delivery contents table

Only other difference is that metadata files don't have processing chain / software info since apl was run only in meta stage to create the xml files after data were already processed.

comment:72 Changed 21 months ago by lah

OWL Delivery

Uploaded to ftp arsf4 on 7/4/16 and notification sent to Gro, Susan and Graham.

Dispatched hard disk to Gro on 11/4/16.

comment:73 Changed 20 months ago by dac

RCD Processing

Reprocessed RCD images to locate missing files.

comment:74 Changed 20 months ago by dac

RCD Processing

Looks like images were removed due to overexposure and a red tint (not recorded in ticket). Will work out out which files were previously missing and generate new delivery with these.

comment:75 Changed 19 months ago by asm

RCD Processing

The red tint was caused by a problem with calibration. Once solved revealed pictures were not delivered due to overexposure only (still not recorded on ticket).

Delivered on 12/May/2016 using ftp 4

comment:76 Changed 19 months ago by lah

OWL renaming

  • Ran new renaming script on original owl raw files
  • Replaced some of these with the modified raw files used in processing (which have now been manually renamed to the correct number)
  • Identified T1 & T2 used in processing by new name before deleting copy
Processed line New raw file Calibration file used in processing
5 1 1113
8 3 1120
10 4 1133
12 5 1146
18 8 1225

i.e. raw files 1,3,4,5, and 8 have been modified, all others have been automatically timestamped but have no dark frames appended.

Note: See TracTickets for help on using tickets.