RANNIS15/28, flight day 241a/2015, Hekla.

[asm]

Data location: ~arsf/arsf_data/2015/flight_data/iceland/RANNIS15_28-2015_241a_Hekla

Data arrived from ARSF via SATA disk 8 on 28/09/2015.

Priority: Unknown

PI: Gro Pedersen

CAFAM code: R028

Sensors:

• Fenix (requested)
• Owl (requested - currently unfunded)
• Leica LIDAR (requested)
• RCD (requested)

[asm]

General Processing
-Logsheet generated.
-Landing and taking off times are approximate.

[asm]

General Processing
Sent to PI ARSF data arrival notification for RANNIS15/28.

[asm]

Navigation Processing,

Started. Needed to download base station information for this project. Used REYK

[asm]

Basestation verification
Needed to download base station information for this project. Used REYK

Lat: 64 08 19.63063
Lon: -21 57 19.7628
Elipsoidal Height: 93.223 (metres)
Antenna Height: 0.057 (metres)

[asm]

Navigation Processing

Could not find a suitable solution on Ipas TC. Will try using Ips Pro.

[gej]

RCD processing

Have produced tif files, waiting for navigation.

[asm]

Navigation Processing

Navigation was done using Ipas Pro. It is located on a different folder on ipas_honeywell since it will be tested during processing chain (specially Lidar). If no further problems appear, navigation is finished.

[lah]

Lidar Processing
Processed with 20150829_090311_ipaspro.sol. No obvious problems yet.

[lah]

Lidar Processing
Found pitch and roll. Seems stable for all lines, though later lines don't have much overlap. No buildings, so had to use rock formations.

Roll = -0.0007
Pitch = -0.0025

[lah]

Lidar Processing
All classified. Volcanic ash cloud required substantial manual classification.

[lah]

Lidar Processing
Delivery made. Ready for checking. Alspp would not process first line, so it is omitted. FW is included even though not requested.

A screenshot from this flight has been uploaded to twitter with PIs approval.

[asm]

Lidar DC

Started
-Readme is not present at delivery but was created and had a look at it on processing directory. Flightline 028 recorded in logsheet is not on delivery, a note should be added to the readme.
-proj_tidy is complaning about every single file name but I can't see anything wrong, I suspect it is caused by a longer than usual project_code.

[lah]

Lidar

Have moved readme into delivery. Missing line 28 is mentioned at the end of page 2 with the line info. Should it be somewhere else? Not sure why alspp wouldn't process it though.

Pretty sure I previously ran proj tidy before the delivery and didn't get any naming problems. Looks like RANNIS is to long for the regex (2-5 chars allowed), so fails everything in the delivery directory.

[asm]

Replying to lah:

Lidar

Have moved readme into delivery. Missing line 28 is mentioned at the end of page 2 with the line info. Should it be somewhere else? Not sure why alspp wouldn't process it though.

Everything looks good now. My mistake, I did not see those additional lines before, I usually write it on the data quality section, that is why I've missed it but I think is better this way.

[asm]

Lidar DC

Navigation is currently being re-doing after some issues found on Ipas Pro. Will check again if line 28 could be processed with the new navigation.

[stgo]

Navigation processing

I've reprocessed in Ipas TC with ppp, separation of 2 cm so will be using this over the Ipas pro solution. Gej is copying off the windows machine now.

[gej]

Navigation Processing

Files have been copied to posatt directory, in ipas_Honeywell.

RCD processing
Event file recreated with new sol files. No Problems anymore (was previously giving nav values of 0 for the first 50 images).

Tagging now.

[stgo]

Navigation processing

That suggests the Lidar file that wouldn't process should be fine now

[asm]

Fenix Processing

Started. Generated config file and found that Owl header files are missing all starting and ending GPS times and points. We will have to discuss how to process the Owl data.

[lah]

Lidar Processing
Line 028 has been processed with new sol and aligns well with adjacent line. Is now classified, so just needs adding to delivery.

Should we include both sol files in delivery?

[stgo]

Lidar Processing

The PPP result is considerably better quality than the differential so I don't think we should be sending the differential. It also doesn't cover the full survey site (hence the problems processing the first lidar line), have the other lines been reprocessed with the new nav file?

[lah]

Lidar Processing

Needs reprocessing as original nav quality not good enough.

[asm]

Fenix Processing

Found Sct values.

Flightline	FENIX
1	0.88
2	1.10
3	0.91
4	1.08
5	0.94
6	1.03
7	0.93
8	1.04
9	1.02
10	1.03
11	0.94
12	1.01
13	1.04
14	0.96
15	0.97
16	0.98
17	1.04
18	1.01

Will split lines since they are fairly long.

[lah]

Lidar Processing
Have reprocessed data with new sol file, copied in classifications and made new delivery

Ready for delivery check.

[gej]

RCD ready for delivery check

[asm]

Fenix Processing

Fenix delivery created included readme file. Flightlines have been split and are still around 40Gb a note should be added to the notification mail in case they are having problems with them. Ready for Delivery Check.

[dac]

Fenix Delivery Check

Starting delivery check

[asm]

Fenix Processing

Notice that the script for creating screenshots is failing to name split flightlines and were not named according to its part. Have solved the issue manually.

[dac]

Fenix Delivery Check

• Missing ASCII view vectors
• On page 7 of readme undefined LaTeX reference (see ?? section)
• Config file / APL commands contain the wrong name for the DEM
• Screenshots still have SCT values in name.
• proj_tidy reports 100s of errors (likely due to name)
• Checked spectral with Py6S - look fine.
• Logsheet has Al HOwlad as the pilot - suggest fix capitalisation
• Part numbering starts at 0 not 1 - could cause confusion. This is the same as 174 but I hadn't realised the script did this - starting at 1 might be better.
• Need to remove fodis directory.

Other than this all look OK.

[asm]

Fenix Processing

All issues have been checked. A couple of comments:
-Readme undefined LaTeX reference - It was referenced before being defined, it took me sometime to realise that nothing was wrong and it had to be compiled at least twice.
-Missing ASCII view vectors - The file "fenix_fov_fullccd_vectors.bil" is there; please, let me know if is missing something else.

[dac]

Fenix

Also need ASCII version of view vectors. Created using get_ascii_fov_for_level1.py the script wasn't finding the files from the directory so had to specify an individual line.

check_apl_cmd works now and generated files look fine.

Ready to deliver

[dac]

Fenix Delivery

Started zipping mapped files.

[dac]

RCD Delivery Check

Starting delivery check

[dac]

RCD Delivery Check

No Read me. Everything else looks OK. Will continue when readme is present.

[dac]

LiDAR Delivery Check

Starting delivery check.

[gej]

RCD Delivery

Have generated readme and moved to delivery directory.

Ready to carry on DC.

[dac]

LiDAR Delivery Check

Couple of minor points found and fixed:

• In overlap table file number not line name used (have fixed)
• The pilots name is wrong in the logsheet - it should be Al Howland not Al Howlad (I think). Have corrected and in hyperspectral delivery as well.
• demcompare.py wouldn't run. Checked patched DEM instead, LiDAR and ASTER line up well.

Ready to deliver.

[dac]

RCD Delivery

Readme look fine. Ready to deliver.

[dac]

Delivery

Send RCD, Fenix and LiDAR to Gro Pederson (University of Iceland) on a hard drive (21/12/2015). Also placed on FTP server (arsf4) for early access.

[lah]

OWL

Used this flight for the missing dark frame tests and example imagery for the Owl report. Only used 5 lines, so project still requires proper unpacking. scts found are in table below, based on GPS times in Fenix headers (owl and fenix line numbers are different due to the way data has been stored - asm is working on automated unpacking to deal with this):

Flightline (Fenix)	Flightline (Owl)	OWL
2	5	501.83
3	8	-3.08
4	10	-2.98
5	12	-3.10
8	18	9.35

Owl line 5 is much shorter than fenix line 2 so probably failed to start at the same time and explains the large sct.

[asm]

Navigation processing

Navigation reprocessed with GPS data provided by Icelandic Met-office to compare data quality.
Basestation verification for FEDG:
Latitude 64 01 30.93075
Lon -19 41 22.05467
Ell Height 503.758
Ant Height 1.00 m, to L1-PC (TRM411249.00, MeasDist 0.929)
Which corresponds on decimal degrees to the values sent by Met-office: -19.68946006 64.02525821 503.73149445

Basestation verification for HESA:
Latitude 64 02 48.62706
Lon -19 33 38.08790
Ell Height 529.112
Ant Height 1.090 m, to L1-PC (TRM411249.00, MeasDist 1.019)
Also match data recieved: -19.56057992 64.04684073 529.08620580 HESA deltaz=0.244

Basestation verification for MJSK:
Latitude 63 55 58.59960
Lon -19 40 20.52995
Ell Height 770.561
Ant Height 1.050 m, to L1-PC (TRM411249.00, MeasDist 0.979)

Best solution found using FEDG and HESA. Copied files and renamed old basestation and ipas_honeywell to old_nav_basestation (Please clean after processing, before archiving).

[asm]

RCD Processing

Tagging pictures with the new navigation. There are two different ImageEvents1 and PhotoId1, the correct one is:
150829-013924

[asm]

RCD Processing

Have created a new delivery folder with the new event file. Tagging all files again. New event file has 8 pictures out of bonds that will be removed on this delivery.

[asm]

RCD Processing

Finished tagging, creating delivery. Noticed that 8 pictures outside bonds where also outside bonds in the first event and therefore there is no problem.

[asm]

RCD Processing

Delivery created and readme file added. Ready for Delivery Check.

[gej]

RCD delivery check

No Problems with delivery,

Ready to deliver

[asm]

Delivery

RCD data dispatched via hard drive on 23/02/2016.

[lah]

Hyperspectral reprocessing

Started reprocessing with new navigation. There were a lot of files still in the hyperspectral processing directory, dated after the previous delivery so I've temporarily moved them to hyperspectral_old as I don't know what they're for. Please delete or move elsewhere if still in use.

[lah]

Lidar reprocessing

Started reprocessing lidar with new nav. Also corrected problems from boresight file which were used in previous delivery:

• Filter range bank B deselected
• max angle set to 45 instead of 7

[dac]

Hyperspectral

The hyperspectral directory lah moved to 'hyperspectral_old' contained gej's tests comparing the old and new navigation data. No longer needed so have removed.

[lah]

Lidar reprocessing

New files created. Reclassifying as more points than previously.

[lah]

Hyperspectral reprocessing

New delivery created with new nav and old scts. Ready for DC.

[lah]

Lidar reprocessing

New delivery created with new nav and new manual classification. Quickly checked the first few elevation differences and they still seem reasonable.

Ready for DC

[stgo]

Lidar delivery check

Beginning check

[asm]

Hyperspectral Delivery Check

Everything looks fine.
Spectra was checked using Py6s and it could not always find vegetation pixel but spectra is OK.
Will zip files and should be ready for delivery tomorrow morning.

[stgo]

Lidar delivery check

Everything looks fine, waiting on demcompare to confirm.

Proj tidy really hates the hyperspectral delivery, we should probably update it to accept the "part_xxx" files

[asm]

Replying to stgo:

Lidar delivery check

Everything looks fine, waiting on demcompare to confirm.

Proj tidy really hates the hyperspectral delivery, we should probably update it to accept the "part_xxx" files

And larger project names.

[stgo]

Lidar delivery check

demcompare output:
Min: -759.149876953125
Max: 78.0656688232422
Sum: -3121742.19369611
Mean: -7.44021286604393
Median: -7.57575
Absolute mean: 9.56823590658078
Std deviation: 11.5387215208943
Total cells: 6486792
Non-null cells: 419577

Mean height difference looks fine for the masked area, ready for delivery.

[asm]

Hyperspectral Delivery Check

Mapped files have been zipped. Project is ready to go.

[asm]

Delivery

Fenix and LiDAR reprocessed with new navigation dispatched via hard drive on 23/02/2016.

[lah]

OWL Reprocessing

Started reprocessing 5 flightlines with improved navigation data using existing scts. Manually renamed raw files as previous copies had been deleted and added GPS times into renamed raw header files.

[lah]

OWL

Created a partial delivery with the 5 flightlines. A couple of things have failed (project info & ascii FOV vectors), but may not be required for sample data.

The duplicated raw files need deleting when this is 'delivered'.

[lah]

OWL

Forced creation of ascii FOV vectors by bypassing getting sensor from level 1 header ID (lines 276 - 286 of get_ascii_fov_for_level1.py). Should only be xml info missing now.

[lah]

OWL

Alas, the delivery scripts overwrote the level 1 data with the mask data so it will have to be reprocessed.

[lah]

OWL

Created new delivery with all problems resolved. Ready for a delivery check.

[dac]

Owl delivery check

Starting delivery check.

[dac]

Owl delivery check

Ran through procedure for hyperspectral data and ran into the following issues:

• Wasn't sure if check_apl_cmd works so didn't run this. Did test APL commands from readme and worked OK though.
• Proj_tidy has problems with project name so didn't run
• 1b files are very faint in fastQC and produce error as sensor isn't recognised. Look OK in TuiView though.

Only problem found is the XML metadata files list Carl as the pilot and Tom as the instrument operator (different to logsheet). This is the same as the Fenix data so not worth blocking the project for but will need fixing in templates.

Ready to deliver.

[lah]

Owl Delivery

• Changed readme GDAL translate command to use owl bands and a size within flightline.
• Also added file descriptions to delivery contents table

Only other difference is that metadata files don't have processing chain / software info since apl was run only in meta stage to create the xml files after data were already processed.

[lah]

OWL Delivery

Uploaded to ftp arsf4 on 7/4/16 and notification sent to Gro, Susan and Graham.

Dispatched hard disk to Gro on 11/4/16.

[dac]

RCD Processing

Reprocessed RCD images to locate missing files.

[dac]

RCD Processing

Looks like images were removed due to overexposure and a red tint (not recorded in ticket). Will work out out which files were previously missing and generate new delivery with these.

[asm]

RCD Processing

The red tint was caused by a problem with calibration. Once solved revealed pictures were not delivered due to overexposure only (still not recorded on ticket).

Delivered on 12/May/2016 using ftp 4

[lah]

OWL renaming

• Ran new renaming script on original owl raw files
• Replaced some of these with the modified raw files used in processing (which have now been manually renamed to the correct number)
• Identified T1 & T2 used in processing by new name before deleting copy

Processed line	New raw file	Calibration file used in processing
5	1	1113
8	3	1120
10	4	1133
12	5	1146
18	8	1225

i.e. raw files 1,3,4,5, and 8 have been modified, all others have been automatically timestamped but have no dark frames appended.

[dac]

Archiving

Data uploaded to CEDA at request of PI.

[dac]

Archiving

Data archived at CEDA. Closing ticket.

[wja]

Owl Processing
Starting Owl processing.

[wja]

Owl Processing

• Calibration files are in separate directories to line data. These have been symlinked to the nearest flightline capture directory (by timestamps) and a config file has been created to match each flightline to a calibration file.
  
• Line 5 has no timestamp, it has been matched to the calibration file for flightline 4.
  
• Darkframes missing on lines 2, 6, 7, 9, 10, 11, 12 and no separate darkframe files are present. These flightlines have been stitched to their closest calibration file. Flightlines which do not need stitching have been symlinked into the stitched directory for processing.
  

Level1b data processed.
The start and end times from hardcopy logsheet have been added to the stitched headers where they are missing (lines 2, 6, 7, 9, 10, 11 & 12).
Error being thrown regarding the timestamp of flightline 2 when trying to generate processing config file. Investigating.

[wja]

Owl Processing
Config generation has not been successful with lines that have been stitched (2, 6, 7, 9, 10, 11, 12).
Have noticed that the start times of stiched headers were one hour ahead, which I have corrected. However, the following needs to added at the bottom of stitched headers manually.

GPS Stop Time = UTC TIME: hh:mm:ss.ssss

[wja]

Owl Processing
Stop times calculated using owl_hdr_fix.py.
Flightline 2 has been omitted from further processing as it is considerably short (only 669 lines).
SCT Values found:

Flightline	SCT
1	503.80
2	N/A
3	-1.12
4	-0.96
5	-1.10
6	2.08
7	2.44
8	11.25
9	3.20
10	2.60
11	2.44
12	2.96

XMLs are being produced by processing full lines and changing the 'sensorid' in the level1b headers to match that found in the hyperspectral parameters CSV.
Need to process split lines for delivery.

[wja]

Owl Processing
Reprocessed to level3 with the closest calibration times for each lines using the flightline times from owl_hdr_fix.py and accounting for the calibration file times being 1 hour ahead:

Line	Cal File
1	T1108
2	T1120
3	T1120
4	T1133
5	T1146
6	T1200
7	T1212
8	T1225
9	T1239
10	T1252
11	T1305
12	T1318

All lines apart from line 1 have been split into 3 parts to be similar to the delivered hyperspectral data.
Ready for Delivery Check

[asm]

Owl Delivery Check

Started.

[asm]

Owl Delivery Check

-Renamed screenshots from "mapped_utm27n.bil.jpg" to ".jpg"
-Screenshots for overlapping parts look a bit odd due to the stretch but that shouldn't be aproblem
-Realised that Fenix line 02 overlaps with owl line 01 (a bit inconvenient but this is OK). Then owl line 02 is not delivered and line 03 overlaps with fenix 03 (and so on). This format is consistent with other deliveries.
-APL commands work fine.
-All others checks were made with positive results

Mapped files have been zipped and will mark project as ready to be delivered

[wja]

Owl Delivered
Data placed on FTP slot 11.
Notification e-mail sent.
Delivery finalised.

[wja]

Archiving Owl
Started archiving.

[wja]

Owl Archiving
Added to PostGIS.
Starting rsync to CEDA (15:18 31/05/2019).

[wja]

Owl Archiving
rsync to CEDA complete (3/6/19).
E-mail sent to confirm it has been received.

[wja]

Owl Archiving
CEDA has confirmed recepit of the reprocessed Owl data.
All data is now archived.