GB18/56, flight day 135/2018, Suffolk, Nottinghamshire and Lincolnshire

[asm]

Data location: ~arsf/arsf_data/2018/flight_data/uk/GB18_56-2018_135_Methane_Sites

Data arrived via USB hard drive on 11/June/2018.

Scientific objective: Investigate well-defined local methane emission sources in the UK to apply and evaluate the hyperspectral processing method currently developed at the University of Leicester.

Priority: High for Fenix, medium for Owl

PI: Hartmut Boesch

Sensors:

• Fenix (requested, flown)
• OWL (requested, flown)
• PhaseOne (not requested, flown)

[asm]

Unpacking.

Project unpacked. Problems found so far:
-Owl has not been renamed for now as the GPS starting time is not recorded in the files.
-No owl darkframes recorded
-Calibration files also placed in different directories (will need to identify each one by the time)
-No timestamps for the nav files.
-All fenix f98 are empty.
-A few dropped frames incidents for the Fenix:
There has been 2 incidents on different lines with a total of 34 dropped frames recorded on file FENIX135-18-14.log
There has been 2 incidents on different lines with a total of 39 dropped frames recorded on file FENIX135-18-7.log
There has been 2 incidents on different lines with a total of 32 dropped frames recorded on file FENIX135-18-1.log
There has been 2 incidents on different lines with a total of 33 dropped frames recorded on file FENIX135-18-5.log

[asm]

Unpacking.

-Renamed Owl files as the time in the header (not GPS but internal) will help processing. Will need to add GPS Starting time to process
-The owl lines do not have darkframes so will need to use the calibration files. Match each calibration with each file and stitch them. Time consuming, specially combined with no navigation files to use and the GPS timing issue.
No KML created yet or logsheet but project but otherwise the unpacking is done.

[asm]

Navigation Processing

Started in order to assess if the fps problem with the Fenix still persists. Needed to use teqc:
teqc -st 180515094209 -e 180515150330 +nav /tmp/teqc_135_2018/135_2018.nav > /tmp/teqc_135_2018/135_2018.18o

Done basestation verification:
Lat: 52 04 27.12178
Lon: -0 37 24.79006
EllHeight: 151.206 m

Done navigation as well. Good results, accuracy is always better than 0.2m in lat and lon sepparation and with ambiguity fix 2. All parameters looking good, best results we have in a long time. Copying files across and navigation will be finished.

[asm]

Fenix processing

Used most recent boresight values found by wja. FPS problems with the Fenix still persists on this flight, I did manually estimated the fps for each flightline over the last week to have a valid geocorrection. Used qgis Open Street Map vector data and google maps to confirm the best values. The results are:

FENIX	SCT	FPS
1	3.52	30.96
2	3.01	31.00
3	2.97	30.98
4	27.87	31.00
5	4.16	30.96
6	22.27	30.96
7	52.05	30.96
8	2.41	30.96
9	53.98	30.96
10	3.30	30.95
11	53.98	30.95
12	7.79	31.00
13	2.92	30.95
14	3.10	31.00
15	30.98	31.00

[asm]

Fenix processing

Delivery created. It has been suggested to split the delivery into the 3 location with 3 sorties but keep them together as raw data so will address that. Logsheet can't be automatically generated so will tailor one for each site.

[asm]

Fenix Delivery

I have created 3 different deliveries with sorties a, b and c with each readme. I am finalising the logsheets and filling them manually but otherwise the project is ready for DC in the meantime.

[dac]

Fenix Delivery Check

Starting delivery check

[dac]

Fenix Delivery Check

All three projects :

• Readme contains a photo of the Dornier - remove it
• Wierd characters in 'Quality and issues with data' section (likely LaTeX problem)
• "framerate...not recorded correctly" change to "framerate ...not set correctly" If the correct framerate was recorded in the logsheet but not the file it wouldn't be a problem as could just update the header based on the logsheet.
• "obtained and validated using Google Maps and Open Street Map vector data via qgis." change to "obtained, checked against Google Maps and Open Street Map data via QGIS." (I assumed you used the raster version of open street map not the actual vector files).
• The DEM covers the area of all projects. However, the filesize is only 170 MB so I think OK.
• Metadata needs to be edited to contain site letter and only the name of the site it contains (sed -i is your friend here!)

Looking into individual areas now.

[asm]

Fenix Delivery

Placed one logsheet in each delivery using the screenshot created by the JSON scripts with the linestring option. Will address comments by dac now.

[asm]

Fenix Delivery

Mapped files have been zipped correctly, no problems in the logfiles.

[dac]

*Fenix Delivery Check

• check_apl_cmd runs through OK
• Checked spectra using Py6S script and look good
• Issues resolved with readme
• Metadata fixed
• Logsheets look good
• b is nearer Lincoln than Nottingham but within the Nottinghamshire boundary county.

Everything else looks good. Ready to deliver.

[asm]

Fenix Delivery

Delivered via FTP (slot 2) and email sent to PI on 06/July/2018.

[dac]

Digital Camera

Converted raw images to tiff to check GPS sync

[wja]

Owl Processing

Started Owl processing.

Total of 19 flightlines. However three of these just contain T1 calibration data and a differnt naming custom. These three lines were recorded at 11:46, 11:55 and 13:00 according to their titles (seem to be UTC).

Of the other 16 lines:

-All appear to have nav sync recorded.
-Flightline 3 has darkframes recorded. All other lines (1-16) have DARKREF*.raw data recorded.
-15 T1 and T2 calibration files (not including the three T1 data previously mentioned). These are in separate directories labelled by date.

[wja]

Owl Processing

Owl data has been stitched to their respective darkframes.
Level 1b processing is currently in progress.

batch_cal_owl_proctool.py originally returned errors about not being able to find flightline data. This was corrected by adding the 'T' before the times on the calibration file directorys in the config file.

[wja]

Owl Processing

Level1b processing undertaken on all 16 fllightlines.
Flightlines look good with no obvious issues. However, flightline 2 is very short (100 rows in total). I will investigate this and then continue with processing georeferenced data.

[wja]

Owl Processing

Flightline 2 has 100 lines recorded in hdr - no issue with processing.
Currently processing georeferenced data. "No sync messages found in Specim nav file" so will continue without using navigation file and will find SCTs manually.

[wja]

Owl Processing

Owl processing for georeferenced data for all bands currently under way.

Processing has been done without nav sync. The following SCT values have been detected:

Flightline	SCT Value
1	4.26
2	N/A
3	4.70
4	2.00
5	3.00
6	2.47
7	2.68
8	3.00
9	2.49
10	2.00
11	2.52
12	3.57
13	2.31
14	3.00
15	2.67
16	2.00

Flightline 2 is just 100 lines long. I could not find the SCT for it so it has been ommitted form further processing.

[wja]

Owl Processing

Updated SCT values:

Flightline	SCT Value
1	4.26
2	N/A
3	4.70
4	1.98
5	3.05
6	2.49
7	2.68
8	2.96
9	2.49
10	2.00
11	2.52
12	3.57
13	2.31
14	2.92
15	2.67
16	1.95

[wja]

Creating Owl Delivery

Creating Owl delivery. Will split delivery into three sites (designated by a sortie) in accordance with the Fenix delivery:

Delivery	Flightlines
135a	1, 2 (only level1b), 3 - 5
135b	6 - 12
135c	13-16

[wja]

Creating Owl Delivery

Three deliveries are being created (labelled a, b & c, corresponding with the areas of the Fenix deliveries).
Flightlines names and XML (project info & line info) data have had the correct letter added after 135.

To do:

• Create screenshot mosaics for a, b & c.
• Update logfiles.
• Create ReadMes.

[wja]

Creating Owl Delivery

Screenshots generated for 135 a, b & c.

[wja]

Creating Owl Delivery

Logsheets updated for a, b & c. Data quality report also present.

[wja]

Creating Owl Delivery

• Level1b sizes were not matching hdr files. These have been successfully reprocessed.
• Level1b data were renamed from *_mask.bil to *proc_mask.bil for Read_Me generation.
• Read_Mes have been generated with some edits. 'Logsheet flightline name' table in the 'Delivery Contents' section has been changed to 'Logsheet flightline ID' and now includes the logsheet (& Fenix) flightline numbers to account for the extra Owl line. Listed Owl filenames remain the same (e.g 1, 3, 4, 5 ... , 16).
• Data quality remarks added to 135a Read_Me.

[wja]

Delivery Created

Awaiting delivery check.

[dac]

Owl Data

Starting delivery check.

[dac]

Owl Delivery Check

All deliveries

• GB18_56-135-owl-20180813 can be removed if no longer needed.
• Suggest changing date for delivery folder to 20180824 to match readme as there has been a long gap between creating the delivery folder and finishing the delivery while waiting for data quality report.
• In logsheet project name is "Novel Remote Sensing Methods for Methane Emissions", "Novel Remote Sensing Methods for detecting Methane Emissions" would be better
• Have comma after each "Logsheet flightline ID" in table on page 2, please remove.
• Missing ASCII format FOV vector, also need to remove *.aux.xml file
• Level 1b files need to be renamed
• check_apl_cmd doesn't run (likely due to the level1b files not having the correct name.

2018/135a

• Logsheet is missing flightline 5 (line 2 was short and isn't in delivery).

2018/135b

• Logsheet includes line 5 which is part of a (looking at time and location) but not line 12

2018/135c

• Logsheet includes line 12 which is part of b.

Once these problems have been resolved will recheck.

[wja]

Owl Delivery Check
Amendments have been made to all deliveries apart from the logsheets.
Final flightline (c; 16) seems to be missing, recorded on hardcopy logsheet as starting at ~14:01 and ending at ~14:03. Will investigate.

[wja]

Owl Delivery Check
Logsheets have been corrected, Read_Mes have been remade.
Ready to deliver.

[dac]

Owl Delivery Check

All problems have been resolved. check_apl_cmd runs though OK and outputs look good.

Ready to deliver.

[wja]

Owl Delivery

Placed on FTP slot 2, notification e-mail sent to PI.

[wja]

Archiving
Started archiving process. JSON files created separately for each of the three project areas.

[dac]

Archiving

Created three project folders for each area and symlinked data into them.

[wja]

Archiving
Project is being archived as three separate flights (135a, 135b, 135c) in accordance with the deliveries.
JSON files have been created and added to postGIS.
Beginning rsync to CEDA.

[wja]

Archiving
Rsync to CEDA complete for 135a, b & c (14/01/2019).