Opened 9 months ago

Last modified 5 months ago

#647 new flight processing

CA18/208, flight day 234/2018, Ontario, Canada

Reported by: dac Owned by:
Priority: immediate Milestone:
Component: Processing: general Keywords:
Cc: Other processors:

Description

Data location: /users/rsg/arsf/arsf_data/2018/flight_data/canada/CA18_207-2018_234_Ontario

Data arrived from NERC-ARF via hard driver on 12/09/2018

Scientific objective: Canadian Wildfire Observations with SLSTR & Aircraft: Directional Effects of Thermally Emitted Radiation & Exploration of Flaming/Smouldering Partitioning and Plume Trace Gas Emission Ratios. Fenix and Owl data are being processed to aid geocorrection of other thermal sensors flown on the aircraft.

PI: Martin Wooster

BAS Project Code: D207

  • Missing Owl calibration data
  • Owl line 234-18_234-18_2018-08-22_16-53-50 contained no header data so removed.
  • Navigation data (.sol format) and level1b (unmapped) data to be delivered first. The level1b data will be used to determine which lines map.

Sensors:

  • Fenix (requested, flown)
  • Owl (requested, flown)

Change History (27)

comment:1 Changed 9 months ago by dac

Fenix

Quickly processed Fenix data to 1b to test script. Command used:

batch_aplcal.py -i hyperspectral/fenix/ -o processing/hyperspectral/flightlines/level1b/ \
               -l processing/hyperspectral/logfiles/ \
              --calfile ~arsf/calibration/2018/fenix/fenix_201805 --grid
Last edited 9 months ago by dac (previous) (diff)

comment:2 Changed 9 months ago by asm

Unpacking
-There are 11 owl lines and 11 fenix.
-All owl lines have T1 and T2 calibration files

comment:3 Changed 9 months ago by asm

Owl Processing

Started in order to create the level1b files.

comment:4 Changed 9 months ago by dac

Navigation Processing

Subset basestation data to single day using:

teqc -st 20180822000000 -e 20180822235959 +nav Default_9488_0509_114526.18n \
           Default_9488_0509_114526.m00 > Default_9488_0509_114526.18o

Got antenna info from file (LEIAX1202GG) and used height of 0.889 m provided by Gary.

Basestation PPP result

Latitude 51 04 01.40114
Longitude -93 48 14.26959
Ellipsoid Height 345.856 m

comment:5 Changed 9 months ago by dac

Navigation Processing

Processed navigation data. Reduced elevation mask to 8 degrees to get better result. Position separation is higher than normal but within 10 cm for most of flight.

comment:6 Changed 8 months ago by wja

Hyperspectral Processing
SRTM DEM created.

comment:7 Changed 8 months ago by wja

Owl Processing
Starting Owl processing.

comment:8 Changed 8 months ago by dac

Fenix Processing

Needed to increase buffer around DEM to 0.5 degrees so generate_apl_config.py would run through.

comment:9 Changed 8 months ago by dac

Fenix Processing

Found SCT values

Flightline FENIX
1 2
2 2
3 2
4 2.9
5 2
6 2
7 2
8 2
9 2
10 2
11 1

Changed pixel size from 5 m to 3 m. Given altitude is 3.5 m at nadir and 3.76 at the edge of swath. Went for 3 rather than 4 as high resolution is required to match other cameras to.

Starting mapping all bands.

Last edited 8 months ago by dac (previous) (diff)

comment:10 Changed 8 months ago by wja

Owl Processing

FlightlineSCT
1N/A
22.00
32.00
42.00
52.00
62.00
72.00
82.00
92.00
102.00
112.00

Flightline 1 header file reports 1024 lines, however, level1b *BIL shows ~520 lines (no errors in log). This is too small to be useful and will be omitted from further processing.
Three band BIL of flightline 10 is noisy.

Last edited 8 months ago by wja (previous) (diff)

comment:11 Changed 8 months ago by dac

Fenix Processing

All bands mapped. Generating delivery.

comment:12 Changed 8 months ago by wja

Owl Processing
Flightline 10 T1 calibration file has some strange features resulting in poor quality. Reprocessed with calibration files (T1 & T2) from flightline 11 (close and only flight with same integration time) and looks acceptable.
All mapped flightlines have been processed.
Generating delivery.

comment:13 Changed 8 months ago by dac

Fenix Processing

Delivery created, ready for checking.

comment:14 Changed 8 months ago by wja

Owl Delivery
Delivery created, ready for delivery check.

comment:15 Changed 8 months ago by dac

Owl

Starting delivery check.

comment:16 Changed 8 months ago by wja

Fenix Delivery Check
Started Fenix delivery check.

comment:17 Changed 8 months ago by dac

Owl delivery check
Generally looks good, couple of comments

  • In data quality section of readme non-latex character '11aZa' needs to be replaced.
  • No underflows or overflows listed in readme - I'm not sure this is correct, suggest re-running autoqc.py script and checking for errors.
  • check_ap_cmd ran through OK, three band for line 9 looks a bit odd though.
  • Spectral for line 9 has some weird features, should add a note to the readme about it.

Once these are addressed ready to deliver.

comment:18 Changed 8 months ago by wja

Fenix Delivery Check

  • Logsheet line 1 has very short duration (looks like from Owl header).
  • Flightline 4 is missing from navigation/, line_information/ & mapped/
  • flightline/fodis needs removing.
  • Logsheet metadata: weather = '?', probably best if left blank.
  • Read_Me: 'logsheet file name' table is incomplete. Also picture of Dornier.
  • XML files: '&' in project discription and title needs changing to '&' so that XMLs can be opened.
  • Py6S looks okay, although perhaps not as precise as normal in some peaks? Can sheck this in wja's machine /tmp/fenix_vs_py6s
  • check_apl_cmd ran through and looks good.
  • Spectra of flightline 11 should be looked at, very little reflectance in higher wavelengths. i notice it has a shorter integration time than the others.
Last edited 8 months ago by wja (previous) (diff)

comment:19 Changed 8 months ago by dac

Fenix processing

  • Updated logsheet to use times from Fenix headers.
  • Removed fodis dir
  • Updated logsheet names in readme
  • Replaced '&' in XML files
  • Spectra look OK to me compared to simulations in Py6S
  • Added note about line 11 in readme. The integration time is very short so I assume it was intentional given project aims and earlier discussions with PI.

Line 4 failed due to navigation data going backwards in time, will investigate and try to process.

comment:20 Changed 8 months ago by wja

Owl Delivery

  • Read_Me corrections:
    • non-LaTex charecter removed
    • auto.qc script gave no under-/over-flows; section has been removed with a mention that they should be expected due to the low integration times of this flight.
    • Description of /mosaic/ directory edited to match that of Fenix delivery.
  • Flightline 9 reprocessed. T1 calibration file contain what looks like errorneous data starting after line 274, a new T1 calibration was created and used up until this line. Spectra looks as expected.

comment:21 Changed 8 months ago by wja

Owl/Fenix Processing
Flightline number reference:

Logsheet (hardcopy)OwlFenix
-1 (Failed)-
121
232
343
454
5-5
6.166
777
8.188
999
8.21010
6.21111

comment:22 Changed 8 months ago by dac

Owl Delivery Check

Problems have been fixed and reprocessed line 9 looks good. Zipping mapped files and marking as ready to deliver.

comment:23 Changed 8 months ago by dac

Fenix processing

Processed line 4 with -force flag and manually synced navigation using SCT value of 2.9. Added to existing delivery and recreated mosaic. Added note about line 4 to readme. Ran check_apl_command to check all files where there for line 4.

comment:24 Changed 8 months ago by wja

Fenix Delivery
Ready to deliver.

comment:25 Changed 8 months ago by wja

Fenix/Owl Delivery
Deliveries hardlinked to FTP slot 1, notification e-mail sent to PI. Delivery finalised.

comment:26 Changed 5 months ago by wja

Archiving
JSONs created and added to PostGIS. In the queue to rsync to CEDA.

comment:27 Changed 5 months ago by wja

Archiving
Upload to CEDA complete (18/01/2019)

Note: See TracTickets for help on using tickets.