Opened 9 months ago

Last modified 8 months ago

#724 new flight processing

FIDEXII/23, flight day 235/2023, Canada

Reported by: mark1 Owned by:
Priority: immediate Milestone:
Component: Processing: general Keywords:
Cc: Other processors:

Description

Data location: /users/rsg/arsf/arsf_data/2023/flight_data/canada/FIDEXII-2023_235_Canada

Data arrived from NERC-ARF via hard drive September/October 2023
PI: Martin Wooster
Sensors:
Fenix1k
IBIS
OWL
PhaseOne

Change History (24)

comment:1 Changed 9 months ago by mark1

Three IBIS flight lines are missing hdr files (orig names):

  • 235-23_DMC001_IBIS_2023-08-23_15-08-27
  • 235-23_DMC001_IBIS_2023-08-23_15-13-53
  • 235-23_DMC001_IBIS_2023-08-23_16-44-36

Also these lines appear to be missing the "calibration" dirs and the quicklook png.

Last edited 9 months ago by mark1 (previous) (diff)

comment:2 Changed 9 months ago by mark1

I ran out of time for unpacking - steps remaining to do:

  • generate KML overviews
  • make logsheet
  • make cfg
  • mv the already processed nav data into the project

UPDATE

Completed steps above on 3/4/2024

Last edited 8 months ago by mark1 (previous) (diff)

comment:3 Changed 8 months ago by wja

Fenix1k & Ibis Processing

Fenix1k needs flipping along with view vectors. Edges of swath also need masking.

Ibis also needs flipping.

Have replaced wavelegnths, created an APL config to include Fenix1k, Ibis and Owl.

comment:4 Changed 8 months ago by wja

Owl Processing

Radiometric processing running on the grid

comment:5 Changed 8 months ago by wja

Fenix1k, Ibis & Owl Processing

Most of the data has been collected from an altitude of ~2600m but some lines have been collected at ~300m.

Need to process these lines separately so we can input significantly different pixels sizes:

AltitudeFenix1kIbisOwl
26001.8m4m2.9m
3000.2m0.45m0.35m

Will make a separate sbatch script for the 300m lines.

Lines flown at 300m:

Fenix1l: 7, 12, 17, 18
Ibis: 5 10, 13, 14
Owl: 7, 8, 14, 15, 20, 21

Last edited 8 months ago by wja (previous) (diff)

comment:6 Changed 8 months ago by wja

Owl Processing

Radiometric calibration has run through. The following lines look bad. Could be due to the present of fire overflowing? Or bad calibration files?

2, 3, 4, 5, 6 & 18.

The following lines are really small. Probably aborted lines.

7, 13 & 19

comment:7 Changed 8 months ago by wja

Fenix1k, Ibis & Owl Processing

Handling different pixel sizes in the APL config for low altitude lines with apper_args_extra = -pixelsize 0.X 0.X

comment:8 Changed 8 months ago by wja

Fenix1b, Ibis Processing

Geoaccuracy looks decent at first look.

Running all bands on the grid, but some Fenix1k lines (see line 7 logs for example) are timing out even with a 3 hour limit!

comment:9 Changed 8 months ago by wja

mapper_args_extra is overwriting the default buffer size, which in turn is causing the timeouts on lines with high resolution (low altitude).

Replaced this per-line config parameter with mapper_args_extra = -pixelsize 0.2 0.2 -buffersize 4096

comment:10 Changed 8 months ago by wja

Fenix1k Processing

Resubmitted processing of low altitude lines.

comment:11 Changed 8 months ago by wja

Creating Fenix1k Delivery

comment:12 Changed 8 months ago by wja

Fenix1k Delivery

Logsheet created.

The following mapped lines are incomplete:

  • 7.1
  • 12.0
  • 12.1
  • 17.0
  • 18.0

Reprocessing.

comment:13 Changed 8 months ago by wja

Fenix1k Delivery

Remaining lines have processing. Making screenshots and readme.

Ibis Delivery

Making screenshots and readme (some code needed updating for Ibis delivery creating).

comment:14 Changed 8 months ago by wja

Last edited 8 months ago by wja (previous) (diff)

comment:15 Changed 8 months ago by wja

Fenix1k Delivery

Ready for checking.

comment:16 Changed 8 months ago by wja

Ibis Delivery

Ready for checking

comment:17 Changed 8 months ago by asm

Fenix1k DC

Started.

comment:18 Changed 8 months ago by asm

Ibis DC

Started

comment:19 Changed 8 months ago by asm

Fenix1k DC

-Fodis directory has been removed.
-Py6S looks good, noisy but good shape. Lines 12, 17 and 18 have an incredible low SWIR integration time (0.09) and it looks like noise only.
-It is worth adding in the Readme a note about quality and high number of underflows due to smoke as well as noise is the data due to specially for lines 12, 17 and 18.
-Check_apl_cmd is not picking up some lines but the data looks good so this must be the script.
-All other checks have been run and passed.
Mapped lines are being zipped and should be ready for delivered once done

comment:20 Changed 8 months ago by wja

Fenix Delivery

  • Added note to ReadMe
  • Replaced viewvectors with the backwards mount view vectors
  • Added missing line info XMLs
  • Ran the APL command check on the lines that failed

Delivered to PI via FTP 18/04. PI notified.

comment:21 Changed 8 months ago by asm

IBIS DC

-Line 01 part 01 has the dropped frames included. This needs to be fixed in the config file and the line recreated.
Then on the readme, the file will not have underflows and therefore all lines have no underflows.

All other checks have been run and everything looks good. A bit noisy due to the smoke and such but nothing worth noting and much better quality than Fenix. I will have another look once the file is recreated and then we will be able to zip the files for delivery.

comment:22 Changed 8 months ago by wja

Ibis Delivery

Reprocessed line 1.01 without the dark frames.

Regenerated ReadMe, no underflows reported for line 1.01.

Zipping mapped lines.

comment:23 Changed 8 months ago by asm

Ibis Delivery

The file has been recreated fine. Checked both level1b and level3 and there are no darkframes. Also apl runs fine. The files have been mapped correctly.

The Readme table with underflows and overflows look good now (no underflows).
The only minor problem is that now the Readme is missing the table in page 3 with the line numbering and the orthomosaic screenshot. Please update and should be ready for delivery.

comment:24 Changed 8 months ago by wja

Ibis Delivery

Recreated Readme.

Sent to PI via FTP slot 5.

Note: See TracTickets for help on using tickets.