Opened 9 months ago
Last modified 8 months ago
#724 new flight processing
FIDEXII/23, flight day 235/2023, Canada
Reported by: | mark1 | Owned by: | |
---|---|---|---|
Priority: | immediate | Milestone: | |
Component: | Processing: general | Keywords: | |
Cc: | Other processors: |
Description
Data location: /users/rsg/arsf/arsf_data/2023/flight_data/canada/FIDEXII-2023_235_Canada
Data arrived from NERC-ARF via hard drive September/October 2023
PI: Martin Wooster
Sensors:
Fenix1k
IBIS
OWL
PhaseOne
Change History (24)
comment:1 Changed 9 months ago by mark1
comment:2 Changed 9 months ago by mark1
I ran out of time for unpacking - steps remaining to do:
- generate KML overviews
- make logsheet
- make cfg
- mv the already processed nav data into the project
UPDATE
Completed steps above on 3/4/2024
comment:3 Changed 8 months ago by wja
Fenix1k & Ibis Processing
Fenix1k needs flipping along with view vectors. Edges of swath also need masking.
Ibis also needs flipping.
Have replaced wavelegnths, created an APL config to include Fenix1k, Ibis and Owl.
comment:4 Changed 8 months ago by wja
Owl Processing
Radiometric processing running on the grid
comment:5 Changed 8 months ago by wja
Fenix1k, Ibis & Owl Processing
Most of the data has been collected from an altitude of ~2600m but some lines have been collected at ~300m.
Need to process these lines separately so we can input significantly different pixels sizes:
Altitude | Fenix1k | Ibis | Owl |
2600 | 1.8m | 4m | 2.9m |
300 | 0.2m | 0.45m | 0.35m |
Will make a separate sbatch script for the 300m lines.
Lines flown at 300m:
Fenix1l: 7, 12, 17, 18
Ibis: 5 10, 13, 14
Owl: 7, 8, 14, 15, 20, 21
comment:6 Changed 8 months ago by wja
Owl Processing
Radiometric calibration has run through. The following lines look bad. Could be due to the present of fire overflowing? Or bad calibration files?
2, 3, 4, 5, 6 & 18.
The following lines are really small. Probably aborted lines.
7, 13 & 19
comment:7 Changed 8 months ago by wja
Fenix1k, Ibis & Owl Processing
Handling different pixel sizes in the APL config for low altitude lines with apper_args_extra = -pixelsize 0.X 0.X
comment:8 Changed 8 months ago by wja
Fenix1b, Ibis Processing
Geoaccuracy looks decent at first look.
Running all bands on the grid, but some Fenix1k lines (see line 7 logs for example) are timing out even with a 3 hour limit!
comment:9 Changed 8 months ago by wja
mapper_args_extra is overwriting the default buffer size, which in turn is causing the timeouts on lines with high resolution (low altitude).
Replaced this per-line config parameter with mapper_args_extra = -pixelsize 0.2 0.2 -buffersize 4096
comment:10 Changed 8 months ago by wja
Fenix1k Processing
Resubmitted processing of low altitude lines.
comment:11 Changed 8 months ago by wja
Creating Fenix1k Delivery
comment:12 Changed 8 months ago by wja
Fenix1k Delivery
Logsheet created.
The following mapped lines are incomplete:
- 7.1
- 12.0
- 12.1
- 17.0
- 18.0
Reprocessing.
comment:13 Changed 8 months ago by wja
Fenix1k Delivery
Remaining lines have processing. Making screenshots and readme.
Ibis Delivery
Making screenshots and readme (some code needed updating for Ibis delivery creating).
comment:14 Changed 8 months ago by wja
comment:15 Changed 8 months ago by wja
Fenix1k Delivery
Ready for checking.
comment:16 Changed 8 months ago by wja
Ibis Delivery
Ready for checking
comment:17 Changed 8 months ago by asm
Fenix1k DC
Started.
comment:18 Changed 8 months ago by asm
Ibis DC
Started
comment:19 Changed 8 months ago by asm
Fenix1k DC
-Fodis directory has been removed.
-Py6S looks good, noisy but good shape. Lines 12, 17 and 18 have an incredible low SWIR integration time (0.09) and it looks like noise only.
-It is worth adding in the Readme a note about quality and high number of underflows due to smoke as well as noise is the data due to specially for lines 12, 17 and 18.
-Check_apl_cmd is not picking up some lines but the data looks good so this must be the script.
-All other checks have been run and passed.
Mapped lines are being zipped and should be ready for delivered once done
comment:20 Changed 8 months ago by wja
Fenix Delivery
- Added note to ReadMe
- Replaced viewvectors with the backwards mount view vectors
- Added missing line info XMLs
- Ran the APL command check on the lines that failed
Delivered to PI via FTP 18/04. PI notified.
comment:21 Changed 8 months ago by asm
IBIS DC
-Line 01 part 01 has the dropped frames included. This needs to be fixed in the config file and the line recreated.
Then on the readme, the file will not have underflows and therefore all lines have no underflows.
All other checks have been run and everything looks good. A bit noisy due to the smoke and such but nothing worth noting and much better quality than Fenix. I will have another look once the file is recreated and then we will be able to zip the files for delivery.
comment:22 Changed 8 months ago by wja
Ibis Delivery
Reprocessed line 1.01 without the dark frames.
Regenerated ReadMe, no underflows reported for line 1.01.
Zipping mapped lines.
comment:23 Changed 8 months ago by asm
Ibis Delivery
The file has been recreated fine. Checked both level1b and level3 and there are no darkframes. Also apl runs fine. The files have been mapped correctly.
The Readme table with underflows and overflows look good now (no underflows).
The only minor problem is that now the Readme is missing the table in page 3 with the line numbering and the orthomosaic screenshot. Please update and should be ready for delivery.
comment:24 Changed 8 months ago by wja
Ibis Delivery
Recreated Readme.
Sent to PI via FTP slot 5.
Three IBIS flight lines are missing hdr files (orig names):
Also these lines appear to be missing the "calibration" dirs and the quicklook png.