Opened 15 months ago
Last modified 10 months ago
#704 new flight processing
TreeScapes, flight day 198/2023, Cardiff
Reported by: | wja | Owned by: | |
---|---|---|---|
Priority: | immediate | Milestone: | |
Component: | Processing: general | Keywords: | |
Cc: | Other processors: |
Description
Flight over York for the TreeScapes project. Fenix1k, Ibis, PhaseOne, LiDAR data received.
Change History (38)
comment:1 Changed 15 months ago by wja
comment:2 Changed 14 months ago by wja
PhaseOne
Started converting images to TIF
comment:3 Changed 14 months ago by wja
Navigation Processing
Starting navigation processing.
comment:4 Changed 14 months ago by wja
- Summary changed from TreeScapes, flight day 193/2023, York to TreeScapes, flight day 198/2023, Cardiff
comment:5 Changed 14 months ago by wja
PhaseOne
Conversion to TIF complete.
comment:6 Changed 14 months ago by wja
Navigation Processing
Post-processed navgiation SBET produced. Data quality looks good.
Lever arms used:
x | 0.154 |
y | 0.170 |
z | 1.377 |
comment:7 Changed 14 months ago by wja
Fenix1k Processing
Starting processing.
comment:8 Changed 14 months ago by wja
Fenix1k Processing
Data has mapping issues, like described for flight 193 (York, ticket 703).
It has been confirmed the IMU was fitted backwards, I have reprocessed the navigation data to account for this. Remapped the Fenix1k data with
I think there may be a slight timing error (possibly as small as a frame offset), along with a poor boresight, the Siena boresight does not look suitable. Will need to calculate the boresight of flight 190 to continue.
Processing with some time offsets.
comment:9 Changed 14 months ago by wja
Fenix1k Processing
Timing offset of +0.04 seconds appears to fix the error in lines 4 -6. Will process all lines with this and check. Boresight remains bad, however.
It's possible this +0.04 second offset is in fact a frame offset...
comment:10 Changed 14 months ago by wja
PhaseOne Processing
Have run the structure from motion processing on all the images on the lowest quality settings.
Orthomosaic resampled to 20cm to save processing power. Final res should be around 7cm.
Geoaccuracy needs improving (~0 - 3 m offset), no reason to think add GCPs wouldn't correct this though. Real time navigation taggs should still be suitable.
Will start creating GCPs from an elevation data source.
Will do final processing on a big gridnode once it is configured.
comment:11 Changed 14 months ago by wja
Fenix boresight underway, see ticket #706
comment:12 Changed 14 months ago by wja
PhaseOne Processing
14 ground control points created using features visible in low-res orthomosaic and 1m LiDAR DSM accessed from the Welsh Gov 2020-2022 dataset.
Will start processing when the 193 York flight is complete (ticket #703).
comment:13 Changed 14 months ago by wja
PhaseOne Processing
GCPs have been applied to calibrate model. Processing started (16:00, 22/09/23). GPU is now working on machine.
comment:14 Changed 14 months ago by wja
Fenix Processing
Running on a range of timing offsets (0 to 0.1, increments of 0.02). Note previous two flights have offsets of 0.04 (190 Alconbury) and 0.06 (York), respectively.
comment:15 Changed 14 months ago by wja
PhaseOne Processing
Processing on Metashape was unexpectedly halted over the weekend. Rerunning now (25/09 7:30).
comment:16 Changed 14 months ago by wja
Fenix1k Processing
Reprocessing with corrected (flipped) navigation and Alconbury (flight 190, ticket #706) boresight values:
Pitch | 0.45 |
Roll | 0.10 |
Yaw | 1.50 |
Timing error still present (previously looked like 0.04 seconds with flipped sensor and view vectors. Currently finding the best offset.
comment:17 Changed 14 months ago by wja
Fenix1k Processing
Processing all bands.
Summary of parameters:
- Nav data 'flipped' 180
- Timing offset +0.04 seconds for all lines
- There are sections of dropped frames in lines 9.1 and 16.1
- 65cm resolution pixels when mapped
- -2 frame delay
- Swath is masked to account for obstruction from aircraft
- Using Alconbury (2023 190) boresight
comment:18 Changed 14 months ago by wja
Note: lines (even though they've been split) are timing out after 3 hours on the slurm grid when processing all bands. Have increased the timeout limit.
comment:19 Changed 14 months ago by wja
Fenix1k Processing
Processing flightlines to level1b complete.
XML generate failed, need to create a data quality report. Also need logsheet and ReadMe for delivery.
comment:20 Changed 14 months ago by wja
Started delivery generation as it may take a long time to run with these huge files.
comment:21 Changed 13 months ago by wja
PhaseOne Processing
Processing complete. DSM, DTM and orthomosaic exported.
Starting delivery creation.
comment:22 Changed 13 months ago by wja
Atmospheric Correction
Started ATCOR-4 preprocessing.
comment:23 Changed 13 months ago by wja
Ready for checking
Fenix1k level1b delivery create. Ready for checking.
comment:24 Changed 13 months ago by asm
Delivery check
Started
comment:25 Changed 13 months ago by asm
Fenix1k Delivery check
-Apl commands ran fine, needed to update first the name of DEM in config as it was not matching but is written correctly in the readme.
-In the readme, the table of underflows is not displaying correctly (please correct)
-And checking with Py6S the data indeed seems to have underflows in the SWIR and below band ~30 in the UV region. This matches with other Fenix1k flights.
Still running the other checks.
comment:26 Changed 13 months ago by wja
Thanks very much. Have corrected ReadMe.
comment:27 Changed 13 months ago by asm
Fenix1k Delivery check
All other checks have been run and all tests passed. I will zip the files and mark the project as ready to be delivered once that is done.
comment:28 Changed 13 months ago by asm
Fenix1k Delivery check
Mapped files were zipped correctly. I will mark this delivery as ready to go now.
comment:29 Changed 13 months ago by wja
PhaseOne
Delivery created for photogrammetry products. Called TreeScapes_23-198-dem-20231005
Ready for checking.
comment:30 Changed 13 months ago by asm
Fenix1k Delivery
Delivered via FTP (slot 6) and notified PI on 13/Oct/2023
comment:31 Changed 11 months ago by wja
Atmocspheric Correction
(same information and status as flight 193 ticket #703)
Data quality is has been inconsistent but is largely due to the changeable lighting conditions on the cloudy day this data was collected. Unfortunately we do not have AOT or in situ reluctance data to do any validations.
The obstructed field of fiew of the Fenix1k has also presented problems when processing within ATCOR4, as the sensor has been calibrated and charecterised using the entire field of view. These values have been masked, which ATCOR recognised. But the outputs have required a further masking step.
The following bands are being dropped due to noisey values throughout:
- 1-49
- 463-467
- 508-594
These are not unexpected regions, the edge of the sensor is expected to be more noisy and the small section in the SWIR corresponds with water absorption.
The final processing is currently running on the grid: masking, dropping bands and mapping.
PI has also requested to subset the mapped data into 4 bands. I have code ready to do this. Spectral binning information below:
Wavelength | FWHM |
530 | 10 |
705 | 10 |
740 | 10 |
800 | 10 |
2445 | 20 |
comment:32 Changed 11 months ago by wja
Atmospheric Correction
New delivery comprising l2 unmapped, l2 mapped has been created. Just waiting for the spectral binning product to finish being generated.
No bands have been removed like previously mentioned. Will need to let users know to take care with wavelengths at either extreme.
comment:33 Changed 11 months ago by wja
Atmospheric Correction
Ready for checking.
comment:34 Changed 11 months ago by asm
Atmos. Corr. DC
Started.
comment:35 Changed 11 months ago by asm
Atmos. Corr. DC
I suggest the inclusion of Readme file, logsheet and at least the orthomosaic screenshot. The Readme should include what is included in this delivery, a note about quality of the data and a description on how the binned data was created including bands and FWHM.
The size of the bil files is correct and the correct bands are mapped for all files. I am looking a bit deeper into the data and this is all the checks I can run for this delivery.
comment:36 Changed 11 months ago by wja
Have updated documentation as suggest by reviewer.
There's an issue with mapping 4 files which has appeared to even influence spectra in limited regions. This has affected lines 1.00, 1.01, 9.01, 16.01 which have been removed from the delivery.
Zipping files. Then will start transfer to CEDA / JASMIN
comment:37 Changed 11 months ago by wja
Started transfer to JASMIN workspace and FTP slot 6.
PI notified.
comment:38 Changed 10 months ago by wja
Transfer complete.
Delivery has been finalised.
OS Basestations Accessed: