Opened 15 months ago

Last modified 10 months ago

#704 new flight processing

TreeScapes, flight day 198/2023, Cardiff

Reported by: wja Owned by:
Priority: immediate Milestone:
Component: Processing: general Keywords:
Cc: Other processors:

Description

Flight over York for the TreeScapes project. Fenix1k, Ibis, PhaseOne, LiDAR data received.

Change History (38)

comment:1 Changed 15 months ago by wja

OS Basestations Accessed:

  • BREC
  • CARI
  • STRU
  • SWAS
  • TAUO

comment:2 Changed 14 months ago by wja

PhaseOne

Started converting images to TIF

comment:3 Changed 14 months ago by wja

Navigation Processing

Starting navigation processing.

comment:4 Changed 14 months ago by wja

  • Summary changed from TreeScapes, flight day 193/2023, York to TreeScapes, flight day 198/2023, Cardiff

comment:5 Changed 14 months ago by wja

PhaseOne

Conversion to TIF complete.

comment:6 Changed 14 months ago by wja

Navigation Processing

Post-processed navgiation SBET produced. Data quality looks good.

Lever arms used:

x0.154
y0.170
z1.377

comment:7 Changed 14 months ago by wja

Fenix1k Processing

Starting processing.

comment:8 Changed 14 months ago by wja

Fenix1k Processing

Data has mapping issues, like described for flight 193 (York, ticket 703).

It has been confirmed the IMU was fitted backwards, I have reprocessed the navigation data to account for this. Remapped the Fenix1k data with

I think there may be a slight timing error (possibly as small as a frame offset), along with a poor boresight, the Siena boresight does not look suitable. Will need to calculate the boresight of flight 190 to continue.

Processing with some time offsets.

comment:9 Changed 14 months ago by wja

Fenix1k Processing

Timing offset of +0.04 seconds appears to fix the error in lines 4 -6. Will process all lines with this and check. Boresight remains bad, however.

It's possible this +0.04 second offset is in fact a frame offset...

comment:10 Changed 14 months ago by wja

PhaseOne Processing

Have run the structure from motion processing on all the images on the lowest quality settings.

Orthomosaic resampled to 20cm to save processing power. Final res should be around 7cm.

Geoaccuracy needs improving (~0 - 3 m offset), no reason to think add GCPs wouldn't correct this though. Real time navigation taggs should still be suitable.

Will start creating GCPs from an elevation data source.

Will do final processing on a big gridnode once it is configured.

comment:11 Changed 14 months ago by wja

Fenix boresight underway, see ticket #706

comment:12 Changed 14 months ago by wja

PhaseOne Processing

14 ground control points created using features visible in low-res orthomosaic and 1m LiDAR DSM accessed from the Welsh Gov 2020-2022 dataset.

Will start processing when the 193 York flight is complete (ticket #703).

comment:13 Changed 14 months ago by wja

PhaseOne Processing

GCPs have been applied to calibrate model. Processing started (16:00, 22/09/23). GPU is now working on machine.

comment:14 Changed 14 months ago by wja

Fenix Processing

Running on a range of timing offsets (0 to 0.1, increments of 0.02). Note previous two flights have offsets of 0.04 (190 Alconbury) and 0.06 (York), respectively.

comment:15 Changed 14 months ago by wja

PhaseOne Processing

Processing on Metashape was unexpectedly halted over the weekend. Rerunning now (25/09 7:30).

comment:16 Changed 14 months ago by wja

Fenix1k Processing

Reprocessing with corrected (flipped) navigation and Alconbury (flight 190, ticket #706) boresight values:

Pitch0.45
Roll0.10
Yaw1.50

Timing error still present (previously looked like 0.04 seconds with flipped sensor and view vectors. Currently finding the best offset.

comment:17 Changed 14 months ago by wja

Fenix1k Processing

Processing all bands.

Summary of parameters:

  • Nav data 'flipped' 180
  • Timing offset +0.04 seconds for all lines
  • There are sections of dropped frames in lines 9.1 and 16.1
  • 65cm resolution pixels when mapped
  • -2 frame delay
  • Swath is masked to account for obstruction from aircraft
  • Using Alconbury (2023 190) boresight

comment:18 Changed 14 months ago by wja

Note: lines (even though they've been split) are timing out after 3 hours on the slurm grid when processing all bands. Have increased the timeout limit.

comment:19 Changed 14 months ago by wja

Fenix1k Processing

Processing flightlines to level1b complete.

XML generate failed, need to create a data quality report. Also need logsheet and ReadMe for delivery.

comment:20 Changed 14 months ago by wja

Started delivery generation as it may take a long time to run with these huge files.

comment:21 Changed 13 months ago by wja

PhaseOne Processing

Processing complete. DSM, DTM and orthomosaic exported.

Starting delivery creation.

comment:22 Changed 13 months ago by wja

Atmospheric Correction

Started ATCOR-4 preprocessing.

comment:23 Changed 13 months ago by wja

Ready for checking

Fenix1k level1b delivery create. Ready for checking.

comment:24 Changed 13 months ago by asm

Delivery check

Started

comment:25 Changed 13 months ago by asm

Fenix1k Delivery check

-Apl commands ran fine, needed to update first the name of DEM in config as it was not matching but is written correctly in the readme.
-In the readme, the table of underflows is not displaying correctly (please correct)
-And checking with Py6S the data indeed seems to have underflows in the SWIR and below band ~30 in the UV region. This matches with other Fenix1k flights.
Still running the other checks.

comment:26 Changed 13 months ago by wja

Thanks very much. Have corrected ReadMe.

comment:27 Changed 13 months ago by asm

Fenix1k Delivery check

All other checks have been run and all tests passed. I will zip the files and mark the project as ready to be delivered once that is done.

comment:28 Changed 13 months ago by asm

Fenix1k Delivery check

Mapped files were zipped correctly. I will mark this delivery as ready to go now.

comment:29 Changed 13 months ago by wja

PhaseOne

Delivery created for photogrammetry products. Called TreeScapes_23-198-dem-20231005

Ready for checking.

comment:30 Changed 13 months ago by asm

Fenix1k Delivery

Delivered via FTP (slot 6) and notified PI on 13/Oct/2023

comment:31 Changed 11 months ago by wja

Atmocspheric Correction

(same information and status as flight 193 ticket #703)

Data quality is has been inconsistent but is largely due to the changeable lighting conditions on the cloudy day this data was collected. Unfortunately we do not have AOT or in situ reluctance data to do any validations.

The obstructed field of fiew of the Fenix1k has also presented problems when processing within ATCOR4, as the sensor has been calibrated and charecterised using the entire field of view. These values have been masked, which ATCOR recognised. But the outputs have required a further masking step.

The following bands are being dropped due to noisey values throughout:

  • 1-49
  • 463-467
  • 508-594

These are not unexpected regions, the edge of the sensor is expected to be more noisy and the small section in the SWIR corresponds with water absorption.

The final processing is currently running on the grid: masking, dropping bands and mapping.

PI has also requested to subset the mapped data into 4 bands. I have code ready to do this. Spectral binning information below:

WavelengthFWHM
53010
70510
74010
80010
244520

comment:32 Changed 11 months ago by wja

Atmospheric Correction

New delivery comprising l2 unmapped, l2 mapped has been created. Just waiting for the spectral binning product to finish being generated.

No bands have been removed like previously mentioned. Will need to let users know to take care with wavelengths at either extreme.

comment:33 Changed 11 months ago by wja

Atmospheric Correction

Ready for checking.

comment:34 Changed 11 months ago by asm

Atmos. Corr. DC

Started.

comment:35 Changed 11 months ago by asm

Atmos. Corr. DC

I suggest the inclusion of Readme file, logsheet and at least the orthomosaic screenshot. The Readme should include what is included in this delivery, a note about quality of the data and a description on how the binned data was created including bands and FWHM.

The size of the bil files is correct and the correct bands are mapped for all files. I am looking a bit deeper into the data and this is all the checks I can run for this delivery.

comment:36 Changed 11 months ago by wja

Have updated documentation as suggest by reviewer.

There's an issue with mapping 4 files which has appeared to even influence spectra in limited regions. This has affected lines 1.00, 1.01, 9.01, 16.01 which have been removed from the delivery.

Zipping files. Then will start transfer to CEDA / JASMIN

comment:37 Changed 11 months ago by wja

Started transfer to JASMIN workspace and FTP slot 6.

PI notified.

comment:38 Changed 10 months ago by wja

Transfer complete.

Delivery has been finalised.

Note: See TracTickets for help on using tickets.