Opened 4 years ago

Closed 8 months ago

#537 closed flight processing (fixed)

RG13/08, AIRSAR, flight day 175b/2014, Wytham Woods

Reported by: knpa Owned by: benj
Priority: alpha 4 medium Milestone: 2014 data processing completion
Component: Archiving Keywords:
Cc: dac Other processors: dap, tec

Description (last modified by dap)

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/uk/RG13_08-2014_175b_Wytham_Woods_Plus_Airsar/

Data arrived from ARSF via network transfer on 25/06/2014

RG13/08 Scientific objective: RG13/08

RG13/08 Priority: 8?

AIRSAR Scientific objective: unknown

AIRSAR Priority: ?

PI: David Coomes

Notes: Fenix line 4 raw is tiny and looks like line failed. Lines 1,3,6 have slightly higher than usual difference between fps_set and fps_qpf (over 0.25 but under 1). Line 4 has a crazy fps_qpf value. Raw fenix 5 is not a proper flightline - don't processs this. There is FW.

Sensors:

Fenix (11/11/2014)
LiDAR (11/11/2014)
FW LiDAR (11/11/2014)
RCD (11/11/2014)

Change History (48)

comment:1 Changed 4 years ago by knpa

  • Description modified (diff)

comment:2 Changed 4 years ago by dap

Navigation Processing

OXFR Basestation Data
Latitude51 49 26.25367
Longitude-1 17 18.72376
El Height119.766m

comment:3 Changed 4 years ago by dac

Started RCD processing.

tec did raw RCD to Tiff conversion as was already set up on his machine.

Created and checked thumbnails.

Waiting on nav data

comment:4 Changed 4 years ago by dap

For the first five seconds of the first flightline, the navigation data has a quality of 2, not 1.

comment:5 Changed 4 years ago by dap

Navigation processing complete, now beginning LiDAR Processing.

Last edited 4 years ago by dap (previous) (diff)

comment:6 Changed 4 years ago by tec

Hyperspectral Processing
Started processing fenix.

comment:7 Changed 4 years ago by dac

Tagged RCD images using navigation data. No navigation data available for 153 and 154.

comment:8 Changed 4 years ago by tec

Navigation Processing
Messed up the nav, reprocessed nav, got even better solution. Copying over now.

comment:9 Changed 4 years ago by dap

Note, logsheet will need reproducing before delivery of LiDAR or Hyperspectral data, since navigation data had to be reprocessed.

Last edited 4 years ago by dap (previous) (diff)

comment:10 Changed 4 years ago by tec

Hyperspectral Processing
Line 4 is unusable.

SCT Values

Flightline FENIX
1 0.98
2 0.99
3 0.95
4 N/A
5 0.94
6 0.99
7 0.90
8 0.98
Last edited 4 years ago by tec (previous) (diff)

comment:11 Changed 4 years ago by tec

Hyperspectral Processing
Ready for delivery check

comment:12 Changed 4 years ago by dap

LiDAR Processing

Found a bug in alsproc on grid while trying to find roll correction values. Bug has been reported, will run on local machine instead.

comment:13 Changed 4 years ago by dac

RCD Processing
Reprocessed RCD data with new navigation data, have made delivery but script to produce readme is giving error that none of the images are in the list of photos from kml file - looking into.

comment:14 Changed 4 years ago by dap

LiDAR Processing

Found roll error for lines 163102 and 162305 to be -0.004264393 radians. Will now process other lines with this roll error to check for consistency.

comment:15 Changed 4 years ago by dap

LiDAR Processing

Found roll error for cross line (161004) to be -0.004434392 radians

comment:16 Changed 4 years ago by dap

LiDAR Processing

All roll errors have now been found. There was a lack of buildings in the overlaps between flightlines, so flat terrain was used where possible to determine these roll errors (table includes roll errors found previously):

FlightlineRoll errorPitch error
155434 (01)-0.003934392-0.000880469
155755 (02)-0.004434392-0.000880469
160129 (03)-0.003934392-0.000880469
160503 (04)-0.003934392-0.000880469
161004 (05)-0.004434392-0.000880469
162305 (AS1)-0.004264393-0.000880469
163102 (AS2)-0.004264393-0.000880469

Will now process deliverable files using ALSPP

comment:17 Changed 4 years ago by dac

RCD Processing

Found error with 'create_latex_camera_readme.py' was naming of files, which used 'AIRSAR' instead of the project code. Renamed files and ran OK. Also updated scripts to print error message for naming errors.

Ready for delivery check.

comment:18 Changed 4 years ago by dac

Starting Hyperspectral Delivery Check

comment:19 Changed 4 years ago by tec

RCD Delivery Check
Delivery check started.

comment:20 Changed 4 years ago by tec

RCD Delivery Check
Would say in data quality 00001 is over exposed, 0004-9 00014,00015 have blue'ish tint around edge.

Once you've done that, its ready to go.

comment:21 Changed 4 years ago by dac

Hyperspectral Delivery Check

Delivery check complete, some minor issues corrected. Ready to deliver.

  • Project code listed in read me as 'RG13/08 AIRSAR' rather than just 'RG13/08' - have corrected.
  • Lines 4 and 5 missing, 5 not on logsheet, line 4 mentioned in readme.
  • In mask file, refers to Eagle smear correction, have changed and in template to just smear correction (even though it only applies to Eagle sensor).
  • In level 1b data bands 240 - 260 are noisy (absorption features?) and in line 6 there is a large cloud shadow in centre of image causing some underflows.
  • Suggest changing delivery directory name from 'AIRSAR-175b' to use project code (RCD also needs to be changed).

comment:22 Changed 4 years ago by dac

RCD Processing
Added text to data quality. Renamed directory from AIRSAR-175b to use project code.
Will send with hyperspectral data once hard drives are available.

comment:23 Changed 4 years ago by tec

Awesome, thanks for corrections. I shall mark the hyperspectral and rcd as ready to deliver. Shall also prod ben about hard drives today.

comment:24 Changed 4 years ago by dap

LiDAR Processing

Read me has been created, now just need to measure elevation differences and add them in to the table in the read me.

comment:25 Changed 4 years ago by dap

LiDAR Processing

Added elevation differences to read me. Processed LiDAR data now ready for quality check.

comment:26 Changed 4 years ago by dac

LiDAR Delivery Check
Started LiDAR Delivery Check.

comment:27 Changed 4 years ago by dac

LiDAR Delivery Check

Delivery check complete, fixed some minor issues. Ready to deliver.

  • ASCII lidar files didn't have windows line endings - fixed.
  • Contains clouds, have been classified out as noise OK.
  • Full waveform data not requested but processed anyway.
  • Lines 1 -- 5 are missing a lot of waveforms when viewed in Waveviewer. These were all low altitude lines so I think the problems could be due to the time between each return pulse being too small for the digitiser. This is just a guess though.
  • No errors converting data to LAZ format.
  • Reworded sentance 'Elevation differences could not be measured between many of the low altitude lines and the cross-line as the areas of overlap lack in flat terrain.' to 'Elevation differences could not be measured between many of the low altitude lines and the cross-line as there were insufficient areas of flat terrain.'
  • Difference to ASTER (from demcompare.py) 0.6 m

comment:28 Changed 4 years ago by dap

Hyperspectral

Renamed individual flightline screenshots from <flightline name>_mapped_osng.bil.jpg to <flightline name>.jpg (ran rename.sh -f _mapped_osng.bil.jpg .jpg from screenshots directory).

comment:29 Changed 4 years ago by dap

  • Cc dac added
  • Component changed from Processing: general to Archiving
  • Description modified (diff)
  • Other processors set to dap, tec
  • Owner set to benj

All data delivered to Dr Coomes at the University of Cambridge (11/11/2014)

comment:30 Changed 4 years ago by benj

Gary says should also be sent to two other PIs: Anne Verhoef (Uni. of Reading) and Haiko Balzter (Uni. of Leicester). Dale, can you please prepare deliveries for them? Will send you the addresses.

comment:31 Changed 4 years ago by benj

Relevant additional project codes are GB13/10 (Anne Verhoef) and AS14/24.

comment:32 Changed 4 years ago by dap

AS14/24 and GB13/10 Deliveries

LiDAR, camera and hyperspectral data delivered to Haiko Balzter at Leicester University and Anne Verhoef at Reading University on 27/03/2015.

comment:33 Changed 3 years ago by asm

Hyperspectral Re-Processing
Started re-processing fenix.

comment:34 Changed 3 years ago by asm

SCT VALUES (RE-PROCESSING)
Found new SCT values. Flightline 4 is not usable as before, 3 and 5 did did not have usable features and needed to compare with google maps. Fightline 5 does not overlap but it was not included in previous deliveries.

Flightline FENIX
1 0.97
2 0.99
3 0.95
4 N/A
5 0.98
6 0.98
7 0.89
8 0.99
Last edited 3 years ago by asm (previous) (diff)

comment:35 Changed 3 years ago by lah

Fenix DC

  • copied in missing logsheet
  • replaced data quality report with most recent
  • scts fine (you shouldn't need to change these when reprocessing for binning problems)
  • moved extra readme files to processing directory(only pdf is required)
  • removed fodis folder
  • corrected dem name in hyp_genreadme-airborne.cfg, check_apl_cmd now runs fine and tifs look good. Have alsocorrected aplcorr command in readme.
  • proj_tidy doesn't like proj info name, probably because of AIRSAR, but this was delivered before, so fine
  • renamed screenshots
  • under & overflows look reasonable

Normal delivery check complete. Will thoroughly check spectra tomorrow.

comment:36 follow-up: Changed 3 years ago by lah

Fenix DC ctd

Have compared spectra with previously delivery and day 156, which is not affected by this problem. VNIR intensity looks about right now. However, all spectra have that SWIR broad peak around 1000 nm that we don't like. They have all used the 2014 rad calibration, so is not an overlap problem. We should check with simulated spectra that this is a real feature before delivery.

comment:37 in reply to: ↑ 36 Changed 3 years ago by lah

Replying to lah:

Fenix DC ctd

Have compared spectra with previously delivery and day 156, which is not affected by this problem. VNIR intensity looks about right now. However, all spectra have that SWIR broad peak around 1000 nm that we don't like. They have all used the 2014 rad calibration, so is not an overlap problem. We should check with simulated spectra that this is a real feature before delivery.

After discussing with mark1 and dac and comparing additional spectra from alternative sources, we have concluded that the spectra are now correct. However, as the PI has ground control points we will wait until these arrive for further quality checking before delivery.

comment:38 Changed 3 years ago by dac

Field data from Anne Verhoef has been copied to:

/users/rsg/arsf/arsf_data/2014/misc/GB13_10_ground_truth

Will check data against this before delivering,

comment:39 Changed 3 years ago by dac

Fenix DC

Checked against ground spectra, collected the previous day, for line 7 (with atmosphere simulated using Py6S) and simulated vegetation spectra for other lines, saved to processing/fenix_spectra_vs_6s.

Given errors in parametrisation of 6S, match with field data is acceptable, radiance is slightly lower than field spectra but overall shape is the same.

Zipping mapped files and marking as ready for delivery.

Last edited 3 years ago by dac (previous) (diff)

comment:40 Changed 3 years ago by dac

Fenix Delivery

Reprocessed data sent to David Coomes (RG13/08), Anne Verhoef (GB13/10) and Heiko Balzter (AS14/24) on USB hard drives 21/08/2015.

comment:41 Changed 3 years ago by dac

Fenix

Heiko Balzter confirmed receipt of hard drive (27/08/2015).

comment:42 Changed 3 years ago by asm

Archiving
Starting archiving. Files currently being uploaded.

comment:43 Changed 3 years ago by asm

Archiving
Finished. Notification mail: on hold

comment:44 Changed 3 years ago by asm

LiDAR delivery
Upon request, data delivered to Mathias Disney via FTP on 2015-10-22

comment:45 Changed 2 years ago by dac

Archiving

Processed data now available from NEODC (http://browse.ceda.ac.uk/browse/neodc/arsf/2014/RG13_08/RG13_08-2014_175b_Wytham_Woods_Plus_Airsar). Raw data not yet available. Will re-upload.

comment:46 Changed 9 months ago by asm

JSON files

This flight was for 2 diferent projects, holding on for now the creation of the JSON files as we need to discuss if we are creating 2 different JSON files.

comment:47 Changed 8 months ago by asm

Archiving

Raw data also archived and found at CEDA. Still no JSON files created but will change the status of the ticket to fixed.

comment:48 Changed 8 months ago by asm

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.