Opened 4 years ago

Closed 3 months ago

#535 closed flight processing (fixed)

GB14/04, flight day 219b/2014, Aberfoyle

Reported by: knpa Owned by:
Priority: alpha 4 medium Milestone: 2014 data processing completion
Component: Processing: general Keywords:
Cc: Other processors: tec, dac

Description (last modified by tec)

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/uk/GB14_04-2014_219b_Aberfoyle/

Data arrived from ARSF via network transfer on 14/08/2014

GB14/04 Scientific objective: unknown

GB14/04 Priority: 10

PI: Rachel Gaulton

Notes: FW is present Owl is present

Sensors:

Camera (Delivered on 14/10/2014)
Fenix (Delivered on 14/10/2014)
LiDAR
FW LiDAR
Owl

Change History (48)

comment:1 Changed 4 years ago by tec

RCD

Started converting RAW images to TIFF

Last edited 4 years ago by tec (previous) (diff)

comment:2 Changed 4 years ago by tec

RCD
No images need removing.

Waiting on NAV.

comment:3 Changed 4 years ago by knpa

  • Description modified (diff)

comment:4 Changed 4 years ago by dac

Started lidar processing.

comment:5 Changed 4 years ago by dac

PPP Result:

Station Latitude Longitude Elevation
GLAS 55 51 14.41473 -4 17 47.34487 71.839
KILN 56 27 17.79080 -4 19 14.64440 232.357

comment:6 Changed 4 years ago by tec

RCD
Tagging TIFF images

comment:7 Changed 4 years ago by tec

RCD
Awaiting delivery check.

comment:8 Changed 4 years ago by lah

Fenix
Processed using existing boresight with SCTs:

1 0.95
2 0.97
3 0.98
4 0.98
5 0.98
6 0.98
7 0.98
8 0.98
9 0.96
10 0.98

Looks ok in mosaic if ignore blurry edges. Ready for DC.

comment:9 Changed 4 years ago by tec

Fenix DC
Starting delivery check

comment:10 Changed 4 years ago by tec

Fenix DC

  • In readme move table of flightlines to next page
  • I removed the empty fodis dir
  • fenix_viewvector_list.txt has CRLF nice ;)
  • Can't remember if we can fix blurry edges, ask mark1, if not then screenshots look fine
  • I would of used a pixel size of 1.9 for the fenix not 2. Should be fine though.
  • Level 1 Files (add stuff about noise and underflows to readme)
    • Line 1-5 high noise and underflows at end of line, bands 405-447
    • Line 6-10 high noise and underflows over the whole line, bands 405-447
    • All lines underflows and noise from around band 310-390

comment:11 Changed 4 years ago by dac

Starting RCD Delivery Check

comment:12 Changed 4 years ago by dac

RCD Delivery Check complete. Corrected spelling of my name in readme (templates had already been corrected).

Ready to deliver

comment:13 Changed 4 years ago by lah

Fenix

  • Have made suggested changes to logsheet.
  • mark1 says there's not currently a way of fixing pixel stretches at boundaries.
  • just used default fenix pixel size

comment:14 Changed 4 years ago by tec

Fenix DC
Ready for delivery. Currently zipping mapped files.

comment:15 Changed 4 years ago by tec

Fenix DC
Ready for delivery

comment:16 Changed 4 years ago by tec

Emailed PI about vectors 09/10/2014 18:15:00

comment:17 Changed 4 years ago by tec

Sent off RCD and Hyperspectral to Rachel Gaulton on 14/10/2014

comment:18 Changed 4 years ago by tec

  • Description modified (diff)
  • Other processors set to tec, dac

comment:19 Changed 4 years ago by dac

Default pitch roll and heading values from new boresight produce good results, although few buildings on the overlap of parallel strips.

Roll: -0.006375
Pitch: 0.00016318
Heading: 8e-005

Starting classification.

comment:20 Changed 4 years ago by dac

Classified data, default parameters classified lots of ground and within-canopy returns as noise. Changed to 4 3 which produced better results. There are powerlines going across the strips - haven't classified these as noise.

Full waveform data available for four lines, line 160350 contains full-waveform data for only a small section of the line. Copied classification from discrete data.

Created DSM and checked registration against Fenix data - looked good.

Making delivery.

comment:21 Changed 4 years ago by dac

Created delivery, waiting for delivery check.

comment:22 Changed 4 years ago by dap

Beginning LiDAR Delivery Check

comment:23 Changed 4 years ago by dap

LiDAR Delivery Check

Complete with the following issues found:

  • knpa still in the contacts section of the read me file - have changed the template but you will need to change the source tex file.
  • Only 4 FW files (and 10 discrete files) with no mention of it in the data quality remarks - are some missing?
  • The number of trj files isn't the same as the number of full waveform files.
  • Last page of logsheet blank - I've removed this.
  • Think we deliver a screenshot for each line now - none were included in this delivery.
  • Flightline names table in read me document goes off page
  • bng dem looks significantly smaller than the latlong dem - not sure if this is a problem or not

Also checked full waveform files with laszip, no errors returned and two files produced for each las1.3 file in delivery. Checked waveform files in Wave Viewer, no problems seem to persist.

comment:24 Changed 4 years ago by dac

LiDAR Delivery Check

Have addressed issues found by dap in delivery:

  • Have added comment about only having 4 FW files to Readme (only 4 were available).
  • Have removed trj files for lines without FW data.
  • Had moved screenshots of individual lines back (had moved out of delivery folder as hasn't realised they needed to be included).
  • Have forced a new page before flightline names table so it is all on a single page.
  • Have recreated latlong DEM, size is now similar to BNG DEM.

Ready to deliver.

comment:25 Changed 4 years ago by dap

LiDAR Delivery Check

  • Renamed screenshots so that they have .jpg extension (as opposed to .txt.jpg) with rename.sh -f .txt.jpg .jpg
  • Ran proj_tidy.sh to check naming conventions and for missing files - output was fine.

LiDAR data delivery GB14_04-219b-lidar-20141015 ready for delivery.

comment:26 Changed 4 years ago by dac

LiDAR Delivery
Sent LiDAR data on 16/10/2014 to

Rachel Gaulton
School of Civil Engineering and Geosciences
Room G.16
Cassie Building
Newcastle University
Newcastle upon Tyne
NE1 7RU
UK

comment:27 Changed 3 years ago by lah

OWL

SCTs:

1 1.95
2 1.98
3 1.95
4 1.95
5 1.96
6 1.97
7 1.96
8 0.96
9 1.95
10 0.98

comment:28 Changed 3 years ago by lah

Experimental delivery created with Owl l3 files (no mask files, no quality reports) GB14_04-219b-owl-20150501. Uploaded to ftp and PI notified 05/05/2015.

comment:29 Changed 3 years ago by lah

Fenix re-DC

  • New delivery looks fine
  • Spectra look good except for the odd pixel with a drop in VNIR e.g line 5 sample 490 line 3539
  • Underflows look reasonable. Still can't see overflows, even raw files in fastQC using 8191

Not zipped files until we've checked ground spectra and made some comparisons.

comment:30 Changed 3 years ago by dac

Fenix

Have compared spectra to 6S simulations - look fine. Waiting to hear back from PI on ground spectra before sending.

Have started zipping mapped files on molucca - will use the level1b files to check the spectra.

comment:31 Changed 3 years ago by dac

Fenix

Received field spectra and have copied to /users/rsg/arsf/arsf_data/2014/misc/GB14_04_ground_truth.

Will check against before dispatching data.

comment:32 Changed 3 years ago by dac

Fenix - Spectra check

Checked Fenix against the following spectra:

TargetLonLatLine
T1 (car park) -4.380651 56.177175 10
T2 (pond) -4.390611 56.193553 8
T3 (tarmac) -4.391132 56.193688 8
T4 (quarry) -4.408332 56.198332 7
T5 (water) -4.448856 56.183071 1

Radiance values match well for T4, for other targets radiance values vary but similar shapes. However, still seeing a small discontinuity between two detectors. Will investigate before sending.

comment:33 Changed 3 years ago by dac

Fenix Spectra

Calibration still doesn't seem right for this flight - trying to atmospherically correct data and compare to field spectra (rather than adding atmospheric effects to field spectra).

comment:34 Changed 3 years ago by lah

Fenix
Created a new delivery without the SWIR bands (as these are too unstable and cannot be dark frame corrected).

/data/molucca2/scratch/arsf-temporary/GB14_04-2014_219b_Aberfoyle/processing/delivery/GB14_04-219b-hyperspectral-20150924

Needs thorough checking

comment:35 Changed 3 years ago by dac

Fenix Delivery Check

Starting delivery check.

comment:36 Changed 3 years ago by dac

Fenix Delivery Check

Compared to field data with atmosphere simulated using 6S:

  • Matches well to T1
  • Much lower than T2
  • Much lower than T3
  • Slightly lower than T4
  • Much lower than T5. Distance increases as wavelength reduces

Given potential errors in parametrisation of 6S and uncertainty with ground spectra (geolocation etc.,) not sufficient proof there is a problem with the spectra.

Other points:

  • Might be worth mentioning in the Readme why us and Specim think the VNIR bands are OK
  • check_apl_cmd fails on aplcorr, error: Spatial binning is missing from the level-1 file header. Please add a line in the header file containing: binning = VALUE where VALUE is the correct spatial and spectral binning of the data in the form, e.g., {1,1} If the data is Fenix data then add binning_VNIR = VALUE and binning_SWIR = VALUE
  • XML has old address for ops (probably not a major problem as readme has correct address).

Ready to go once these are addressed.

comment:37 Changed 3 years ago by lah

Fenix
Addressed all problems in delivery. Added binning_SWIR to all level 1 headers has removed printed error in check_apl_cmd. Added some more words to readme, which are probably worth double checking before delivering.

comment:38 Changed 3 years ago by dac

Fenix

New text looks fine but would remove:

To check the validity of the VNIR bands, ground spectra with atmosphere
simulated using 6S have been compared to the fenix data. The results of this comparison indicate
that the detector instability has not noticeably affected the VNIR bands.

As I don't think the comparison with 6S simulations are sufficient to indicate there are not problems with the VNIR bands. Neither are they sufficient to indicate there are problems.

Ready to deliver but add note in delivery email about problems.

comment:39 Changed 3 years ago by lah

Fenix delivery

Have uploaded delivery to ftp (arsf7) and emailed PI 02/10/15 (asking to be contacted about SWIR bands if they are still required).

comment:40 Changed 3 years ago by dac

Fenix

Copied delivered data to visuyan and finalised delivery.

comment:41 Changed 2 years ago by asm

Fenix

Recent in depth analysis included exhaustive inspection of the ground spectra using py6s for fligtlines 4-10.

Target Lon Lat Line
T1 (car park) -4.380651 56.177175 10
T2 (pond) -4.390611 56.193553 8
T3 (tarmac) -4.391132 56.193688 8
T4 (quarry) -4.408332 56.198332 7, 6, 5
T5 (water) -4.448856 56.183071 10

Radiance values match well for T4, for other targets radiance values vary but similar shapes. For T5 there were cloud shadows over flightline 10 and T1 matches fine. T2 and T3 have a shape that matches but the spectra recorded is much lower because they were recoded at the edge of the swath.

comment:42 Changed 2 years ago by dac

Archiving

Started archiving processed data to NEODC. Includes Owl but not Fenix SWIR bands (as unstable). Have left processing directory as might still need files for SWIR tests. Will archive raw data later.

comment:43 Changed 2 years ago by lah

Owl tidying

  • Moved pixstabi results to my scratch area for now
  • Tidied up owl processing directory (removed copies of delivered data, post processed nav, igm etc) - should have been removed before archiving!

comment:44 Changed 2 years ago by dac

Tidied up owl processing directory (removed copies of delivered data, post processed nav, igm etc) - should have been removed before archiving!

Raw data hasn't been archived yet (tests with SWIR haven't been completed yet so I left processing files for now).

comment:45 Changed 16 months ago by dac

Generated atmospherically corrected versions of lines 4 and 5 using ATCOR.

Copied to ~arsf/arsf_data/2014/misc/GB14_04_atmospheric_correction and delivered via FTP.

comment:46 Changed 9 months ago by dac

Tidied up delivery directory:

  • GB14_04-219b-hyperspectral-20150924 is the 'official' delivery. SWIR bands were dropped due to instability
  • GB14_04-219b-fenix-swirbands-reprocessing-20160422 is the investigation and SWIR reprocessing carried out for the PI. Doesn't have hyperspectral in the name so 'official' delivery is picked up by our scripts not this one.

comment:47 Changed 3 months ago by asm

Archiving

All data present at CEDA. Processed data at: http://data.ceda.ac.uk/neodc/arsf/2014/GB14_04/GB14_04-2014_219b_Aberfoyle/

comment:48 Changed 3 months ago by asm

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.