Opened 5 years ago

Closed 3 years ago

Last modified 19 months ago

#556 closed flight processing (fixed)

RG13/06, flight day 303/2014, Maliau Basin

Reported by: dap Owned by:
Priority: alpha 5 Milestone:
Component: Processing: general Keywords:
Cc: Other processors:

Description (last modified by benj)

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_303_Maliau_Basin

Data arrived by network transfer on 17/12/2014.

Scientific objective: Investigating the impact of humans on tropical forests.

Priority: alpha 5

PI: David Coomes

Full waveform LiDAR data are present.

Owl data are present, but proj_tidy suggests that not enough dark frames were collected for header file *0207.hdr (82 were collected).

Sensors:

  • Fenix (requested)
  • Leica FW LIDAR (?)
  • Leica LIDAR (requested)
  • RCD (requested)

Duplicate lines

Process only one for Fenix and RCD - check for which has least cloud cover, failing that choose closest to local noon

  • 1257-1259 (3 times, repeated day 302a)
  • 1260-1275

Change History (66)

comment:1 Changed 5 years ago by tec

Navigation Processing

MALIAU_4t Basestation Data
Latitude4 44 08.45856
Longitude116 58 34.69396
El Height272.640m

comment:2 Changed 5 years ago by tec

Navigation Processing
Finished Nav

Used IPAS Pro. Settings are as follows
C/A Code: 1.10m
Ell Mask 15 degrees
Max dual frequency distance 80km

Log sheet says no RCD, if the images exist and convert then I'm pretty sure some can be salvaged

comment:3 Changed 5 years ago by dap

RCD Processing

Started. Converting raw images.

comment:4 Changed 5 years ago by dac

Started lidar processing

comment:5 Changed 5 years ago by dac

LiDAR Processing
26 strips processed in ALSPP. Height offsets of 1 - 2 m between strips. Due to dense vegetation and lack of buildings applying standard approach to determine roll offsets will be difficult.

comment:6 Changed 5 years ago by benj

  • Description modified (diff)

Added duplicate line info

comment:7 Changed 5 years ago by dap

RCD Processing

Delivery and read me have been created. Note that almost all images had to be removed due as they did not contain any useful data due to a failure in the RCD. Also, the photographs that could be included could not be tagged with positional and attitude information as the GPS times recorded in the event file were negative.

RCD delivery at RG13_06-303-camera-20150120 now ready for checking.

comment:8 Changed 5 years ago by lah

Fenix Processing

Started. Lines 1-16 do not appear to be covered by nav data. Carrying on with lines 17-41.

comment:9 Changed 5 years ago by lah

Fenix Processing
Remaining duplicates: 1257, 1258, 1259, 1264, 1265, 1266. Will remove cloudiest once processed.

Last edited 5 years ago by lah (previous) (diff)

comment:10 Changed 5 years ago by dac

RCD Processing
Started delivery check

comment:11 Changed 5 years ago by dac

RCD Processing

As mentioned, there are only 6 images in the delivery directory and they don't have positional information (therefore, no KML).

Made the following changes:

  • Moved Readme file from processing to delivery directory
  • Made some slight changes to Readme text

Ready to deliver.

comment:12 Changed 5 years ago by dac

Fenix Processing

Checked hyperspectral processing for lah.

  • DEM creation words with all lines, as will just skip those with no navigation data and print a warning.
  • Creating a config file for APL doesn't work for lines 1 - 16.
  • There was an error at the end of the .sol file where the time when backwards (epoch 1240002), have cut off this part of the navigation data and created an updated .sol file (20141029_222120_subset.sol)
  • Processed scenes 17 - 41 in APL with standard sct values.
  • Checked approximate location against Landsat 8 panchromatic data - look OK. Landsat data saved in processing directory (move after processing).

comment:13 Changed 5 years ago by dac

  • Reprocessed navigation data in IPAC TC. Updated sol file doesn't have problem with time going backwards.
  • Using reprocessed nav file in APL produces no errors and outputs look OK.

comment:14 Changed 5 years ago by dac

Fenix Processing

Found SCT values for first three lines:

17 0
18 0.33
19 0.65

comment:15 Changed 5 years ago by lah

Fenix Processing
Discarding lines 37 (1258), 36 (1259), 24 (1264), 25 (1265) and 26 (1266) as these are the cloudiest of the duplicates.

comment:16 Changed 5 years ago by lah

Fenix Processing
Found SCTs

Flightline FENIX
17 0.00
18 0.35
19 0.60
20 0.00
21 0.17
22 0.53
23 0.18
27 0.82
28 1.10
29 0.90
30 0.00
31 0.90
32 0.95
33 0.90
34 1.10
35 0.90
38 0.00
39 0.94
40 1.00
41 0.98

comment:17 Changed 5 years ago by lah

Fenix Processing
Disjoint in road observed in mosiac between 2 flightlines (18 and 40) that does not appear to be correctable by changing SCTs without causing severe distortion of the road shape. Could be geometric effect as near edge of strip...

comment:18 Changed 5 years ago by dac

LiDAR Processing
Found pitch and roll values, consistent for all lines:

Roll -0.006095
Pitch -0.00061

Starting classification.

comment:19 Changed 5 years ago by lah

Fenix
Delivery created, ready for checking.

comment:20 Changed 5 years ago by dac

LiDAR Processing

Classification complete. Lots of lines included rain which has been flagged as noise.
Creating delivery.

comment:21 Changed 5 years ago by dac

Fenix DC

Looks good couple of minor things:

  • The notes column in the logsheet PDF was going off the page, I've improved this and copied.
  • I removed '.bil' from the screenshots filenames
  • There were 'aux.xml' files created by GDAL which I have removed (and added to DC procedure).
  • There was a sentence in the readme about using gdal_translate 'to export all bands to RGB'. I removed the 'to RGB' as it made no sense.
  • I couldn't see the overflows mentioned in the readme highlighted in fastQC, underflows were all correct.
  • The metadata gives 'http://www.eufar.net' as the online resource, which probably needs changing to the ARSF website.

Ready to deliver once these issues have been checked.

comment:22 Changed 5 years ago by lah

Fenix

  • Have re-done the logsheet pdf as some text was still going off the page.
  • Have changed eufar links to http://arsf.nerc.ac.uk in metadata in project_info & line_info
  • Checked fastQC and corresponding bands look to be overflowing by visual inspection, though depends on how data is displayed (stretched). fastQC won't show overflows for fenix VNIR bands (needs resetting for 12 bit data, as default seems to be for 16 bit (SWIR) detector for all bands). However using 12 bit setting shows up far more overflowing bands. Mask is probably most consistent method to use at this point.

Once wavelengths have been checked, should be ready to deliver.

comment:23 Changed 5 years ago by dac

Navigation Processing

Reprocessed basestation data (only change to height)

Latitude4 44 08.45856
Longitude116 58 34.69421
El Height272.571 m

comment:24 Changed 5 years ago by lah

Fenix processing - duplicates
Lines 1,2,3,8,10,14 should replace lines 21,22,23,28,30 and 34 as they are slightly less cloudy. Line 16 can also be processed with the reprocessed nav data and is not a duplicate (740).

Last edited 5 years ago by lah (previous) (diff)

comment:25 Changed 5 years ago by dac

LiDAR Processing

Processed additional 15 lidar lines covered by reprocessed navigation data. Same pitch and roll values used for previously processed flights provided good results. Starting to classify and create new delivery.

comment:26 Changed 5 years ago by lah

Fenix Processing
Reprocessed using new nav. Original SCTs still look fine, so new list is:

Flightline FENIX
1 1.08
2 0.00
3 0.90
8 0.29
10 0.50
14 0.71
16 0.74
17 0.00
18 0.35
19 0.60
20 0.00
27 0.82
29 0.90
31 0.90
32 0.95
33 0.90
35 0.90
38 0.00
39 0.94
40 1.00
41 0.98

comment:27 Changed 5 years ago by lah

Fenix Delivery Created - 20150219

  • Have recreated logsheet with extra line info
  • Have changed eufar to https://arsf-dan.nerc.ac.uk in metadata in project_info & line_info
  • Line 10 looks a bit blurry in screenshot but appears best of SCTs
  • Had to run autoqc.py separately as some lines were missed (not passed to get underflows?) Table format is fine though.

comment:28 Changed 5 years ago by dac

LiDAR Processing

Created new delivery. Ready for delivery check.

comment:29 Changed 5 years ago by lah

Lidar DC - some points to address

  • Roll & pitch looks fine.
  • Line 15 classification needs revisiting - really odd patch before first big cloud.
  • Looks like a small patch of trees in line 35 have been classified as noise, but probably not worth reclassifying given the huge amount of surrounding noise.
  • Copied across missing logsheet.
  • There are 42 .rj files for 41 lines.
  • Removed .txt from screenshot names.
  • dem seems a little large.
  • There are 2 .sol files in the nav directory.
  • laszip batch script doesn't work and laszip is only producing 1 output file. FWs do display in WaveViewer, though there are missing waveforms at the start and end of files. Many lines are also missing discrete points throughout the file. Lines 36 and 39 appear to be exceptions and all viewed records have both discrete points and waveforms.
  • demcompare did not output full difference statistics, will re-run tomorrow on other dem.
  • Could also mention in readme about potential reduced accuracy of data due to lack of buildings to determine roll & pitch offset.
Last edited 5 years ago by lah (previous) (diff)

comment:30 Changed 5 years ago by lah

Lidar DC
demcompare returns difference mean -5,67 and unmasked mean 0.13.

comment:31 Changed 5 years ago by dac

LiDAR Processing

  • Have looked at line 15 classification, the area looks like it is a block of forest so have kept as not-noise.
  • Have fixed .trj files.
  • Removed additional .sol file.
  • The DEM is coverage does extend beyond the hyperspectral lines but is the same size as provided with the hyperspectral data.
  • Tried running laszip on a couple of FW files and produced two files as expected.
  • FW lidar isn't listed as being requested for the project so missing points shouldn't be a problem.

comment:32 Changed 4 years ago by dac

Fenix Delivery Check

  • Lines 1 and 2 offset from each other by about 30 m, possibly due to incorrect SCT value.
  • Line 10 looks stretched around large bend, likely due to interpolation.

Apart from these everything looks good and ready to deliver.

comment:33 Changed 4 years ago by lah

Fenix Processing

Found better SCTs for lines 1 & 2. Will reprocess and add to delivery.

10.97
2-0.20

comment:34 Changed 4 years ago by lah

Fenix

  • Moved new lines 1 & 2 in to delivery and created new screenshots and readme. Just noticed that old readme had 289's mosaic not 303's!
  • Should be ready now pending wavelength check.

comment:35 Changed 4 years ago by lah

Fenix

  • Have removed surplus Eagle & Hawk pages from logsheet
  • Have changed Under/overflow from subsubsection to subsection in readme

comment:36 Changed 4 years ago by dac

RCD / LiDAR Delivery

RCD and LiDAR data sent to PI via FTP 03/03/2015

comment:37 Changed 4 years ago by lah

Fenix
Some files still have a large number of bad pixels. Adding to bad pixel file and reprocessing.

comment:38 Changed 4 years ago by lah

Fenix ready for DC
Will need updated data quality report from recent calibration.

comment:39 Changed 4 years ago by dac

Fenix DC

Rechecking updated delivery.

comment:40 Changed 4 years ago by dac

Fenix DC

Looks good, ready to deliver.

Proj tidy printed error about wavelengths not matching 2014 cal (which isn't surprising as the 2015 cal was used).

Don't forget to remove old delivery.

comment:41 Changed 4 years ago by lah

PI notified of delivery 05/05/2015 (Fenix, Lidar & RCD)

comment:42 Changed 4 years ago by dac

Fenix

Replaced wavelengths in headers with ones from updated calibration.

comment:43 Changed 4 years ago by stgo

Started Fenix reprocessing

Have changed config file back to full processing run, was previously stopping at masking step.

comment:44 Changed 4 years ago by stgo

Fenix reprocessing

Ready for delivery check.

This delivery needs to be thoroughly checked for:

  • correct lines included
  • correct linenames from logsheet
  • data quality comments
  • spectra shifts

comment:45 Changed 4 years ago by asm

Fenix reprocessing DC

Started
-Removed fodis directory.
-Added logsheet.
-All flightlines has correct name compared with logsheet.
-Flightlines included on the delivery are in accordance with the info on the ticket.
-Please, add some extra data quality remakrs on Readme about clouds on flightline 38 and not included flightlines due to duplicated areas with more cloud coverage.

comment:46 Changed 4 years ago by asm

Fenix reprocessing DC

-Checked green vegetation spectra with Py6S. Good match for all
flightlines.
-Tested all tif files created from apl commands.

comment:47 Changed 4 years ago by lah

Fenix DC - spectra
Checked spectra of a few lines and spectra mostly look ok. There is a persistent spike in the masked region, as with most other flights. Line 1 shows an odd strip down the centre of the image for very low intensity bands e.g. band 436 that should be looked into more thoroughly and checked that it doesn't appear on more lines. The spectral shift is within 5 nm compared to previous data (2014 169).

comment:48 Changed 4 years ago by lah

Fenix DC - spectra

  • Line 1 has bright stripe down middle of detector for bands 425 - 443, dark stripe for bands 458 - 489 and bright stripe for bands 490 - 520. Lines 2 and 3 also have this pattern. This should probably be mentioned in readme if we're going to deliver these lines.

Spectra for all lines look reasonable.

comment:49 Changed 4 years ago by lah

Fenix
Although previously delivered, mark1 + stgo decided to check whether duplicate lines can be used instead of lines 1-3.

comment:50 Changed 4 years ago by stgo

Fenix Processing

I have replaced lines 1, 2 and 3 with lines 21, 22 and 23. Although these have more cloud cover they do not suffer from the spectral problems identified.

I have now created a delivery and this project is ready for a delivery check.

comment:51 Changed 4 years ago by asm

Fenix Delivery Check

-Removed old readme file.
-Renamed screenshots.
-All other checks have been made and everything is orrect.

Only needs to check spectra before zipping files.

comment:52 Changed 4 years ago by dac

Fenix Spectra Check

Checked spectra. Look fine, radiance is a little low compared to Py6S simulations but not enough to be worried about.

In data quality report "Problems were identified with lines 005a, 006a and 007a were identified during our quality checks and have been replaced with lines 005b, 006b and 007b." probably don't need "were identified" twice.

comment:53 Changed 4 years ago by stgo

Fenix delivery

Fixed repeat of were identified, think I must have changed my mind mid sentence!

comment:54 Changed 4 years ago by asm

Fenix delivery

Readme changed and mapped files zipped. Delivery is ready to go.

comment:55 Changed 4 years ago by stgo

Fenix delivery

Updated data quality and project information with new report

comment:56 Changed 4 years ago by lah

Archiving
Checked and transferred 01/12/15.

comment:57 Changed 4 years ago by lah

OWL

Processed all lines to level 1b and calculated scts for lines 20 and 22 for use as example imagery in Owl report.

Flightline FENIX
20 0.99
22 1.01

comment:58 Changed 3 years ago by lah

OWL

Created a partial delivery with the 2 flightlines. A couple of things have failed (project info & ascii FOV vectors), but may not be required for sample data.

comment:59 Changed 3 years ago by lah

OWL

Generated new delivery with corrected processing chain. Used an owl_genreadme-airborne.cfg to create the readme, but check_apl_cmd still doesn't work, so manually checked commands. Ready for checking.

comment:60 Changed 3 years ago by dac

Owl delivery check

  • Renamed data quality report from 'owl_data_quality_report2014.pdf' to 'owl_data_quality_report2015.pdf' as it is the 2015 data quality report - there wasn't one in 2014.
  • Proj_tidy didn't recognise any of the files in the delivery. Problem with proj tidy as all look OK,

Ready to deliver.

comment:61 Changed 3 years ago by lah

delivered sample OWL data to PI by ftp (arsf19) and sent email notification 14/4/16.

comment:62 Changed 3 years ago by dac

Archiving

Raw and processed Fenix, RCD and LiDAR data available from NEODC (http://browse.ceda.ac.uk/browse/neodc/arsf/2014/RG13_06/RG13_06-2014_303_Maliau_Basin). Have uploaded Owl sample.

Leaving open until Owl has been archived.

comment:63 Changed 3 years ago by lah

Owl tidying

Owl files were previously all be correctly named, just moved to a subdirectory to avoid processing, so have just moved all raw folders into thermal/owl.

comment:64 Changed 3 years ago by lah

Owl Archiving

Uploaded Owl sample files delivery to NEODC again as they seem to have disappeared. Wendy is checking that they are properly ingested this time.

comment:65 Changed 3 years ago by asm

  • Resolution set to fixed
  • Status changed from new to closed

comment:66 Changed 19 months ago by asm

JSON files

Could not create the JSON files for the camera delivery, no eventfile present. Created JSON for the rest of deliveries.

Note: See TracTickets for help on using tickets.