RG13/06, flight day 303/2014, Maliau Basin

[dap]

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_303_Maliau_Basin

Data arrived by network transfer on 17/12/2014.

Scientific objective: Investigating the impact of humans on tropical forests.

Priority: alpha 5

PI: David Coomes

Full waveform LiDAR data are present.

Owl data are present, but proj_tidy suggests that not enough dark frames were collected for header file *0207.hdr (82 were collected).

Sensors:

• Fenix (requested)
• Leica FW LIDAR (?)
• Leica LIDAR (requested)
• RCD (requested)

Duplicate lines

Process only one for Fenix and RCD - check for which has least cloud cover, failing that choose closest to local noon

• 1257-1259 (3 times, repeated day 302a)
• 1260-1275

[tec]

Navigation Processing

MALIAU_4t Basestation Data
Latitude	4 44 08.45856
Longitude	116 58 34.69396
El Height	272.640m

[tec]

Navigation Processing
Finished Nav

Used IPAS Pro. Settings are as follows
C/A Code: 1.10m
Ell Mask 15 degrees
Max dual frequency distance 80km

Log sheet says no RCD, if the images exist and convert then I'm pretty sure some can be salvaged

[dap]

RCD Processing

Started. Converting raw images.

[dac]

Started lidar processing

[dac]

LiDAR Processing
26 strips processed in ALSPP. Height offsets of 1 - 2 m between strips. Due to dense vegetation and lack of buildings applying standard approach to determine roll offsets will be difficult.

[benj]

Added duplicate line info

[dap]

RCD Processing

Delivery and read me have been created. Note that almost all images had to be removed due as they did not contain any useful data due to a failure in the RCD. Also, the photographs that could be included could not be tagged with positional and attitude information as the GPS times recorded in the event file were negative.

RCD delivery at RG13_06-303-camera-20150120 now ready for checking.

[lah]

Fenix Processing

Started. Lines 1-16 do not appear to be covered by nav data. Carrying on with lines 17-41.

[lah]

Fenix Processing
Remaining duplicates: 1257, 1258, 1259, 1254, 1255, 1266. Will remove cloudiest once processed.

[dac]

RCD Processing
Started delivery check

[dac]

RCD Processing

As mentioned, there are only 6 images in the delivery directory and they don't have positional information (therefore, no KML).

Made the following changes:

• Moved Readme file from processing to delivery directory
• Made some slight changes to Readme text

Ready to deliver.

[dac]

Fenix Processing

Checked hyperspectral processing for lah.

• DEM creation words with all lines, as will just skip those with no navigation data and print a warning.
• Creating a config file for APL doesn't work for lines 1 - 16.
• There was an error at the end of the .sol file where the time when backwards (epoch 1240002), have cut off this part of the navigation data and created an updated .sol file (20141029_222120_subset.sol)
• Processed scenes 17 - 41 in APL with standard sct values.
• Checked approximate location against Landsat 8 panchromatic data - look OK. Landsat data saved in processing directory (move after processing).

[dac]

• Reprocessed navigation data in IPAC TC. Updated sol file doesn't have problem with time going backwards.
• Using reprocessed nav file in APL produces no errors and outputs look OK.

[dac]

Fenix Processing

Found SCT values for first three lines:

17	0
18	0.33
19	0.65

[lah]

Fenix Processing
Discarding lines 37 (1258), 36 (1259), 24 (1264), 25 (1265) and 26 (1266) as these are the cloudiest of the duplicates.

[lah]

Fenix Processing
Found SCTs

Flightline	FENIX
17	0.00
18	0.35
19	0.60
20	0.00
21	0.17
22	0.53
23	0.18
27	0.82
28	1.10
29	0.90
30	0.00
31	0.90
32	0.95
33	0.90
34	1.10
35	0.90
38	0.00
39	0.94
40	1.00
41	0.98

[lah]

Fenix Processing
Disjoint in road observed in mosiac between 2 flightlines (18 and 40) that does not appear to be correctable by changing SCTs without causing severe distortion of the road shape. Could be geometric effect as near edge of strip...

[dac]

LiDAR Processing
Found pitch and roll values, consistent for all lines:

Roll	-0.006095
Pitch	-0.00061

Starting classification.

[lah]

Fenix
Delivery created, ready for checking.

[dac]

LiDAR Processing

Classification complete. Lots of lines included rain which has been flagged as noise.
Creating delivery.

[dac]

Fenix DC

Looks good couple of minor things:

• The notes column in the logsheet PDF was going off the page, I've improved this and copied.
• I removed '.bil' from the screenshots filenames
• There were 'aux.xml' files created by GDAL which I have removed (and added to DC procedure).
• There was a sentence in the readme about using gdal_translate 'to export all bands to RGB'. I removed the 'to RGB' as it made no sense.
• I couldn't see the overflows mentioned in the readme highlighted in fastQC, underflows were all correct.
• The metadata gives '​http://www.eufar.net' as the online resource, which probably needs changing to the ARSF website.

Ready to deliver once these issues have been checked.

[lah]

Fenix

• Have re-done the logsheet pdf as some text was still going off the page.
• Have changed eufar links to ​http://arsf.nerc.ac.uk in metadata in project_info & line_info
• Checked fastQC and corresponding bands look to be overflowing by visual inspection, though depends on how data is displayed (stretched). fastQC won't show overflows for fenix VNIR bands (needs resetting for 12 bit data, as default seems to be for 16 bit (SWIR) detector for all bands). However using 12 bit setting shows up far more overflowing bands. Mask is probably most consistent method to use at this point.

Once wavelengths have been checked, should be ready to deliver.

[dac]

Navigation Processing

Reprocessed basestation data (only change to height)

Latitude	4 44 08.45856
Longitude	116 58 34.69421
El Height	272.571 m

[lah]

Fenix processing - duplicates
Lines 1,2,3,8,10,14 should replace lines 21,22,23,28,30 and 34 as they are slightly less cloudy. Line 16 can also be processed with the reprocessed nav data and is not a duplicate (740).

[dac]

LiDAR Processing

Processed additional 15 lidar lines covered by reprocessed navigation data. Same pitch and roll values used for previously processed flights provided good results. Starting to classify and create new delivery.

[lah]

Fenix Processing
Reprocessed using new nav. Original SCTs still look fine, so new list is:

Flightline	FENIX
1	1.08
2	0.00
3	0.90
8	0.29
10	0.50
14	0.71
16	0.74
17	0.00
18	0.35
19	0.60
20	0.00
27	0.82
29	0.90
31	0.90
32	0.95
33	0.90
35	0.90
38	0.00
39	0.94
40	1.00
41	0.98

[lah]

Fenix Delivery Created - 20150219

• Have recreated logsheet with extra line info
• Have changed eufar to ​https://arsf-dan.nerc.ac.uk in metadata in project_info & line_info
• Line 10 looks a bit blurry in screenshot but appears best of SCTs
• Had to run autoqc.py separately as some lines were missed (not passed to get underflows?) Table format is fine though.

[dac]

LiDAR Processing

Created new delivery. Ready for delivery check.

[lah]

Lidar DC - some points to address

• Roll & pitch looks fine.
• Line 15 classification needs revisiting - really odd patch before first big cloud.
• Looks like a small patch of trees in line 35 have been classified as noise, but probably not worth reclassifying given the huge amount of surrounding noise.
• Copied across missing logsheet.
• There are 42 .rj files for 41 lines.
• Removed .txt from screenshot names.
• dem seems a little large.
• There are 2 .sol files in the nav directory.
• laszip batch script doesn't work and laszip is only producing 1 output file. FWs do display in WaveViewer, though there are missing waveforms at the start and end of files. Many lines are also missing discrete points throughout the file. Lines 36 and 39 appear to be exceptions and all viewed records have both discrete points and waveforms.
• demcompare did not output full difference statistics, will re-run tomorrow on other dem.
• Could also mention in readme about potential reduced accuracy of data due to lack of buildings to determine roll & pitch offset.

[lah]

Lidar DC
demcompare returns difference mean -5,67 and unmasked mean 0.13.

[dac]

LiDAR Processing

• Have looked at line 15 classification, the area looks like it is a block of forest so have kept as not-noise.
• Have fixed .trj files.
• Removed additional .sol file.
• The DEM is coverage does extend beyond the hyperspectral lines but is the same size as provided with the hyperspectral data.
• Tried running laszip on a couple of FW files and produced two files as expected.
• FW lidar isn't listed as being requested for the project so missing points shouldn't be a problem.

[dac]

Fenix Delivery Check

• Lines 1 and 2 offset from each other by about 30 m, possibly due to incorrect SCT value.
• Line 10 looks stretched around large bend, likely due to interpolation.

Apart from these everything looks good and ready to deliver.

[lah]

Fenix Processing

Found better SCTs for lines 1 & 2. Will reprocess and add to delivery.

1	0.97
2	-0.20

[lah]

Fenix

• Moved new lines 1 & 2 in to delivery and created new screenshots and readme. Just noticed that old readme had 289's mosaic not 303's!
• Should be ready now pending wavelength check.

[lah]

Fenix

• Have removed surplus Eagle & Hawk pages from logsheet
• Have changed Under/overflow from subsubsection to subsection in readme

[dac]

RCD / LiDAR Delivery

RCD and LiDAR data sent to PI via FTP 03/03/2015

[lah]

Fenix
Some files still have a large number of bad pixels. Adding to bad pixel file and reprocessing.

[lah]

Fenix ready for DC
Will need updated data quality report from recent calibration.

[dac]

Fenix DC

Rechecking updated delivery.

[dac]

Fenix DC

Looks good, ready to deliver.

Proj tidy printed error about wavelengths not matching 2014 cal (which isn't surprising as the 2015 cal was used).

Don't forget to remove old delivery.

[lah]

PI notified of delivery 05/05/2015 (Fenix, Lidar & RCD)

[dac]

Fenix

Replaced wavelengths in headers with ones from updated calibration.

[stgo]

Started Fenix reprocessing

Have changed config file back to full processing run, was previously stopping at masking step.

[stgo]

Fenix reprocessing

Ready for delivery check.

This delivery needs to be thoroughly checked for:

• correct lines included
• correct linenames from logsheet
• data quality comments
• spectra shifts

[asm]

Fenix reprocessing DC

Started
-Removed fodis directory.
-Added logsheet.
-All flightlines has correct name compared with logsheet.
-Flightlines included on the delivery are in accordance with the info on the ticket.
-Please, add some extra data quality remakrs on Readme about clouds on flightline 38 and not included flightlines due to duplicated areas with more cloud coverage.

[asm]

Fenix reprocessing DC

-Checked green vegetation spectra with Py6S. Good match for all
flightlines.
-Tested all tif files created from apl commands.

[lah]

Fenix DC - spectra
Checked spectra of a few lines and spectra mostly look ok. There is a persistent spike in the masked region, as with most other flights. Line 1 shows an odd strip down the centre of the image for very low intensity bands e.g. band 436 that should be looked into more thoroughly and checked that it doesn't appear on more lines. The spectral shift is within 5 nm compared to previous data (2014 169).

[lah]

Fenix DC - spectra

• Line 1 has bright stripe down middle of detector for bands 425 - 443, dark stripe for bands 458 - 489 and bright stripe for bands 490 - 520. Lines 2 and 3 also have this pattern. This should probably be mentioned in readme if we're going to deliver these lines.

Spectra for all lines look reasonable.

[lah]

Fenix
Although previously delivered, mark1 + stgo decided to check whether duplicate lines can be used instead of lines 1-3.

[stgo]

Fenix Processing

I have replaced lines 1, 2 and 3 with lines 21, 22 and 23. Although these have more cloud cover they do not suffer from the spectral problems identified.

I have now created a delivery and this project is ready for a delivery check.

[asm]

Fenix Delivery Check

-Removed old readme file.
-Renamed screenshots.
-All other checks have been made and everything is orrect.

Only needs to check spectra before zipping files.

[dac]

Fenix Spectra Check

Checked spectra. Look fine, radiance is a little low compared to Py6S simulations but not enough to be worried about.

In data quality report "Problems were identified with lines 005a, 006a and 007a were identified during our quality checks and have been replaced with lines 005b, 006b and 007b." probably don't need "were identified" twice.

[stgo]

Fenix delivery

Fixed repeat of were identified, think I must have changed my mind mid sentence!

[asm]

Fenix delivery

Readme changed and mapped files zipped. Delivery is ready to go.

[stgo]

Fenix delivery

Updated data quality and project information with new report

[lah]

Archiving
Checked and transferred 01/12/15.

[lah]

OWL

Processed all lines to level 1b and calculated scts for lines 20 and 22 for use as example imagery in Owl report.

Flightline	FENIX
20	0.99
22	1.01

[lah]

OWL

Created a partial delivery with the 2 flightlines. A couple of things have failed (project info & ascii FOV vectors), but may not be required for sample data.

[lah]

OWL

Generated new delivery with corrected processing chain. Used an owl_genreadme-airborne.cfg to create the readme, but check_apl_cmd still doesn't work, so manually checked commands. Ready for checking.

[dac]

Owl delivery check

• Renamed data quality report from 'owl_data_quality_report2014.pdf' to 'owl_data_quality_report2015.pdf' as it is the 2015 data quality report - there wasn't one in 2014.
• Proj_tidy didn't recognise any of the files in the delivery. Problem with proj tidy as all look OK,

Ready to deliver.

[lah]

delivered sample OWL data to PI by ftp (arsf19) and sent email notification 14/4/16.

[dac]

Archiving

Raw and processed Fenix, RCD and LiDAR data available from NEODC (​http://browse.ceda.ac.uk/browse/neodc/arsf/2014/RG13_06/RG13_06-2014_303_Maliau_Basin). Have uploaded Owl sample.

Leaving open until Owl has been archived.

[lah]

Owl tidying

Owl files were previously all be correctly named, just moved to a subdirectory to avoid processing, so have just moved all raw folders into thermal/owl.

[lah]

Owl Archiving

Uploaded Owl sample files delivery to NEODC again as they seem to have disappeared. Wendy is checking that they are properly ingested this time.

[asm]

JSON files

Could not create the JSON files for the camera delivery, no eventfile present. Created JSON for the rest of deliveries.