RG13/06, flight day 297/2014, SAFE Area

[dap]

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_297_Danum_Valley

Data arrived from network transfer on 17/12/2014.

Scientific objective: Investigating the impact of humans on tropical forests.

Priority: alpha 5

PI: David Coomes

No owl data present

Sensors:

• Fenix (requested)
• Leica FW LIDAR (?)
• Leica LIDAR (requested)
• RCD (delivered 22/05/2015)

Duplicate lines

Process only one for Fenix and RCD - check for which has least cloud cover, failing that choose closest to local noon

• 1201-1219 (day 295)
• 1137-1158 (day 293, 1137-1138 repeated 3 times, also day 294)
• 1188-1200 (day 298, 1188-1189 repeated 3 times, also day 294

[tec]

Navigation Processing

SAFE_13th Basestation Data
Latitude	4 43 30.56057
Longitude	117 36 19.18874
El Height	479.796m

[tec]

Navigation Processing
Used IPAS pro
Ell Mask 14
C/A 1.1
Max Dual Freq Dist 80

Line 27 first 15 seconds are quality 3 nav :(
Line 20 most of the line is comprised of quality 2 nav data

[benj]

Added duplicate line details

[dap]

Note for Fenix Processor

Files 21-26, 28-39 (40 was not mentioned in the logsheet) and 41-44 (lines 1137-1158) were duplicated in flight day 293 and were processed as part of that flight, so there is no need to process these.

Processing of line 27 will need to be done as it could not be processed as part of flight 293.

[tec]

Nav
Data in ~/arsf_data/2014/flight_data/malaysia/RG13_06-Basestation_data/RINEX DATA SABAH/Rinex/temporary_gps_dataz/297, move out into project folder once processed.

[dap]

Navigation Processing

Started. Navigation data produced in past processing is not usable.

[tec]

RCD Processing
Started converting TIFFS

[dap]

Navigation Processing

Basestation Data:
Latitude: 4 43 30.56106
Longitude: 117 36 19.18916
Ell. height: 480.745m

[tec]

Antenna height 1.34

[tec]

Quality 3 nav from 438606 to 438615

[dap]

Replying to tec:

Quality 3 nav from 438606 to 438615

These times correspond to logsheet flight line 027.

Image Events file has been processed with IPAS CO and copied to processing/rcd.

Navigation processing complete.

[tec]

LiDAR Processing
Starting

[tec]

RCD Processing
Removing images 224 225 226 227 235 236 238 239 248 249 250 263 264 265 283 284 285 286 287 299 300 354 355 366 367 368 369 385 386 387 388 389 398 399 400 401 418 419 429 430 431 432 445 446 447 448 449 450 456 457 458 472 473 482 494 496 502 503 504 513 514 515 519 520 521 527 528 529 531 532 533 553 554 556 557 558 559 560

[dap]

Fenix Processing

Fenix processing started.

[tec]

RCD Processing
Awaiting DC

[tec]

LiDAR Processing
Trying alsproc

Line	Roll	Pitch	FW	Classified	Classified FW	Notes
232143	-0.0062	-0.00254	Y	Y	Y
232450	-0.0062	-0.00254	Y	Y	Y
232832	-0.0062	-0.00254	Y	Y	Y
233207	-0.0062	-0.00234	Y	Y	Y
233537	-0.0060	-0.00274	Y	Y	Y
233852	-0.0060	-0.00234	Y	Y	Y
234205	-0.0062	-0.00264	Y	Y	Y
234518	-0.0062	-0.00234	Y	Y	Y
234904	-0.0062	-0.00254	Y	Y	Y
235226	-0.0060	-0.00244	Y	Y	Y
235557	-0.0062	-0.00254	Y	Y	Y
235925	-0.0062	-0.00234	Y	Y	Y
000243	-0.0062	-0.00274	Y	Y	Y
000601	-0.0062	-0.00224	Y	Y	Y
000901	-0.0060	-0.00264	Y	Y	Y
001231	-0.0062	-0.00234	Y	Y	Y
001546	-0.0062	-0.00254	Y	Y	Y
001839	-0.0060	-0.00234	Y	Y	Y
002142	-0.0062	-0.00254	Y	Y	Y
002401	-0.0062	-0.00274	Y	Y	Y	Crossline 232143-234904
003219	-0.0062	-0.00254	Y	Y	Y	Start of new site
003625	-0.0062	-0.00254	Y	Y	Y
004041	-0.0058	-0.00234	Y	Y	Y
004451	-0.0064	-0.00254	Y	Y	Y
004837	-0.0058	-0.00234	Y	Y	Y
005247	-0.0064	-0.00254	Y	Y	Y
005632	-0.0058	-0.00234	Y	Y	Y
010040	-0.0064	-0.00254	Y	Y	Y
010451	-0.0058	-0.00234	Y	Y	Y
010842	-0.0064	-0.00254	Y	Y	Y
011232	-0.0058	-0.00224	Y	Y	Y
011635	-0.0064	-0.00254	Y	Y	Y
012000	-0.0058	-0.00234	Y	Y	Y
012327	-0.0064	-0.00254	Y	Y	Y
012659	-0.0058	-0.00234	Y	Y	Y
013027	-0.0064	-0.00254	Y	Y	Y
013351	-0.0058	-0.00234	Y	Y	Y
013706	-0.0064	-0.00264	Y	Y	Y
014055	-0.0058	-0.00244	Y	Y	Y
014744	-0.0066	-0.00274	Y	Y	Y
015034	-0.0056	-0.00234	Y	Y	Y
015322	-0.0067	-0.00264	Y	Y	Y
015605	-0.0062	-0.00254	Y	Y	Y	Wiggly crossline 003625 - 013027
020130	-0.0062	-0.00254	Y	Y	Y	Isolated, No Roll or Pitch
020716	-0.0062	-0.00244	Y	Y	Y
021035	-0.0062	-0.00244	Y	Y	Y

[dap]

Fenix Processing

Have checked for duplicated flight lines in 295 (ticket:551). I will process 297 flight lines 1-15 as they were less affected by cloud than the 295 duplicates.

Flight lines 21-26, 28-39 and 41-44 were duplicates of some of the 293 flight lines (as mentioned in comment:4) so these won't be processed.

I'll be processing 297 flight lines 1-15, 27 and 45-47.

[dap]

RCD Delivery Check

Beginning delivery check.

[dap]

RCD Delivery Check

Complete, no issues found. Ready for delivery.

[dap]

rsyncing RCD Delivery to ftp server (arsf1).

rsync complete and PI has been notified of delivery (22/05/2015)

[dap]

Fenix Processing

Flight line number 12 did not process. Upon inspection of the log files for that flight line, I found this:

No navigation file found to match times in /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_297_Danum_Valley/hyperspectral/fenix/ for FENIX297-14-12
Closest navigation file found /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_297_Danum_Valley/hyperspectral/fenix/FENIX297-14-11.nav. Does not cover last 207.0000 seconds
1) To manually set/force the navigation file nav2_args_extra = -nav <specim_nav_path_and_filename> in your line section
2) To continue without it and skip real time navigation then set use_nav = false in the line section

Have added nav2_args_extra = -nav /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_297_Danum_Valley/hyperspectral/fenix/FENIX297-14-12.nav to the config file under line 12, which seems to have resolved the issue.

[dap]

Fenix Processing

Found all SCT values:

Flightline	FENIX
1	1.10
2	0.92
3	1.10
4	0.89
5	0.12
6	0.73
7	1.10
8	0.94
9	1.10
10	0.99
11	1.10
12	0.87
13	0.99
14	-0.03
15	0.03
16	0.96
17	1.07
18	0.98
19	0.98
20	1.03
27	1.07
45	0.95
46	0.97
47	1.07

[dap]

Fenix Processing

Complete. Ready for delivery check.

[lah]

Fenix DC

• Copied 2015 data quality report into delivery. Probably should be automated...
• removed fodis directory.
• should note in readme that not all lines collected have been processsed.
• Line 1 is mapped in the wrong place (very far off other lines). SCT could be wrong, but this wouldn't fix the across track offset. Need to check what's gone wrong here..
• Many SCTs look off, particularly lines 6, 8, 11, 14, 16, 18 and 47; others may also not be optimal. Have you looked at all lines together? The roads don't align up and there are different road wobbles for each different line; the road wobbles should be the same for each flightline and there are plenty of overlaps to check in this dataset. Line 2 also looks a bit odd going round the corner, so worth checking this SCT as well.
• Line 27 lines up well with day 293 data.
• There are definitely underflows in line 7, bands 422-440, 493-633. Is this a copy/paste error in the readme?
• Line 27 has massive underflows from band 493 onwards that haven't been picked up until band 559. Could some of the numbers have ended up in the wrong place in the table or been missed out somehow?
• check_apl_cmd runs fine (produces greyscale tifs, not colour).
• There are quite a few bad pixels remaining, so must be different for every flight. Can't do anything about this unless we change our method to correct bad pixels per flight.

[dap]

Fenix Processing

Reprocessing the data in order to find the correct SCT values.

[tec]

LiDAR Processing
Nearly done, need to do readme.

[tec]

LiDAR Processing
Ready for DC

[dac]

LiDAR DC

Starting delivery check.

[dac]

LiDAR DC

Some noise was missed in the classification in lines 30 and 43. I have fixed for the 1.2 files but ASCII need regenerating and classification needs to be copied to the 1.3 files.

You can see some of the noise in the DEM so this will also need to be regenerated.

Might be a good idea to add to the readme lines were acquired through rain, which has been flagged as noise.

demcompare gave a difference of 18 m compared to ASTER. I suspect this offset is in the ASTER data.

Once these issues have been addressed ready to deliver.

[tec]

LiDAR
Rsyncing to FTP, marking as ready to deliver.

[dap]

Fenix Processing
Have recalculated the SCT values and reprocessed the data. Now ready for delivery check (new delivery directory).

Replying to lah:

Fenix DC

• Copied 2015 data quality report into delivery. Probably should be automated...
• removed fodis directory.
• should note in readme that not all lines collected have been processsed.

Added this to the read me in the latest delivery

• Line 1 is mapped in the wrong place (very far off other lines). SCT could be wrong, but this wouldn't fix the across track offset. Need to check what's gone wrong here..

This will need to be investigated before delivery of data. I have reprocessed this flight line with a new SCT value but not sure if this has fixed it.

• Many SCTs look off, particularly lines 6, 8, 11, 14, 16, 18 and 47; others may also not be optimal. Have you looked at all lines together? The roads don't align up and there are different road wobbles for each different line; the road wobbles should be the same for each flightline and there are plenty of overlaps to check in this dataset. Line 2 also looks a bit odd going round the corner, so worth checking this SCT as well.

Have reprocessed data with newly calculated SCTs.

• Line 27 lines up well with day 293 data.
• There are definitely underflows in line 7, bands 422-440, 493-633. Is this a copy/paste error in the readme?

It seems as though this was a copy and paste mistake in the read me. I've corrected this and gone through all the level1b files to check the rest of the table is correct.

• Line 27 has massive underflows from band 493 onwards that haven't been picked up until band 559. Could some of the numbers have ended up in the wrong place in the table or been missed out somehow?
• check_apl_cmd runs fine (produces greyscale tifs, not colour).
• There are quite a few bad pixels remaining, so must be different for every flight. Can't do anything about this unless we change our method to correct bad pixels per flight.

I will correct the SCT values in comment:25 to the new values.

[lah]

Fenix DC

• copied in 2015 doc to new delivery.
• SCTs look much better.
• line 1 now lines up, so was an SCT error?
• In readme filelist changed second (typo) f297131b to f297271b.
• proj_tidy is looking at the wrong .wls file (2014 instead of 2015)
• proj_tidy says large difference between fps_set and fps_qpf for lines 20 & 40 (~0.3)
• removed *masked.bil* files from level 1b directory
• removed *sct* from nav directory
• renamed .bil_mask to _mask in level1b directory
• check_apl_cmd works ok now after renamed files
• files zipped

Data ready to deliver pending spectral binning investigation resolution

Would be useful to fix proj_tidy error looking at the wrong .wls file

[lah]

Fenix DC problem
Just checked and unzipped filesize has not been produced by grid script :(

[dac]

LiDAR delivery

Delivered to David Coomes via FTP (arsf1) 15/06/2015

[stgo]

Starting Fenix reprocessing

[stgo]

Fenix reprocessing

Ready for delivery check.

This delivery needs to be thoroughly checked for:

• correct lines included
• correct linenames from logsheet
• data quality comments
• spectra shifts

[asm]

Fenix reprocessing DC

Started
-Removed fodis directory.
-Added logsheet.
-Need to change data quality report to the latest one.
-All flightlines has correct name compared with logsheet.
-Flightline 45 almost does not have useful data. Could be removed.
-A remark could be added on readme about clouds on flightline 43 and coverages on flightline 14.
-Flightlines on this delivery are in accordance with previous comments on this ticket and duplicated flightlines from flight 293. Flightline 40 is not present and there is not any remark if it is caused by cloud or if it was previously covered on other flights.

[asm]

Fenix reprocessing DC

-Checked green vegetation spectra with Py6S. Good match for all flightlines.
-Tested all tif files created from apl commands.

[lah]

Fenix DC - spectra
Checked spectra of a few lines and no major problems. There is a persistent spike in the mask region for this flight, not unlike many other flights. The spectral shift is within 10 nm compared to previous data (2014 169).

[stgo]

Fenix reprocessing

Have removed flightline 45 due to cloud, fixed the screenshot names and updated readme.

should be good for another delivery check.

[asm]

Fenix DC:

-Flighline 45 is not included (no useful data), please remove this sentence from readme:
There is significant cloud coverage affecting flight line 45, meaning that the geocorrection of this file may not be as accurate as we would usually be able to achieve.

[stgo]

fenix dc

I have removed the comment above from the readme

[asm]

Fenix DC

-Two readme were present, removed the old one and made sure the new one is correct now.
-Replaced old data quality report with the new one.
-All other comments addressed and solved. Creating zipped files now. Would be ready to go once they have been created.

[asm]

Fenix delivery

Delivery placed on FTP server (arsf1). Notification mail to PI still pending remaining deliveries within the same project.

[asm]

Fenix delivery

Notification sent to PI on 3rd November 2015. Delivery finalised.

[dac]

Archiving

Tidied up project - started uploading to NEODC.