Opened 5 years ago

Closed 3 years ago

#552 closed flight processing (fixed)

RG13/06, flight day 297/2014, SAFE Area

Reported by: dap Owned by:
Priority: alpha 5 Milestone:
Component: Processing: general Keywords:
Cc: Other processors: tec

Description (last modified by dap)

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_297_Danum_Valley

Data arrived from network transfer on 17/12/2014.

Scientific objective: Investigating the impact of humans on tropical forests.

Priority: alpha 5

PI: David Coomes

No owl data present

Sensors:

  • Fenix (requested)
  • Leica FW LIDAR (?)
  • Leica LIDAR (requested)
  • RCD (delivered 22/05/2015)

Duplicate lines

Process only one for Fenix and RCD - check for which has least cloud cover, failing that choose closest to local noon

  • 1201-1219 (day 295)
  • 1137-1158 (day 293, 1137-1138 repeated 3 times, also day 294)
  • 1188-1200 (day 298, 1188-1189 repeated 3 times, also day 294

Change History (50)

comment:1 Changed 5 years ago by tec

  • Other processors set to tec

Navigation Processing

SAFE_13th Basestation Data
Latitude4 43 30.56057
Longitude117 36 19.18874
El Height479.796m

comment:2 Changed 5 years ago by tec

Navigation Processing
Used IPAS pro
Ell Mask 14
C/A 1.1
Max Dual Freq Dist 80

Line 27 first 15 seconds are quality 3 nav :(
Line 20 most of the line is comprised of quality 2 nav data

comment:3 Changed 5 years ago by benj

  • Description modified (diff)

Added duplicate line details

comment:4 Changed 4 years ago by dap

Note for Fenix Processor

Files 21-26, 28-39 (40 was not mentioned in the logsheet) and 41-44 (lines 1137-1158) were duplicated in flight day 293 and were processed as part of that flight, so there is no need to process these.

Processing of line 27 will need to be done as it could not be processed as part of flight 293.

Last edited 4 years ago by dap (previous) (diff)

comment:5 Changed 4 years ago by tec

Nav
Data in ~/arsf_data/2014/flight_data/malaysia/RG13_06-Basestation_data/RINEX DATA SABAH/Rinex/temporary_gps_dataz/297, move out into project folder once processed.

comment:6 Changed 4 years ago by dap

Navigation Processing

Started. Navigation data produced in past processing is not usable.

comment:7 Changed 4 years ago by tec

RCD Processing
Started converting TIFFS

comment:8 Changed 4 years ago by dap

Navigation Processing

Basestation Data:
Latitude: 4 43 30.56106
Longitude: 117 36 19.18916
Ell. height: 480.745m

comment:9 Changed 4 years ago by tec

Antenna height 1.34

comment:10 follow-up: Changed 4 years ago by tec

Quality 3 nav from 438606 to 438615

comment:11 in reply to: ↑ 10 Changed 4 years ago by dap

Replying to tec:

Quality 3 nav from 438606 to 438615

These times correspond to logsheet flight line 027.

Image Events file has been processed with IPAS CO and copied to processing/rcd.

Navigation processing complete.

comment:12 Changed 4 years ago by tec

LiDAR Processing
Starting

comment:13 Changed 4 years ago by tec

RCD Processing
Removing images 224 225 226 227 235 236 238 239 248 249 250 263 264 265 283 284 285 286 287 299 300 354 355 366 367 368 369 385 386 387 388 389 398 399 400 401 418 419 429 430 431 432 445 446 447 448 449 450 456 457 458 472 473 482 494 496 502 503 504 513 514 515 519 520 521 527 528 529 531 532 533 553 554 556 557 558 559 560

comment:14 Changed 4 years ago by dap

Fenix Processing

Fenix processing started.

comment:15 Changed 4 years ago by tec

RCD Processing
Awaiting DC

comment:16 Changed 4 years ago by tec

LiDAR Processing
Trying alsproc

Line Roll Pitch FW Classified Classified FW Notes
232143 -0.0062 -0.00254 Y Y Y
232450 -0.0062 -0.00254 Y Y Y
232832 -0.0062 -0.00254 Y Y Y
233207 -0.0062 -0.00234 Y Y Y
233537 -0.0060 -0.00274 Y Y Y
233852 -0.0060 -0.00234 Y Y Y
234205 -0.0062 -0.00264 Y Y Y
234518 -0.0062 -0.00234 Y Y Y
234904 -0.0062 -0.00254 Y Y Y
235226 -0.0060 -0.00244 Y Y Y
235557 -0.0062 -0.00254 Y Y Y
235925 -0.0062 -0.00234 Y Y Y
000243 -0.0062 -0.00274 Y Y Y
000601 -0.0062 -0.00224 Y Y Y
000901 -0.0060 -0.00264 Y Y Y
001231 -0.0062 -0.00234 Y Y Y
001546 -0.0062 -0.00254 Y Y Y
001839 -0.0060 -0.00234 Y Y Y
002142 -0.0062 -0.00254 Y Y Y
002401 -0.0062 -0.00274 Y Y Y Crossline 232143-234904
003219 -0.0062 -0.00254 Y Y Y Start of new site
003625 -0.0062 -0.00254 Y Y Y
004041 -0.0058 -0.00234 Y Y Y
004451 -0.0064 -0.00254 Y Y Y
004837 -0.0058 -0.00234 Y Y Y
005247 -0.0064 -0.00254 Y Y Y
005632 -0.0058 -0.00234 Y Y Y
010040 -0.0064 -0.00254 Y Y Y
010451 -0.0058 -0.00234 Y Y Y
010842 -0.0064 -0.00254 Y Y Y
011232 -0.0058 -0.00224 Y Y Y
011635 -0.0064 -0.00254 Y Y Y
012000 -0.0058 -0.00234 Y Y Y
012327 -0.0064 -0.00254 Y Y Y
012659 -0.0058 -0.00234 Y Y Y
013027 -0.0064 -0.00254 Y Y Y
013351 -0.0058 -0.00234 Y Y Y
013706 -0.0064 -0.00264 Y Y Y
014055 -0.0058 -0.00244 Y Y Y
014744 -0.0066 -0.00274 Y Y Y
015034 -0.0056 -0.00234 Y Y Y
015322 -0.0067 -0.00264 Y Y Y
015605 -0.0062 -0.00254 Y Y Y Wiggly crossline 003625 - 013027
020130 -0.0062 -0.00254 Y Y Y Isolated, No Roll or Pitch
020716 -0.0062 -0.00244 Y Y Y
021035 -0.0062 -0.00244 Y Y Y
Last edited 4 years ago by tec (previous) (diff)

comment:17 Changed 4 years ago by dap

Fenix Processing

Have checked for duplicated flight lines in 295 (ticket:551). I will process 297 flight lines 1-15 as they were less affected by cloud than the 295 duplicates.

Flight lines 21-26, 28-39 and 41-44 were duplicates of some of the 293 flight lines (as mentioned in comment:4) so these won't be processed.

I'll be processing 297 flight lines 1-15, 27 and 45-47.

Last edited 4 years ago by dap (previous) (diff)

comment:18 Changed 4 years ago by dap

RCD Delivery Check

Beginning delivery check.

comment:19 Changed 4 years ago by dap

RCD Delivery Check

Complete, no issues found. Ready for delivery.

comment:20 Changed 4 years ago by dap

rsyncing RCD Delivery to ftp server (arsf1).

rsync of RCD delivery complete and PI has been notified of RCD delivery (22/05/2015)

Last edited 4 years ago by dap (previous) (diff)

comment:21 Changed 4 years ago by dap

  • Description modified (diff)

comment:22 Changed 4 years ago by dap

  • Summary changed from RG13/06, flight day 297/2014, Danum Valley to RG13/06, flight day 297/2014, SAFE Area

comment:23 Changed 4 years ago by dap

  • Description modified (diff)

comment:24 Changed 4 years ago by dap

Fenix Processing

Flight line number 12 did not process. Upon inspection of the log files for that flight line, I found this:

No navigation file found to match times in /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_297_Danum_Valley/hyperspectral/fenix/ for FENIX297-14-12
Closest navigation file found /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_297_Danum_Valley/hyperspectral/fenix/FENIX297-14-11.nav. Does not cover last 207.0000 seconds
1) To manually set/force the navigation file nav2_args_extra = -nav <specim_nav_path_and_filename> in your line section
2) To continue without it and skip real time navigation then set use_nav = false in the line section

Have added nav2_args_extra = -nav /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_297_Danum_Valley/hyperspectral/fenix/FENIX297-14-12.nav to the config file under line 12, which seems to have resolved the issue.

comment:25 Changed 4 years ago by dap

Fenix Processing

Found all SCT values:

Flightline FENIX
1 1.10
2 0.92
3 1.10
4 0.89
5 0.12
6 0.73
7 1.10
8 0.94
9 1.10
10 0.99
11 1.10
12 0.87
13 0.99
14 -0.03
15 0.03
16 0.96
17 1.07
18 0.98
19 0.98
20 1.03
27 1.07
45 0.95
46 0.97
47 1.07
Last edited 4 years ago by dap (previous) (diff)

comment:26 Changed 4 years ago by dap

Fenix Processing

Complete. Ready for delivery check.

comment:27 follow-up: Changed 4 years ago by lah

Fenix DC

  • Copied 2015 data quality report into delivery. Probably should be automated...
  • removed fodis directory.
  • should note in readme that not all lines collected have been processsed.
  • Line 1 is mapped in the wrong place (very far off other lines). SCT could be wrong, but this wouldn't fix the across track offset. Need to check what's gone wrong here..
  • Many SCTs look off, particularly lines 6, 8, 11, 14, 16, 18 and 47; others may also not be optimal. Have you looked at all lines together? The roads don't align up and there are different road wobbles for each different line; the road wobbles should be the same for each flightline and there are plenty of overlaps to check in this dataset. Line 2 also looks a bit odd going round the corner, so worth checking this SCT as well.
  • Line 27 lines up well with day 293 data.
  • There are definitely underflows in line 7, bands 422-440, 493-633. Is this a copy/paste error in the readme?
  • Line 27 has massive underflows from band 493 onwards that haven't been picked up until band 559. Could some of the numbers have ended up in the wrong place in the table or been missed out somehow?
  • check_apl_cmd runs fine (produces greyscale tifs, not colour).
  • There are quite a few bad pixels remaining, so must be different for every flight. Can't do anything about this unless we change our method to correct bad pixels per flight.

comment:28 Changed 4 years ago by dap

Fenix Processing

Reprocessing the data in order to find the correct SCT values.

comment:29 Changed 4 years ago by tec

LiDAR Processing
Nearly done, need to do readme.

comment:30 Changed 4 years ago by tec

LiDAR Processing
Ready for DC

comment:31 Changed 4 years ago by dac

LiDAR DC

Starting delivery check.

comment:32 Changed 4 years ago by dac

LiDAR DC

Some noise was missed in the classification in lines 30 and 43. I have fixed for the 1.2 files but ASCII need regenerating and classification needs to be copied to the 1.3 files.

You can see some of the noise in the DEM so this will also need to be regenerated.

Might be a good idea to add to the readme lines were acquired through rain, which has been flagged as noise.

demcompare gave a difference of 18 m compared to ASTER. I suspect this offset is in the ASTER data.

Once these issues have been addressed ready to deliver.

comment:33 Changed 4 years ago by tec

LiDAR
Rsyncing to FTP, marking as ready to deliver.

comment:34 in reply to: ↑ 27 Changed 4 years ago by dap

Fenix Processing
Have recalculated the SCT values and reprocessed the data. Now ready for delivery check (new delivery directory).

Replying to lah:

Fenix DC

  • Copied 2015 data quality report into delivery. Probably should be automated...
  • removed fodis directory.
  • should note in readme that not all lines collected have been processsed.

Added this to the read me in the latest delivery

  • Line 1 is mapped in the wrong place (very far off other lines). SCT could be wrong, but this wouldn't fix the across track offset. Need to check what's gone wrong here..

This will need to be investigated before delivery of data. I have reprocessed this flight line with a new SCT value but not sure if this has fixed it.

  • Many SCTs look off, particularly lines 6, 8, 11, 14, 16, 18 and 47; others may also not be optimal. Have you looked at all lines together? The roads don't align up and there are different road wobbles for each different line; the road wobbles should be the same for each flightline and there are plenty of overlaps to check in this dataset. Line 2 also looks a bit odd going round the corner, so worth checking this SCT as well.

Have reprocessed data with newly calculated SCTs.

  • Line 27 lines up well with day 293 data.
  • There are definitely underflows in line 7, bands 422-440, 493-633. Is this a copy/paste error in the readme?

It seems as though this was a copy and paste mistake in the read me. I've corrected this and gone through all the level1b files to check the rest of the table is correct.

  • Line 27 has massive underflows from band 493 onwards that haven't been picked up until band 559. Could some of the numbers have ended up in the wrong place in the table or been missed out somehow?
  • check_apl_cmd runs fine (produces greyscale tifs, not colour).
  • There are quite a few bad pixels remaining, so must be different for every flight. Can't do anything about this unless we change our method to correct bad pixels per flight.

I will correct the SCT values in comment:25 to the new values.

Last edited 4 years ago by dap (previous) (diff)

comment:35 Changed 4 years ago by lah

Fenix DC

  • copied in 2015 doc to new delivery.
  • SCTs look much better.
  • line 1 now lines up, so was an SCT error?
  • In readme filelist changed second (typo) f297131b to f297271b.
  • proj_tidy is looking at the wrong .wls file (2014 instead of 2015)
  • proj_tidy says large difference between fps_set and fps_qpf for lines 20 & 40 (~0.3)
  • removed *masked.bil* files from level 1b directory
  • removed *sct* from nav directory
  • renamed .bil_mask to _mask in level1b directory
  • check_apl_cmd works ok now after renamed files
  • files zipped

Data ready to deliver pending spectral binning investigation resolution

Would be useful to fix proj_tidy error looking at the wrong .wls file

comment:36 Changed 4 years ago by lah

Fenix DC problem
Just checked and unzipped filesize has not been produced by grid script :(

comment:37 Changed 4 years ago by dac

LiDAR delivery

Delivered to David Coomes via FTP (arsf1) 15/06/2015

comment:38 Changed 4 years ago by stgo

Starting Fenix reprocessing

comment:39 Changed 4 years ago by stgo

Fenix reprocessing

Ready for delivery check.

This delivery needs to be thoroughly checked for:

  • correct lines included
  • correct linenames from logsheet
  • data quality comments
  • spectra shifts

comment:40 Changed 4 years ago by asm

Fenix reprocessing DC

Started
-Removed fodis directory.
-Added logsheet.
-Need to change data quality report to the latest one.
-All flightlines has correct name compared with logsheet.
-Flightline 45 almost does not have useful data. Could be removed.
-A remark could be added on readme about clouds on flightline 43 and coverages on flightline 14.
-Flightlines on this delivery are in accordance with previous comments on this ticket and duplicated flightlines from flight 293. Flightline 40 is not present and there is not any remark if it is caused by cloud or if it was previously covered on other flights.

Last edited 4 years ago by asm (previous) (diff)

comment:41 Changed 4 years ago by asm

Fenix reprocessing DC

-Checked green vegetation spectra with Py6S. Good match for all flightlines.
-Tested all tif files created from apl commands.

comment:42 Changed 4 years ago by lah

Fenix DC - spectra
Checked spectra of a few lines and no major problems. There is a persistent spike in the mask region for this flight, not unlike many other flights. The spectral shift is within 10 nm compared to previous data (2014 169).

comment:43 Changed 4 years ago by stgo

Fenix reprocessing

Have removed flightline 45 due to cloud, fixed the screenshot names and updated readme.

should be good for another delivery check.

comment:44 Changed 4 years ago by asm

Fenix DC:

-Flighline 45 is not included (no useful data), please remove this sentence from readme:
There is significant cloud coverage affecting flight line 45, meaning that the geocorrection of this file may not be as accurate as we would usually be able to achieve.

comment:45 Changed 4 years ago by stgo

fenix dc

I have removed the comment above from the readme

comment:46 Changed 4 years ago by asm

Fenix DC

-Two readme were present, removed the old one and made sure the new one is correct now.
-Replaced old data quality report with the new one.
-All other comments addressed and solved. Creating zipped files now. Would be ready to go once they have been created.

Last edited 4 years ago by asm (previous) (diff)

comment:47 Changed 4 years ago by asm

Fenix delivery

Delivery placed on FTP server (arsf1). Notification mail to PI still pending remaining deliveries within the same project.

comment:48 Changed 4 years ago by asm

Fenix delivery

Notification sent to PI on 3rd November 2015. Delivery finalised.

comment:49 Changed 3 years ago by dac

Archiving

Tidied up project - started uploading to NEODC.

comment:50 Changed 3 years ago by asm

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.