Opened 10 years ago
Closed 8 years ago
#552 closed flight processing (fixed)
RG13/06, flight day 297/2014, SAFE Area
Reported by: | dap | Owned by: | |
---|---|---|---|
Priority: | alpha 5 | Milestone: | |
Component: | Processing: general | Keywords: | |
Cc: | Other processors: | tec |
Description (last modified by dap)
Data location: /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_297_Danum_Valley
Data arrived from network transfer on 17/12/2014.
Scientific objective: Investigating the impact of humans on tropical forests.
Priority: alpha 5
PI: David Coomes
No owl data present
Sensors:
- Fenix (requested)
- Leica FW LIDAR (?)
- Leica LIDAR (requested)
- RCD (delivered 22/05/2015)
Duplicate lines
Process only one for Fenix and RCD - check for which has least cloud cover, failing that choose closest to local noon
- 1201-1219 (day 295)
- 1137-1158 (day 293, 1137-1138 repeated 3 times, also day 294)
- 1188-1200 (day 298, 1188-1189 repeated 3 times, also day 294
Change History (50)
comment:1 Changed 10 years ago by tec
- Other processors set to tec
comment:2 Changed 10 years ago by tec
Navigation Processing
Used IPAS pro
Ell Mask 14
C/A 1.1
Max Dual Freq Dist 80
Line 27 first 15 seconds are quality 3 nav :(
Line 20 most of the line is comprised of quality 2 nav data
comment:4 Changed 10 years ago by dap
Note for Fenix Processor
Files 21-26, 28-39 (40 was not mentioned in the logsheet) and 41-44 (lines 1137-1158) were duplicated in flight day 293 and were processed as part of that flight, so there is no need to process these.
Processing of line 27 will need to be done as it could not be processed as part of flight 293.
comment:5 Changed 10 years ago by tec
Nav
Data in ~/arsf_data/2014/flight_data/malaysia/RG13_06-Basestation_data/RINEX DATA SABAH/Rinex/temporary_gps_dataz/297, move out into project folder once processed.
comment:6 Changed 10 years ago by dap
Navigation Processing
Started. Navigation data produced in past processing is not usable.
comment:7 Changed 10 years ago by tec
RCD Processing
Started converting TIFFS
comment:8 Changed 10 years ago by dap
Navigation Processing
Basestation Data:
Latitude: 4 43 30.56106
Longitude: 117 36 19.18916
Ell. height: 480.745m
comment:9 Changed 9 years ago by tec
Antenna height 1.34
comment:10 follow-up: ↓ 11 Changed 9 years ago by tec
Quality 3 nav from 438606 to 438615
comment:11 in reply to: ↑ 10 Changed 9 years ago by dap
Replying to tec:
Quality 3 nav from 438606 to 438615
These times correspond to logsheet flight line 027.
Image Events file has been processed with IPAS CO and copied to processing/rcd.
Navigation processing complete.
comment:12 Changed 9 years ago by tec
LiDAR Processing
Starting
comment:13 Changed 9 years ago by tec
RCD Processing
Removing images 224 225 226 227 235 236 238 239 248 249 250 263 264 265 283 284 285 286 287 299 300 354 355 366 367 368 369 385 386 387 388 389 398 399 400 401 418 419 429 430 431 432 445 446 447 448 449 450 456 457 458 472 473 482 494 496 502 503 504 513 514 515 519 520 521 527 528 529 531 532 533 553 554 556 557 558 559 560
comment:14 Changed 9 years ago by dap
Fenix Processing
Fenix processing started.
comment:15 Changed 9 years ago by tec
RCD Processing
Awaiting DC
comment:16 Changed 9 years ago by tec
LiDAR Processing
Trying alsproc
Line | Roll | Pitch | FW | Classified | Classified FW | Notes |
---|---|---|---|---|---|---|
232143 | -0.0062 | -0.00254 | Y | Y | Y | |
232450 | -0.0062 | -0.00254 | Y | Y | Y | |
232832 | -0.0062 | -0.00254 | Y | Y | Y | |
233207 | -0.0062 | -0.00234 | Y | Y | Y | |
233537 | -0.0060 | -0.00274 | Y | Y | Y | |
233852 | -0.0060 | -0.00234 | Y | Y | Y | |
234205 | -0.0062 | -0.00264 | Y | Y | Y | |
234518 | -0.0062 | -0.00234 | Y | Y | Y | |
234904 | -0.0062 | -0.00254 | Y | Y | Y | |
235226 | -0.0060 | -0.00244 | Y | Y | Y | |
235557 | -0.0062 | -0.00254 | Y | Y | Y | |
235925 | -0.0062 | -0.00234 | Y | Y | Y | |
000243 | -0.0062 | -0.00274 | Y | Y | Y | |
000601 | -0.0062 | -0.00224 | Y | Y | Y | |
000901 | -0.0060 | -0.00264 | Y | Y | Y | |
001231 | -0.0062 | -0.00234 | Y | Y | Y | |
001546 | -0.0062 | -0.00254 | Y | Y | Y | |
001839 | -0.0060 | -0.00234 | Y | Y | Y | |
002142 | -0.0062 | -0.00254 | Y | Y | Y | |
002401 | -0.0062 | -0.00274 | Y | Y | Y | Crossline 232143-234904 |
003219 | -0.0062 | -0.00254 | Y | Y | Y | Start of new site |
003625 | -0.0062 | -0.00254 | Y | Y | Y | |
004041 | -0.0058 | -0.00234 | Y | Y | Y | |
004451 | -0.0064 | -0.00254 | Y | Y | Y | |
004837 | -0.0058 | -0.00234 | Y | Y | Y | |
005247 | -0.0064 | -0.00254 | Y | Y | Y | |
005632 | -0.0058 | -0.00234 | Y | Y | Y | |
010040 | -0.0064 | -0.00254 | Y | Y | Y | |
010451 | -0.0058 | -0.00234 | Y | Y | Y | |
010842 | -0.0064 | -0.00254 | Y | Y | Y | |
011232 | -0.0058 | -0.00224 | Y | Y | Y | |
011635 | -0.0064 | -0.00254 | Y | Y | Y | |
012000 | -0.0058 | -0.00234 | Y | Y | Y | |
012327 | -0.0064 | -0.00254 | Y | Y | Y | |
012659 | -0.0058 | -0.00234 | Y | Y | Y | |
013027 | -0.0064 | -0.00254 | Y | Y | Y | |
013351 | -0.0058 | -0.00234 | Y | Y | Y | |
013706 | -0.0064 | -0.00264 | Y | Y | Y | |
014055 | -0.0058 | -0.00244 | Y | Y | Y | |
014744 | -0.0066 | -0.00274 | Y | Y | Y | |
015034 | -0.0056 | -0.00234 | Y | Y | Y | |
015322 | -0.0067 | -0.00264 | Y | Y | Y | |
015605 | -0.0062 | -0.00254 | Y | Y | Y | Wiggly crossline 003625 - 013027 |
020130 | -0.0062 | -0.00254 | Y | Y | Y | Isolated, No Roll or Pitch |
020716 | -0.0062 | -0.00244 | Y | Y | Y | |
021035 | -0.0062 | -0.00244 | Y | Y | Y |
comment:17 Changed 9 years ago by dap
Fenix Processing
Have checked for duplicated flight lines in 295 (ticket:551). I will process 297 flight lines 1-15 as they were less affected by cloud than the 295 duplicates.
Flight lines 21-26, 28-39 and 41-44 were duplicates of some of the 293 flight lines (as mentioned in comment:4) so these won't be processed.
I'll be processing 297 flight lines 1-15, 27 and 45-47.
comment:18 Changed 9 years ago by dap
RCD Delivery Check
Beginning delivery check.
comment:19 Changed 9 years ago by dap
RCD Delivery Check
Complete, no issues found. Ready for delivery.
comment:20 Changed 9 years ago by dap
rsyncing RCD Delivery to ftp server (arsf1).
rsync of RCD delivery complete and PI has been notified of RCD delivery (22/05/2015)
comment:21 Changed 9 years ago by dap
- Description modified (diff)
comment:22 Changed 9 years ago by dap
- Summary changed from RG13/06, flight day 297/2014, Danum Valley to RG13/06, flight day 297/2014, SAFE Area
comment:23 Changed 9 years ago by dap
- Description modified (diff)
comment:24 Changed 9 years ago by dap
Fenix Processing
Flight line number 12 did not process. Upon inspection of the log files for that flight line, I found this:
No navigation file found to match times in /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_297_Danum_Valley/hyperspectral/fenix/ for FENIX297-14-12 Closest navigation file found /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_297_Danum_Valley/hyperspectral/fenix/FENIX297-14-11.nav. Does not cover last 207.0000 seconds 1) To manually set/force the navigation file nav2_args_extra = -nav <specim_nav_path_and_filename> in your line section 2) To continue without it and skip real time navigation then set use_nav = false in the line section
Have added nav2_args_extra = -nav /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_297_Danum_Valley/hyperspectral/fenix/FENIX297-14-12.nav to the config file under line 12, which seems to have resolved the issue.
comment:25 Changed 9 years ago by dap
Fenix Processing
Found all SCT values:
Flightline | FENIX |
1 | 1.10 |
2 | 0.92 |
3 | 1.10 |
4 | 0.89 |
5 | 0.12 |
6 | 0.73 |
7 | 1.10 |
8 | 0.94 |
9 | 1.10 |
10 | 0.99 |
11 | 1.10 |
12 | 0.87 |
13 | 0.99 |
14 | -0.03 |
15 | 0.03 |
16 | 0.96 |
17 | 1.07 |
18 | 0.98 |
19 | 0.98 |
20 | 1.03 |
27 | 1.07 |
45 | 0.95 |
46 | 0.97 |
47 | 1.07 |
comment:26 Changed 9 years ago by dap
Fenix Processing
Complete. Ready for delivery check.
comment:27 follow-up: ↓ 34 Changed 9 years ago by lah
Fenix DC
- Copied 2015 data quality report into delivery. Probably should be automated...
- removed fodis directory.
- should note in readme that not all lines collected have been processsed.
- Line 1 is mapped in the wrong place (very far off other lines). SCT could be wrong, but this wouldn't fix the across track offset. Need to check what's gone wrong here..
- Many SCTs look off, particularly lines 6, 8, 11, 14, 16, 18 and 47; others may also not be optimal. Have you looked at all lines together? The roads don't align up and there are different road wobbles for each different line; the road wobbles should be the same for each flightline and there are plenty of overlaps to check in this dataset. Line 2 also looks a bit odd going round the corner, so worth checking this SCT as well.
- Line 27 lines up well with day 293 data.
- There are definitely underflows in line 7, bands 422-440, 493-633. Is this a copy/paste error in the readme?
- Line 27 has massive underflows from band 493 onwards that haven't been picked up until band 559. Could some of the numbers have ended up in the wrong place in the table or been missed out somehow?
- check_apl_cmd runs fine (produces greyscale tifs, not colour).
- There are quite a few bad pixels remaining, so must be different for every flight. Can't do anything about this unless we change our method to correct bad pixels per flight.
comment:28 Changed 9 years ago by dap
Fenix Processing
Reprocessing the data in order to find the correct SCT values.
comment:29 Changed 9 years ago by tec
LiDAR Processing
Nearly done, need to do readme.
comment:30 Changed 9 years ago by tec
LiDAR Processing
Ready for DC
comment:31 Changed 9 years ago by dac
LiDAR DC
Starting delivery check.
comment:32 Changed 9 years ago by dac
LiDAR DC
Some noise was missed in the classification in lines 30 and 43. I have fixed for the 1.2 files but ASCII need regenerating and classification needs to be copied to the 1.3 files.
You can see some of the noise in the DEM so this will also need to be regenerated.
Might be a good idea to add to the readme lines were acquired through rain, which has been flagged as noise.
demcompare gave a difference of 18 m compared to ASTER. I suspect this offset is in the ASTER data.
Once these issues have been addressed ready to deliver.
comment:33 Changed 9 years ago by tec
LiDAR
Rsyncing to FTP, marking as ready to deliver.
comment:34 in reply to: ↑ 27 Changed 9 years ago by dap
Fenix Processing
Have recalculated the SCT values and reprocessed the data. Now ready for delivery check (new delivery directory).
Replying to lah:
Fenix DC
- Copied 2015 data quality report into delivery. Probably should be automated...
- removed fodis directory.
- should note in readme that not all lines collected have been processsed.
Added this to the read me in the latest delivery
- Line 1 is mapped in the wrong place (very far off other lines). SCT could be wrong, but this wouldn't fix the across track offset. Need to check what's gone wrong here..
This will need to be investigated before delivery of data. I have reprocessed this flight line with a new SCT value but not sure if this has fixed it.
- Many SCTs look off, particularly lines 6, 8, 11, 14, 16, 18 and 47; others may also not be optimal. Have you looked at all lines together? The roads don't align up and there are different road wobbles for each different line; the road wobbles should be the same for each flightline and there are plenty of overlaps to check in this dataset. Line 2 also looks a bit odd going round the corner, so worth checking this SCT as well.
Have reprocessed data with newly calculated SCTs.
- Line 27 lines up well with day 293 data.
- There are definitely underflows in line 7, bands 422-440, 493-633. Is this a copy/paste error in the readme?
It seems as though this was a copy and paste mistake in the read me. I've corrected this and gone through all the level1b files to check the rest of the table is correct.
- Line 27 has massive underflows from band 493 onwards that haven't been picked up until band 559. Could some of the numbers have ended up in the wrong place in the table or been missed out somehow?
- check_apl_cmd runs fine (produces greyscale tifs, not colour).
- There are quite a few bad pixels remaining, so must be different for every flight. Can't do anything about this unless we change our method to correct bad pixels per flight.
I will correct the SCT values in comment:25 to the new values.
comment:35 Changed 9 years ago by lah
Fenix DC
- copied in 2015 doc to new delivery.
- SCTs look much better.
- line 1 now lines up, so was an SCT error?
- In readme filelist changed second (typo) f297131b to f297271b.
- proj_tidy is looking at the wrong .wls file (2014 instead of 2015)
- proj_tidy says large difference between fps_set and fps_qpf for lines 20 & 40 (~0.3)
- removed *masked.bil* files from level 1b directory
- removed *sct* from nav directory
- renamed .bil_mask to _mask in level1b directory
- check_apl_cmd works ok now after renamed files
- files zipped
Data ready to deliver pending spectral binning investigation resolution
Would be useful to fix proj_tidy error looking at the wrong .wls file
comment:36 Changed 9 years ago by lah
Fenix DC problem
Just checked and unzipped filesize has not been produced by grid script :(
comment:37 Changed 9 years ago by dac
LiDAR delivery
Delivered to David Coomes via FTP (arsf1) 15/06/2015
comment:38 Changed 9 years ago by stgo
Starting Fenix reprocessing
comment:39 Changed 9 years ago by stgo
Fenix reprocessing
Ready for delivery check.
This delivery needs to be thoroughly checked for:
- correct lines included
- correct linenames from logsheet
- data quality comments
- spectra shifts
comment:40 Changed 9 years ago by asm
Fenix reprocessing DC
Started
-Removed fodis directory.
-Added logsheet.
-Need to change data quality report to the latest one.
-All flightlines has correct name compared with logsheet.
-Flightline 45 almost does not have useful data. Could be removed.
-A remark could be added on readme about clouds on flightline 43 and coverages on flightline 14.
-Flightlines on this delivery are in accordance with previous comments on this ticket and duplicated flightlines from flight 293. Flightline 40 is not present and there is not any remark if it is caused by cloud or if it was previously covered on other flights.
comment:41 Changed 9 years ago by asm
Fenix reprocessing DC
-Checked green vegetation spectra with Py6S. Good match for all flightlines.
-Tested all tif files created from apl commands.
comment:42 Changed 9 years ago by lah
Fenix DC - spectra
Checked spectra of a few lines and no major problems. There is a persistent spike in the mask region for this flight, not unlike many other flights. The spectral shift is within 10 nm compared to previous data (2014 169).
comment:43 Changed 9 years ago by stgo
Fenix reprocessing
Have removed flightline 45 due to cloud, fixed the screenshot names and updated readme.
should be good for another delivery check.
comment:44 Changed 9 years ago by asm
Fenix DC:
-Flighline 45 is not included (no useful data), please remove this sentence from readme:
There is significant cloud coverage affecting flight line 45, meaning that the geocorrection of this file may not be as accurate as we would usually be able to achieve.
comment:45 Changed 9 years ago by stgo
fenix dc
I have removed the comment above from the readme
comment:46 Changed 9 years ago by asm
Fenix DC
-Two readme were present, removed the old one and made sure the new one is correct now.
-Replaced old data quality report with the new one.
-All other comments addressed and solved. Creating zipped files now. Would be ready to go once they have been created.
comment:47 Changed 9 years ago by asm
Fenix delivery
Delivery placed on FTP server (arsf1). Notification mail to PI still pending remaining deliveries within the same project.
comment:48 Changed 9 years ago by asm
Fenix delivery
Notification sent to PI on 3rd November 2015. Delivery finalised.
comment:49 Changed 8 years ago by dac
Archiving
Tidied up project - started uploading to NEODC.
comment:50 Changed 8 years ago by asm
- Resolution set to fixed
- Status changed from new to closed
Navigation Processing