Opened 4 years ago

Closed 3 years ago

#548 closed flight processing (fixed)

RG13/06, flight day 291/2014, Danum Valley

Reported by: dap Owned by:
Priority: alpha 5 Milestone:
Component: Processing: general Keywords:
Cc: Other processors:

Description (last modified by tec)

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_291_Danum_Valley

Data arrived by network transfer on 17/12/2014.

Scientific objective: Investigating the impact of humans on tropical forests.

Priority: alpha 5

PI: David Coomes

Full Waveform data present
No Owl data present

proj_tidy indicates that there are differences between some of the fps_set and fps_qpf values in the following Fenix header files:

  • FENIX290-14-10.hdr (difference of 11.451)
  • FENIX290-14-11.hdr (difference of 0.332)
  • FENIX290-14-13.hdr (difference of 0.311)
  • FENIX290-14-21.hdr (difference of 0.528)

Sensors:

  • Fenix (requested)
  • Leica FW LIDAR (delivered 27/04/2015)
  • Leica LIDAR (requested, delivered 27/04/2015)
  • RCD (requested, delivered 23/03/2015)

Duplicate lines

Process only one for Fenix and RCD - check for which has least cloud cover, failing that choose closest to local noon

  • 1244-1249
  • 1120
  • 1131-1138 (day 293)

Change History (47)

comment:1 Changed 4 years ago by dap

  • Description modified (diff)

comment:2 Changed 4 years ago by tec

RCD Processing
Finished converting tifs

comment:3 Changed 4 years ago by tec

Navigation Processing
Started navigation processing

SAFE_6th_ Basestation Data
Latitude4 43 30.56081
Longitude117 36 19.18785
El Height479.827m

comment:4 Changed 4 years ago by tec

Navigation Processing
Some lines have a second of two of quality 3 gps data but other than that the nav is sound.

comment:5 Changed 4 years ago by tec

Navigation Processing
Processed with IPAS Pro, settings below:
Elevation Mask: 14.0 degrees
KAR Maximum dual frequency distance: 80.00 km
Code: 1.20m

Nav finished

comment:6 Changed 4 years ago by tec

RCD Processing
Started RCD Processing

comment:7 Changed 4 years ago by benj

  • Description modified (diff)

Added duplicate line details

comment:8 Changed 4 years ago by dac

Navigation Processing

Reprocessed base station data

Latitude4 43 30.56073
Longitude117 36 19.18705
El Height480.162 m

comment:9 Changed 4 years ago by dap

RCD Processing

Moving the following images to to_remove, as they were affected by significant cloud coverage:

  • 128
  • 129
  • 130
  • 167
  • 261
  • 280

comment:10 Changed 4 years ago by dac

Navigation Processing

Reprocessed navigation data in IPAS Pro. Apart from a noisy patch between 519500 - 520500 s quality is good.

comment:11 Changed 4 years ago by dap

RCD Processing Complete

Ready for delivery check

comment:12 Changed 4 years ago by dac

RCD Delivery Check
Starting delivery check

comment:13 Changed 4 years ago by dac

RCD Delivery Check

Ready to deliver. Two minor comments:

  • Needed to copy Readme into delivery directory.
  • A lot of the images are cloudy (as noted in the Readme). I would say 106 and 291 fall into the 'significantly affected' by cloud coverage but they still have some usable pixels so OK to keep.

comment:14 Changed 4 years ago by tec

Hyperspectrial Processing
Have some time so thought I'd do this
Scratch this, now have no time again

Last edited 4 years ago by tec (previous) (diff)

comment:15 Changed 4 years ago by dap

Beginning Fenix Processing. Renamed Fenix raw files so that they are named FENIX291* instead of FENIX290*.

comment:16 Changed 4 years ago by dap

RCD Data Delivery

Rsyncing RCD delivery to ftp server

comment:17 Changed 4 years ago by dap

Fenix Processing

Processed all lines in 'fast mode' to start identifying SCT values. File number 10 is not usable as it is tiny, this line wasn't recorded in the log sheet anyway so will ignore this file.

comment:18 Changed 4 years ago by dap

Fenix Processing

apl was returning the following error when processing line 12:

No navigation file found to match times in /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_291_Danum_Valley/hyperspectral/fenix/ for FENIX291-14-12
Closest navigation file found /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_291_Danum_Valley/hyperspectral/fenix/FENIX291-14-11.nav. Does not cover last 192.0000 seconds

Solved this by adding

nav2_args_extra = -nav /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_291_Danum_Valley/hyperspectral/fenix/FENIX291-14-12.nav

to the flight line's section in the config file.

Last edited 4 years ago by dap (previous) (diff)

comment:19 Changed 4 years ago by dac

LiDAR Processing

Handing project over to tec. Best pitch and roll values found so far for first two lines:

Pitch: -0.00063
Roll: -0.006095

comment:20 Changed 4 years ago by dap

Fenix Processing

Have looked at the raw flight lines (lines 1-7, 1131-1138) duplicated in flight 293 in FastQC in both flights. The ones collected in flight 293 are less affected by cloud so will not process lines 1 - 7 for flight 291.

comment:21 Changed 4 years ago by tec

LiDAR Processing

Line Roll Pitch Done FW Classified
LDR141018_000915_1.LAS -0.006315 -0.00025 Y Y
LDR141018_001308_1.LAS -0.006315 -0.00025 Y Y
LDR141018_001706_1.LAS -0.006185 -0.00025 Y Y
LDR141018_002109_1.LAS -0.006185 -0.00025 Y Y
LDR141018_002455_1.LAS -0.006185 -0.00025 Y Y
LDR141018_002841_1.LAS -0.006185 -0.00025 Y Y
LDR141018_003305_1.LAS -0.006185 -0.00025 Y Y
LDR141018_003659_1.LAS -0.006185 -0.00025 Y Y
LDR141018_004332_1.LAS -0.006185 -0.00025 Y Y
LDR141018_004611_1.LAS -0.006185 -0.00025 Y Y
LDR141018_004920_1.LAS -0.006185 -0.00025 Y Y
LDR141018_005158_1.LAS -0.006185 -0.00025 Y Y
LDR141018_005447_1.LAS -0.006185 -0.00025 Y Y
LDR141018_005735_1.LAS -0.006185 -0.00025 Y Y
LDR141018_010239_1.LAS -0.006185 -0.00025 Y Y
LDR141018_010520_1.LAS -0.006185 -0.00025 Y Y
LDR141018_010809_1.LAS -0.006185 -0.00025 Y Y
LDR141018_011038_1.LAS -0.006185 -0.00025 Y Y
LDR141018_011322_1.LAS -0.006185 -0.00025 Y Y
LDR141018_011630_1.LAS -0.006185 -0.00025 Y Y
LDR141018_012256_1.LAS -0.006185 -0.00025 Y Y
LDR141018_013134_1.LAS -0.006185 -0.00025 Y Y
LDR141018_013726_1.LAS -0.006185 -0.00025 Y Y
LDR141018_014315_1.LAS -0.006185 -0.00025 Y Y
LDR141018_015135_1.LAS -0.006185 -0.00025 Y Y
LDR141018_015428_1.LAS -0.006185 -0.00025 Y Y
LDR141018_015711_1.LAS -0.006185 -0.00025 Y Y
Last edited 4 years ago by tec (previous) (diff)

comment:22 Changed 4 years ago by tec

LiDAR Processing

Finished finding roll/pitch, was quite difficult, as there was no houses or decent structures but apart from the first two lines, the rest of the flight was rather consistent.

Processing FW now.

comment:23 Changed 4 years ago by tec

LiDAR Processing
Finished creating LiDAR files, now proceeding with classification.

comment:24 Changed 4 years ago by dap

Fenix Processing

Found all SCTs

SAFE Site Letter Flightline Fenix SCT
D 1 Use 293 duplicate
2 Use 293 duplicate
3 Use 293 duplicate
4 Use 293 duplicate
5 Use 293 duplicate
6 Use 293 duplicate
7 Use 293 duplicate
8 Use 293 duplicate
M 9 Use line 13
10 No usable data contained by file
11 Use line 14
12 Use line 15
13 0.92
14 0.94
15 0.95
L 16 1.05
17 0.89
18 1.05
19 Use line 16
20 Use line 17
21 Use line 18
U 22 Use line 25
23 0.75
24 1.03
25 0.70
S 26 -0.05
27 0.91
28 0.99

comment:25 Changed 4 years ago by tec

LiDAR Processing
Finished Classification, shall move onto creating a delivery tomorrow

comment:26 Changed 4 years ago by dap

Fenix Processing

Created delivery and read me, Fenix data now ready for delivery check.

comment:27 Changed 4 years ago by tec

LiDAR Processing
Ready for delivery check.

comment:28 Changed 4 years ago by dap

LiDAR Delivery Check

Beginning delivery check.

comment:29 Changed 4 years ago by dap

LiDAR Delivery Check

  • Roll/pitch offsets seem good.
  • Part of flight line 21 seems to have a kink in the end of the line. Dan suggested that this is to be trimmed off. I tried to process the hyperspectral flight line 21 and it didn't process (not a problem since it was a duplicate anyway). You'll also need to mention this in the data quality remarks.
    • Trim was done during delivery check, the new flight line looks fine.
  • Classifications are good.
  • Ran unix2dos on the ascii flight lines.
  • Add cloud, lack of buildings and kink trimming (mentioned above) to data quality remarks.
  • DEMs will need recreating due to the trimmed flight line (don't forget to also trim the resolution in the new DEM header).
  • demcompare.py was not run as the DEM will need re-producing.
  • Screenshots look fine, but I would suggest regenerating flight line 21's screenshot.
  • laszip caused no errors on the full waveform files and they opened fine in Wave Viewer with no errors.

I'll recheck once the above issues have been seen to.

comment:30 Changed 4 years ago by dap

LiDAR Delivery Check

  • Renamed new read me so that it follows proj_tidy regex.
  • New data quality remarks are fine.
  • Trimmed the resolution in the UTM DEM header to 2.0

Looks fine, ready for delivery.

comment:31 Changed 4 years ago by tec

  • Description modified (diff)

Delivery
LiDAR -> FTP server 27/04/2015

comment:32 Changed 4 years ago by dap

Fenix Processing

Final files have been created with new calibration files and they've been moved into the delivery directory. Now ready for delivery check.

comment:33 Changed 4 years ago by lah

Started Fenix DC
Says BALI-SAFE in logsheet, not Danum Valley...

comment:34 Changed 4 years ago by lah

Fenix DC

  • Mapped files only have 3 bands so need reprocessing.
  • Only the readme has Danum_Valley, so that just needs altering.
  • Most other things were ok, but will need rechecking when all bands created.

As an aside, proj_tidy is looking at the wrong calibration file when doing checks.

comment:35 Changed 4 years ago by dap

Fenix Processing

Seems I had either forgotten to set the correct options in the config file or not saved the config file before putting the processing on the grid when I changed the options. I've reprocessed the mapped files and moved them to the delivery directory.

comment:36 Changed 4 years ago by lah

Fenix DC

  • Just needs site location changing in readme then good to go.
  • I have added the 2015 quality report.
  • Under/overflows look ok, but could consider reducing the threshold, particularly for short flight lines. Only a couple of lines have noticeable missed bands; the threshold still works well for the majority of lines.
  • Some bad pixels, remain, but can't get rid of all all the time.

comment:37 Changed 4 years ago by dap

Fenix Processing

Changed readme site location to 'SAFE Area, Malaysia'.

Fenix delivery ready to go.

comment:38 Changed 4 years ago by lah

Fenix
Uploaded delivery to ftp. (arsf1)

comment:39 Changed 4 years ago by lah

PI notified of delivery 05/05/2015 (Fenix, Lidar & RCD)

comment:40 Changed 4 years ago by stgo

Started reprocessing hyperspectral

comment:41 Changed 4 years ago by gej

Created hyperspectral delivery
creating readme

Last edited 4 years ago by gej (previous) (diff)

comment:42 Changed 4 years ago by gej

Ready for Delivery Check

comment:43 Changed 4 years ago by lah

Fenix DC

  • renamed screenshots.
  • under/overflows look ok. Just a couple of bands missed in the shorter flight lines.
  • spectra all good.

Ready to deliver when data quality report is finished

comment:44 Changed 4 years ago by dac

Fenix

  • Updated data quality report
  • Copied to ftp (arsf1) - likely to have more projects ready today so will notify PI with these.

comment:45 Changed 4 years ago by dac

Fenix

Delivered to PI via FTP (arsf1) 21/10/2015

comment:46 Changed 3 years ago by lah

Archiving
Tidied up and transferred to NEODC.

comment:47 Changed 3 years ago by dac

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.