Opened 5 years ago

Closed 3 years ago

#555 closed flight processing (fixed)

RG13/06, flight day 288/2014, Danum Valley

Reported by: dap Owned by:
Priority: alpha 5 Milestone:
Component: Processing: general Keywords:
Cc: Other processors: tec

Description

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_288_Danum_Valley

Data arrived by network transfer on 17/12/2014.

Scientific objective: Investigating the impact of humans on tropical forests.

Priority: alpha 5

PI: David Coomes

Although Owl data are present, proj_tidy suggests that we do not have dark frame information for the following header files:

  • David_Coomes_Safe_0124.hdr
  • David_Coomes_Safe_0125.hdr
  • David_Coomes_Safe_0126.hdr
  • David_Coomes_Safe_0127.hdr
  • David_Coomes_Safe_0128.hdr

Sensors:

  • Fenix (requested)
  • Leica FW LIDAR (?)
  • Leica LIDAR (requested)
  • RCD (requested)

Change History (61)

comment:1 Changed 5 years ago by tec

Navigation Processing
Starting basestation processing.

comment:2 Changed 5 years ago by tec

Navigation Processing

SAFE_5th_ Basestation Data
Latitude4 43 30.56082
Longitude117 36 19.18784
El Height479.784m
Last edited 5 years ago by tec (previous) (diff)

comment:3 Changed 5 years ago by tec

Navigation Processing
Basestation nav does not cover all of the IMU data, but it covers the flightlines so its fine. Processing with Ipas Pro as Ipas TC crashes.

Settings used which are different to the defaults are as follows:
Elevation Mask: 14.0 degrees
KAR Maximum dual frequency distance: 80.00 km
Code: 1.50m

There is a gap in the IMU data for line 166 from 01:31:49.00 -> 01:32:00.00
At the start of line 168 the gps data is of quality 2 for from 01:34:03.00 -> 01:34:11.00

Processing finished.

comment:4 Changed 5 years ago by tec

RCD Processing
Started converting tifs, created image event csv

comment:5 Changed 5 years ago by tec

Fenix Processing
Started fenix processing

comment:6 Changed 5 years ago by tec

RCD Processing
Ready for delivery check

comment:7 Changed 5 years ago by dap

Beginning RCD delivery check

comment:8 Changed 5 years ago by dap

RCD Delivery Check

Read me

  • 'a' in LaTeX template should be `a'. I've changed the read me template so this doesn't happen again. Waiting for the change to be reviewed merged in to the master branch.
  • Images 1 - 60 should probably not be included in the delivery as they are out of bounds. Don't forget to change the data quality remarks section of the read me, where you mention that the photographs are not in the area.

Images

  • 00001 is over-exposed and should be removed.
  • Would suggest removing images 522 - 524 as they have excessive cloud coverage.

I'll recheck the delivery when the above issues have been addressed.

comment:9 Changed 5 years ago by tec

RCD
Made changes
Ready for DC

comment:10 Changed 5 years ago by dap

Rechecking RCD Delivery

comment:11 Changed 5 years ago by dap

RCD Delivery Checking complete - delivery at
processing/delivery/RG13_06-288-camera-20150107 ready for delivery.

comment:12 Changed 5 years ago by dap

Beginning LiDAR Processing

comment:13 Changed 5 years ago by tec

  • Other processors set to tec

Fenix Processing
Started finding SCT values

comment:14 Changed 4 years ago by dap

Began LiDAR processing on 15/01/2015

comment:15 Changed 4 years ago by dap

Since the delivery check of the RCD delivery for this flight, I have noticed that some of the thumbnails are missing from the kml files in flightline no. 12. Have therefore marked RCD as "in progress".

comment:16 follow-up: Changed 4 years ago by tec

Fenix Processing
Sent to the grid to process all bands. Line 13 seems to be broke, will look into that

Flightline FENIX
1 0.10
2 0.90
3 0.90
4 0.90
5 0.90
6 1.06
7 0.79
8 1.13
9 0.67
10 1.30
11 0.53
12 1.41
13 ?

comment:17 Changed 4 years ago by dap

Fenix Processing

Taken over from tec. Investigated problem with line 13. Problem was that apl could not find the raw .nav file associated with this line. Have added nav2_args_extra = -nav <path/to/FENIX288-14-13.nav to DEFAULT section of the hyperspectral config and started the processing of line 13.

comment:18 in reply to: ↑ 16 Changed 4 years ago by dap

Replying to tec:

Fenix Processing
Sent to the grid to process all bands. Line 13 seems to be broke, will look into that

Flightline FENIX
1 0.10
2 0.90
3 0.90
4 0.90
5 0.90
6 1.06
7 0.79
8 1.13
9 0.67
10 1.30
11 0.53
12 1.41
13 0.90

Added SCT value for flight line 13 to table

comment:19 Changed 4 years ago by dap

Fenix Processing

Line information xml files were not generated or have been deleted - reprocessing the data to regenerate them.

comment:20 Changed 4 years ago by dap

Fenix Processing

Delivery has been created and read me created. Fenix data now ready for check.

comment:21 Changed 4 years ago by dap

RCD

Found problems with RCD delivery created. Processing has been handed to me. Re-processing the images to create a new delivery.

comment:22 Changed 4 years ago by dap

RCD Processing

Ran rcdimages_in_pdf.py and found that 60 of the thumbnails (the ones beginning with 140143221) were not displayed in the pdf. Have searched the rcd log for the names of the missing files and they are not in there. This suggests that the images that weren't displayed in the pdf file could be from a different flight.

Have emailed Tom Millard to find out what went wrong.

Last edited 4 years ago by dap (previous) (diff)

comment:23 Changed 4 years ago by lah

Fenix DC

  • Removed fodis folder.
  • Converted txt file.
  • SCTs mostly look ok, but slightly off between 1-2, 7-8, 10-11 and way off for 11-12 (roads twist in different directions on the different lines). Worth re-checking, though could be slope viewing effects in some places.
  • Readme has “0cm” at top of first page. Also not much point of having a table start with only 1 line (pg 2). Many paragraphs start indented, whilst other do not – definitely needs fixing. Line missing between files 2 & 3 in under/overflow table.
  • Processing command produces tiffs ok, though does state error 6: “SetColorTable() can only be called on band 1” and “SetColorTable() not supported for multi-sample TIFF files” a lot. Lots of code still complains that it doesn't recognise 'WGS84' because it's in quotes.
    • Under/overflows seem reasonable, though some may have been missed, possibly due to the threshold being set too low.

comment:24 Changed 4 years ago by dac

Navigation Processing

Reprocessed base station data.

Latitude4 43 30.56087
Longitude117 36 19.18807
El Height480.154 m

comment:25 Changed 4 years ago by dac

Navigation Processing
Reprocessed navigation data in IPAS TC

comment:26 Changed 4 years ago by dap

LiDAR Processing

All flight lines I had found pitch offsets for with the old sol file have now been reprocessed with new sol file. The pitch offsets used for the data processed with the old sol file seem to provide the same result with the new one in terms of pitch and roll. Pitch error offsets found up to now:

Flightline Pitch Error Roll Error
004022 0.000163180 -0.006375
005105 -0.000921318 -0.006175
010204 0.000090032 -0.006375
011317 -0.000863180 -0.006175
012339 0.000443180 -0.006375
013422 -0.001243180 -0.005975
014456 0.000203180 -0.006375
015527 -0.001203180 -0.005875
020544 0.000163180 -0.006375
021635 -0.001203180 -0.005875
022705 0.000263180 -0.006375
023739 -0.001203180 -0.005875
025033 0.000163180 -0.006375

Edit: added rest of pitch error values and all roll error values

Last edited 4 years ago by dap (previous) (diff)

comment:27 Changed 4 years ago by dap

RCD Processing

Have received a reply from Tom regarding the spurious RCD images. The spurious images are not noted in the flight log sheet so it is assumed that these were just test images. They have been moved to raw_images/spurious and won't be processed.

comment:28 Changed 4 years ago by dap

RCD Processing

RCD processing with re-processed navigation data now complete. RG13_06-288-camera-20150224 ready for delivery check.

Last edited 4 years ago by dap (previous) (diff)

comment:29 Changed 4 years ago by lah

Fenix DC

Suggest new SCT set:

Flightline FENIX
1 -0.10
2 1.00
3 0.90
4 0.90
5 0.90
6 1.06
7 0.95
8 0.99
9 1.00
10 0.97
11 0.93
12 0.97
13 0.90

comment:30 Changed 4 years ago by dap

LiDAR Processing

Roll and pitch error values for all flight lines have all been found (comment:26 edited accordingly). Now producing LAS1.3 files.

comment:31 Changed 4 years ago by dap

LiDAR Processing

All files have been generated and delivery has been created. Tried to create a read me document, and consistently ran in to the following error:

pdfTeX warning: pdflatex: arithmetic: number too big


pdfTeX warning: pdflatex: arithmetic: number too big
</tmp/RG13_06-2014_288-intensity.jpg, id=49, --32768.0pt x 0.0pt>
<use /tmp/RG13_06-2014_288-intensity.jpg>
! Dimension too large.
\Gscale@box ...fdim #1\p@ <\z@ \hb@xt@ -#1\wd \z@
                                                  {\kern -#1\wd \z@ \box
\tw...
l.365 ...5cm]{/tmp/RG13_06-2014_288-intensity.jpg}
                                                 
?
! Dimension too large.
\Gscale@box ...b@xt@ -#1\wd \z@ {\kern -#1\wd \z@
                                                  \box \tw@ \hss }\else
\wd ...
l.365 ...5cm]{/tmp/RG13_06-2014_288-intensity.jpg}
                                                 
?
[6 </tmp/RG13_06-2014_288-intensity.jpg>]

pdfTeX warning: pdflatex: arithmetic: number too big


pdfTeX warning: pdflatex: arithmetic: number too big
</tmp/RG13_06-2014_288-dem.jpg, id=57, --32768.0pt x 0.0pt>
<use /tmp/RG13_06-2014_288-dem.jpg>
! Dimension too large.
\Gscale@box ...fdim #1\p@ <\z@ \hb@xt@ -#1\wd \z@
                                                  {\kern -#1\wd \z@ \box
\tw...
l.372 ...th=16.5cm]{/tmp/RG13_06-2014_288-dem.jpg}
                                                 
?
! Dimension too large.
\Gscale@box ...b@xt@ -#1\wd \z@ {\kern -#1\wd \z@
                                                  \box \tw@ \hss }\else
\wd ...
l.372 ...th=16.5cm]{/tmp/RG13_06-2014_288-dem.jpg}

Currently investigating this.

comment:32 Changed 4 years ago by dap

LiDAR Processing

Read me document has now been created. Had to reconvert the DEM and intensity images for use in the read me using convert -thumbnail 1024 -density 300 (the create_latex_lidar_readme.py script usually uses convert -thumbnail 1024). Will keep an eye on this to see if this is a problem that occurs frequently.

LiDAR dataset now ready for delivery check.

comment:33 Changed 4 years ago by dac

LiDAR DC
Starting delivery check

comment:34 Changed 4 years ago by dac

LiDAR DC

As with 302a, too many within canopy returns are classified as noise. Suggest redoing classification using a different threshold, then will resume delivery check.

comment:35 Changed 4 years ago by tec

RCD Delivery Check
Starting DC

comment:36 Changed 4 years ago by tec

RCD Delivery Check
Write text in readme about blue tint to images.
Remove photos:
00015
00016
00110
00143
00270
00409 possibly
00476
00477
00532
00542
00574
00575
00634
00635
00642
00656
00657
00658
00821
00824
00836
00837
00838
00839

KML looks ok but images, not showing. Inspected KML and it looks ok. Could be an issue with google earth.

Delivery will be ok once images are removed and preferably check the KML on google earth on windoze machine.

Also theres an old camera delivery, remove it if its not needed.

DC Done

Last edited 4 years ago by tec (previous) (diff)

comment:37 Changed 4 years ago by dap

LiDAR Processing

  • Reclassified all files using the isolated points algorithm (5 points within 10 metres) and manually and converted the new files to ascii (with Windows line endings).
  • Regenerated DEMs with new LAS files.
  • Copied new classifications to FW files.
  • Regenerated read me with new DEM screenshot and point density information.

LiDAR DC can now be resumed.

Last edited 4 years ago by dap (previous) (diff)

comment:38 Changed 4 years ago by dap

RCD Processing

Removed the following images (from the thumbnails and photographs directories):

  • 00015
  • 00143
  • 00270
  • 00477
  • 00532
  • 00574
  • 00634
  • 00635
  • 00642
  • 00657
  • 00658
  • 00821
  • 00824
  • 00837

The rest have been kept as they have a sufficient number of usable pixels.
Have also renamed files so that they are numbered sequentially (and changed the file name entries in the event file) and regenerated the kml file and read me.

Removed previous deliveries.

RCD ready for delivery.

comment:39 Changed 4 years ago by dac

LiDAR DC

Starting LiDAR Delivery Check.

comment:40 Changed 4 years ago by dac

LiDAR DC

Finished delivery check. Looks fine, and updated classification is much better and ready to send. Couple of minor comments:

  • I couldn't get the DEM screenshot to open in eog, it opened in GIMP and on Windows fine though.
  • Mean difference between LiDAR and ASTER from dem_compare.py: 13.7 m
  • Some of the height differences between lines are larger than normal (as noted in the Readme), there is also a comment explaining why this is so no issue there.
  • The differences in the readme should be rounded to the nearest cm.

comment:41 Changed 4 years ago by dap

RCD and LiDAR Data Delivery

Rsyncing deliveries to ftp server for delivery.

comment:42 Changed 4 years ago by dap

Fenix Processing

Processing flight lines in slow mode with all bands using SCT values suggested by Laura. Also had to replace the wavelengths in the header files using the ones from the 2015 calibration. I've moved the header files generated by Terry from the 2014 calibration to old_cal_headers directory.

comment:43 Changed 4 years ago by dap

Fenix Processing

Now complete and ready for delivery check.

The calibration had to be redone so these files could not be used.

Last edited 4 years ago by dap (previous) (diff)

comment:44 Changed 4 years ago by dap

Fenix Processing

Fenix processing with new calibration results complete and delivery created. Newer delivery now ready for delivery check (I'll delete the old one when delivery check complete).

comment:45 Changed 4 years ago by tec

Delivery Check
Starting Fenix delivery check

comment:46 Changed 4 years ago by tec

Delivery Check
Delete old delivery

flightlines/mapped/f288083b_mapped_utm50n.bil <-- has no header file
Other than that, all is ok. Once header file is present then it will be ready. You'll need to zip fenix files and then move to /delivery

comment:47 Changed 4 years ago by dap

Fenix Processing

Reprocessing line 8 so that header file is remade.

comment:48 Changed 4 years ago by dap

Fenix Delivery

rsync of data to FTP:arsf1 complete, following Terry saying that the new mapped bil and hdr files for flight line 8 is OK to deliver. PI has not yet been notified of delivery.

Last edited 4 years ago by dap (previous) (diff)

comment:49 Changed 4 years ago by lah

PI notified of delivery 05/05/2015 (Fenix, Lidar & RCD)

comment:50 Changed 4 years ago by dac

Fenix

Replaced wavescales in Fenix headers with updated calibration. Processed but locations of peaks and troughs are offset compared to 309 and 219a. Previous analysis indicated minima for this flight were different to other flights from the same campaign so it is possible there is a problem specific to this flight. Will investigate further after processing other data from other days.

comment:51 Changed 3 years ago by dac

Fenix

Still haven't got to the bottom of wavelength shift for this flight - the sensor wasn't stable so we can't be sure wavelengths are consistent for all lines,

Has been on hold for a while now so will process using best available calibration and provide 'as-is' for now.

Replaced wavelengths with those from 2015 calibration (as used for the other data) and set mapping running.

comment:52 Changed 3 years ago by dac

Mapped files. XML generation failed with error:

argument --boresight: expected 3 argument(s)

Also noticed after running mapped files have pixel size of 4.19 m. Will redo with a more sensible pixel size.

comment:53 Changed 3 years ago by dac

Fenix

Re-mapped files with a pixel size of 4 m and generated delivery.

Ready for delivery check.

comment:54 Changed 3 years ago by lah

Fenix DC

Readme:

  • Probably worth mentioning that the believed offset is also not consistent for each flightline and may also be non-uniform across the wavelengths.
  • We can be more specific and say during the 2014 Malaysia campaign the shift occurred, which could be compensated for with the 2015 calibration, but some instability also occurred which is worst for this data - could guess that environmental conditions might have been a cause or an unknown cause.
  • warning: compared to the true value. Again should we mention here that they are unstable and the offset may be different for different flightlines and bands
  • copied logsheet into delivery
  • xml files have 2014 DQ details, probably should be 2015 if we've used 2015 calibration

comment:55 Changed 3 years ago by dac

  • Made suggested changes to readme text
  • Updated data quality report details in XML files

Started zipping mapped files (as no changes needed on these).

comment:56 Changed 3 years ago by lah

Fenix DC

Everything else as good as can be, so ready to deliver.

comment:57 Changed 3 years ago by dac

Fenix

Delivered to PI via FTP (arsf10).

comment:58 Changed 3 years ago by dac

Archiving

Owl data in processing directory needs to be tidied up before archiving. Currently have owl, owl1 and owl2.

comment:59 Changed 3 years ago by lah

Removed owl processing directories as these were only for testing the processing chain, not actually related to operational processing.

comment:60 Changed 3 years ago by dac

Archiving

Started uploading data to NEODC.

comment:61 Changed 3 years ago by dac

  • Resolution set to fixed
  • Status changed from new to closed

Archiving

Data now available from NEODC: http://browse.ceda.ac.uk/browse/neodc/arsf/2014/RG13_06/RG13_06-2014_288_Danum_Valley

Marking as closed.

Note: See TracTickets for help on using tickets.