RG13/06, flight day 288/2014, Danum Valley

[dap]

Data location: /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_288_Danum_Valley

Data arrived by network transfer on 17/12/2014.

Scientific objective: Investigating the impact of humans on tropical forests.

Priority: alpha 5

PI: David Coomes

Although Owl data are present, proj_tidy suggests that we do not have dark frame information for the following header files:

• David_Coomes_Safe_0124.hdr
• David_Coomes_Safe_0125.hdr
• David_Coomes_Safe_0126.hdr
• David_Coomes_Safe_0127.hdr
• David_Coomes_Safe_0128.hdr

Sensors:

• Fenix (requested)
• Leica FW LIDAR (?)
• Leica LIDAR (requested)
• RCD (requested)

[tec]

Navigation Processing
Starting basestation processing.

[tec]

Navigation Processing

SAFE_5th_ Basestation Data
Latitude	4 43 30.56082
Longitude	117 36 19.18784
El Height	479.784m

[tec]

Navigation Processing
Basestation nav does not cover all of the IMU data, but it covers the flightlines so its fine. Processing with Ipas Pro as Ipas TC crashes.

Settings used which are different to the defaults are as follows:
Elevation Mask: 14.0 degrees
KAR Maximum dual frequency distance: 80.00 km
Code: 1.50m

There is a gap in the IMU data for line 166 from 01:31:49.00 -> 01:32:00.00
At the start of line 168 the gps data is of quality 2 for from 01:34:03.00 -> 01:34:11.00

Processing finished.

[tec]

RCD Processing
Started converting tifs, created image event csv

[tec]

Fenix Processing
Started fenix processing

[tec]

RCD Processing
Ready for delivery check

[dap]

Beginning RCD delivery check

[dap]

RCD Delivery Check

Read me

• 'a' in LaTeX template should be `a'. I've changed the read me template so this doesn't happen again. Waiting for the change to be reviewed merged in to the master branch.
• Images 1 - 60 should probably not be included in the delivery as they are out of bounds. Don't forget to change the data quality remarks section of the read me, where you mention that the photographs are not in the area.

Images

• 00001 is over-exposed and should be removed.
• Would suggest removing images 522 - 524 as they have excessive cloud coverage.

I'll recheck the delivery when the above issues have been addressed.

[tec]

RCD
Made changes
Ready for DC

[dap]

Rechecking RCD Delivery

[dap]

RCD Delivery Checking complete - delivery at
processing/delivery/RG13_06-288-camera-20150107 ready for delivery.

[dap]

Beginning LiDAR Processing

[tec]

Fenix Processing
Started finding SCT values

[dap]

Began LiDAR processing on 15/01/2015

[dap]

Since the delivery check of the RCD delivery for this flight, I have noticed that some of the thumbnails are missing from the kml files in flightline no. 12. Have therefore marked RCD as "in progress".

[tec]

Fenix Processing
Sent to the grid to process all bands. Line 13 seems to be broke, will look into that

Flightline	FENIX
1	0.10
2	0.90
3	0.90
4	0.90
5	0.90
6	1.06
7	0.79
8	1.13
9	0.67
10	1.30
11	0.53
12	1.41
13	?

[dap]

Fenix Processing

Taken over from tec. Investigated problem with line 13. Problem was that apl could not find the raw .nav file associated with this line. Have added nav2_args_extra = -nav <path/to/FENIX288-14-13.nav to DEFAULT section of the hyperspectral config and started the processing of line 13.

[dap]

Replying to tec:

Fenix Processing
Sent to the grid to process all bands. Line 13 seems to be broke, will look into that

Flightline	FENIX
1	0.10
2	0.90
3	0.90
4	0.90
5	0.90
6	1.06
7	0.79
8	1.13
9	0.67
10	1.30
11	0.53
12	1.41
13	0.90

Added SCT value for flight line 13 to table

[dap]

Fenix Processing

Line information xml files were not generated or have been deleted - reprocessing the data to regenerate them.

[dap]

Fenix Processing

Delivery has been created and read me created. Fenix data now ready for check.

[dap]

RCD

Found problems with RCD delivery created. Processing has been handed to me. Re-processing the images to create a new delivery.

[dap]

RCD Processing

Ran rcdimages_in_pdf.py and found that 60 of the thumbnails (the ones beginning with 140143221) were not displayed in the pdf. Have searched the rcd log for the names of the missing files and they are not in there. This suggests that the images that weren't displayed in the pdf file could be from a different flight.

Have emailed Tom Millard to find out what went wrong.

[lah]

Fenix DC

• Removed fodis folder.
• Converted txt file.

• SCTs mostly look ok, but slightly off between 1-2, 7-8, 10-11 and way off for 11-12 (roads twist in different directions on the different lines). Worth re-checking, though could be slope viewing effects in some places.
• Readme has “0cm” at top of first page. Also not much point of having a table start with only 1 line (pg 2). Many paragraphs start indented, whilst other do not – definitely needs fixing. Line missing between files 2 & 3 in under/overflow table.
• Processing command produces tiffs ok, though does state error 6: “SetColorTable() can only be called on band 1” and “SetColorTable() not supported for multi-sample TIFF files” a lot. Lots of code still complains that it doesn't recognise 'WGS84' because it's in quotes.
  • Under/overflows seem reasonable, though some may have been missed, possibly due to the threshold being set too low.

[dac]

Navigation Processing

Reprocessed base station data.

Latitude	4 43 30.56087
Longitude	117 36 19.18807
El Height	480.154 m

[dac]

Navigation Processing
Reprocessed navigation data in IPAS TC

[dap]

LiDAR Processing

All flight lines I had found pitch offsets for with the old sol file have now been reprocessed with new sol file. The pitch offsets used for the data processed with the old sol file seem to provide the same result with the new one in terms of pitch and roll. Pitch error offsets found up to now:

Flightline	Pitch Error	Roll Error
004022	0.000163180	-0.006375
005105	-0.000921318	-0.006175
010204	0.000090032	-0.006375
011317	-0.000863180	-0.006175
012339	0.000443180	-0.006375
013422	-0.001243180	-0.005975
014456	0.000203180	-0.006375
015527	-0.001203180	-0.005875
020544	0.000163180	-0.006375
021635	-0.001203180	-0.005875
022705	0.000263180	-0.006375
023739	-0.001203180	-0.005875
025033	0.000163180	-0.006375

Edit: added rest of pitch error values and all roll error values

[dap]

RCD Processing

Have received a reply from Tom regarding the spurious RCD images. The spurious images are not noted in the flight log sheet so it is assumed that these were just test images. They have been moved to raw_images/spurious and won't be processed.

[dap]

RCD Processing

RCD processing with re-processed navigation data now complete.

[lah]

Fenix DC

Suggest new SCT set:

Flightline	FENIX
1	-0.10
2	1.00
3	0.90
4	0.90
5	0.90
6	1.06
7	0.95
8	0.99
9	1.00
10	0.97
11	0.93
12	0.97
13	0.90

[dap]

LiDAR Processing

Roll and pitch error values for all flight lines have all been found (comment:26 edited accordingly). Now producing LAS1.3 files.

[dap]

LiDAR Processing

All files have been generated and delivery has been created. Tried to create a read me document, and consistently ran in to the following error:

pdfTeX warning: pdflatex: arithmetic: number too big


pdfTeX warning: pdflatex: arithmetic: number too big
</tmp/RG13_06-2014_288-intensity.jpg, id=49, --32768.0pt x 0.0pt>
<use /tmp/RG13_06-2014_288-intensity.jpg>
! Dimension too large.
\Gscale@box ...fdim #1\p@ <\z@ \hb@xt@ -#1\wd \z@
                                                  {\kern -#1\wd \z@ \box
\tw...
l.365 ...5cm]{/tmp/RG13_06-2014_288-intensity.jpg}
                                                 
?
! Dimension too large.
\Gscale@box ...b@xt@ -#1\wd \z@ {\kern -#1\wd \z@
                                                  \box \tw@ \hss }\else
\wd ...
l.365 ...5cm]{/tmp/RG13_06-2014_288-intensity.jpg}
                                                 
?
[6 </tmp/RG13_06-2014_288-intensity.jpg>]

pdfTeX warning: pdflatex: arithmetic: number too big


pdfTeX warning: pdflatex: arithmetic: number too big
</tmp/RG13_06-2014_288-dem.jpg, id=57, --32768.0pt x 0.0pt>
<use /tmp/RG13_06-2014_288-dem.jpg>
! Dimension too large.
\Gscale@box ...fdim #1\p@ <\z@ \hb@xt@ -#1\wd \z@
                                                  {\kern -#1\wd \z@ \box
\tw...
l.372 ...th=16.5cm]{/tmp/RG13_06-2014_288-dem.jpg}
                                                 
?
! Dimension too large.
\Gscale@box ...b@xt@ -#1\wd \z@ {\kern -#1\wd \z@
                                                  \box \tw@ \hss }\else
\wd ...
l.372 ...th=16.5cm]{/tmp/RG13_06-2014_288-dem.jpg}

Currently investigating this.

[dap]

LiDAR Processing

Read me document has now been created. Had to reconvert the DEM and intensity images for use in the read me using convert -thumbnail 1024 -density 300 (the create_latex_lidar_readme.py script usually uses convert -thumbnail 1024). Will keep an eye on this to see if this is a problem that occurs frequently.

LiDAR dataset now ready for delivery check.

[dac]

LiDAR DC
Starting delivery check

[dac]

LiDAR DC

As with 302a, too many within canopy returns are classified as noise. Suggest redoing classification using a different threshold, then will resume delivery check.

[tec]

RCD Delivery Check
Starting DC

[tec]

RCD Delivery Check
Write text in readme about blue tint to images.
Remove photos:
00015
00016
00110
00143
00270
00409 possibly
00476
00477
00532
00542
00574
00575
00634
00635
00642
00656
00657
00658
00821
00824
00836
00837
00838
00839

KML looks ok but images, not showing. Inspected KML and it looks ok. Could be an issue with google earth.

Delivery will be ok once images are removed and preferably check the KML on google earth on windoze machine.

Also theres an old camera delivery, remove it if its not needed.

DC Done

[dap]

LiDAR Processing

• Reclassified all files using the isolated points algorithm (5 points within 10 metres) and manually and converted the new files to ascii (with Windows line endings).
• Regenerated DEMs with new LAS files.
• Copied new classifications to FW files.
• Regenerated read me with new DEM screenshot and point density information.

LiDAR DC can now be resumed.

[dap]

RCD Processing

Removed the following images (from the thumbnails and photographs directories):

• 00015
• 00143
• 00270
• 00477
• 00532
• 00574
• 00634
• 00635
• 00642
• 00657
• 00658
• 00821
• 00824
• 00837

The rest have been kept as they have a sufficient number of usable pixels.
Have also renamed files so that they are numbered sequentially (and changed the file name entries in the event file) and regenerated the kml file and read me.

Removed previous deliveries.

RCD ready for delivery.

[dac]

LiDAR DC

Starting LiDAR Delivery Check.

[dac]

LiDAR DC

Finished delivery check. Looks fine, and updated classification is much better and ready to send. Couple of minor comments:

• I couldn't get the DEM screenshot to open in eog, it opened in GIMP and on Windows fine though.
• Mean difference between LiDAR and ASTER from dem_compare.py: 13.7 m
• Some of the height differences between lines are larger than normal (as noted in the Readme), there is also a comment explaining why this is so no issue there.
• The differences in the readme should be rounded to the nearest cm.

[dap]

RCD and LiDAR Data Delivery

Rsyncing deliveries to ftp server for delivery.

[dap]

Fenix Processing

Processing flight lines in slow mode with all bands using SCT values suggested by Laura. Also had to replace the wavelengths in the header files using the ones from the 2015 calibration. I've moved the header files generated by Terry from the 2014 calibration to old_cal_headers directory.

[dap]

Fenix Processing

Now complete and ready for delivery check.

The calibration had to be redone so these files could not be used.

[dap]

Fenix Processing

Fenix processing with new calibration results complete and delivery created. Newer delivery now ready for delivery check (I'll delete the old one when delivery check complete).

[tec]

Delivery Check
Starting Fenix delivery check

[tec]

Delivery Check
Delete old delivery

flightlines/mapped/f288083b_mapped_utm50n.bil <-- has no header file
Other than that, all is ok. Once header file is present then it will be ready. You'll need to zip fenix files and then move to /delivery

[dap]

Fenix Processing

Reprocessing line 8 so that header file is remade.

[dap]

Fenix Delivery

rsync of data to FTP:arsf1 complete, following Terry saying that the new mapped bil and hdr files for flight line 8 is OK to deliver. PI has not yet been notified of delivery.

[lah]

PI notified of delivery 05/05/2015 (Fenix, Lidar & RCD)

[dac]

Fenix

Replaced wavescales in Fenix headers with updated calibration. Processed but locations of peaks and troughs are offset compared to 309 and 219a. Previous analysis indicated minima for this flight were different to other flights from the same campaign so it is possible there is a problem specific to this flight. Will investigate further after processing other data from other days.

[dac]

Fenix

Still haven't got to the bottom of wavelength shift for this flight - the sensor wasn't stable so we can't be sure wavelengths are consistent for all lines,

Has been on hold for a while now so will process using best available calibration and provide 'as-is' for now.

Replaced wavelengths with those from 2015 calibration (as used for the other data) and set mapping running.

[dac]

Mapped files. XML generation failed with error:

argument --boresight: expected 3 argument(s)

Also noticed after running mapped files have pixel size of 4.19 m. Will redo with a more sensible pixel size.

[dac]

Fenix

Re-mapped files with a pixel size of 4 m and generated delivery.

Ready for delivery check.

[lah]

Fenix DC

Readme:

• Probably worth mentioning that the believed offset is also not consistent for each flightline and may also be non-uniform across the wavelengths.
• We can be more specific and say during the 2014 Malaysia campaign the shift occurred, which could be compensated for with the 2015 calibration, but some instability also occurred which is worst for this data - could guess that environmental conditions might have been a cause or an unknown cause.
• warning: compared to the true value. Again should we mention here that they are unstable and the offset may be different for different flightlines and bands

• copied logsheet into delivery
• xml files have 2014 DQ details, probably should be 2015 if we've used 2015 calibration

[dac]

• Made suggested changes to readme text
• Updated data quality report details in XML files

Started zipping mapped files (as no changes needed on these).

[lah]

Fenix DC

Everything else as good as can be, so ready to deliver.

[dac]

Fenix

Delivered to PI via FTP (arsf10).

[dac]

Archiving

Owl data in processing directory needs to be tidied up before archiving. Currently have owl, owl1 and owl2.

[lah]

Removed owl processing directories as these were only for testing the processing chain, not actually related to operational processing.

[dac]

Archiving

Started uploading data to NEODC.

[dac]

Archiving

Data now available from NEODC: ​http://browse.ceda.ac.uk/browse/neodc/arsf/2014/RG13_06/RG13_06-2014_288_Danum_Valley

Marking as closed.