Opened 10 years ago
Closed 8 years ago
#555 closed flight processing (fixed)
RG13/06, flight day 288/2014, Danum Valley
Reported by: | dap | Owned by: | |
---|---|---|---|
Priority: | alpha 5 | Milestone: | |
Component: | Processing: general | Keywords: | |
Cc: | Other processors: | tec |
Description
Data location: /users/rsg/arsf/arsf_data/2014/flight_data/malaysia/RG13_06-2014_288_Danum_Valley
Data arrived by network transfer on 17/12/2014.
Scientific objective: Investigating the impact of humans on tropical forests.
Priority: alpha 5
PI: David Coomes
Although Owl data are present, proj_tidy suggests that we do not have dark frame information for the following header files:
- David_Coomes_Safe_0124.hdr
- David_Coomes_Safe_0125.hdr
- David_Coomes_Safe_0126.hdr
- David_Coomes_Safe_0127.hdr
- David_Coomes_Safe_0128.hdr
Sensors:
- Fenix (requested)
- Leica FW LIDAR (?)
- Leica LIDAR (requested)
- RCD (requested)
Change History (61)
comment:1 Changed 10 years ago by tec
comment:2 Changed 10 years ago by tec
Navigation Processing
SAFE_5th_ Basestation Data | |
---|---|
Latitude | 4 43 30.56082 |
Longitude | 117 36 19.18784 |
El Height | 479.784m |
comment:3 Changed 10 years ago by tec
Navigation Processing
Basestation nav does not cover all of the IMU data, but it covers the flightlines so its fine. Processing with Ipas Pro as Ipas TC crashes.
Settings used which are different to the defaults are as follows:
Elevation Mask: 14.0 degrees
KAR Maximum dual frequency distance: 80.00 km
Code: 1.50m
There is a gap in the IMU data for line 166 from 01:31:49.00 -> 01:32:00.00
At the start of line 168 the gps data is of quality 2 for from 01:34:03.00 -> 01:34:11.00
Processing finished.
comment:4 Changed 10 years ago by tec
RCD Processing
Started converting tifs, created image event csv
comment:5 Changed 10 years ago by tec
Fenix Processing
Started fenix processing
comment:6 Changed 10 years ago by tec
RCD Processing
Ready for delivery check
comment:7 Changed 10 years ago by dap
Beginning RCD delivery check
comment:8 Changed 10 years ago by dap
RCD Delivery Check
Read me
- 'a' in LaTeX template should be `a'. I've changed the read me template so this doesn't happen again. Waiting for the change to be reviewed merged in to the master branch.
- Images 1 - 60 should probably not be included in the delivery as they are out of bounds. Don't forget to change the data quality remarks section of the read me, where you mention that the photographs are not in the area.
Images
- 00001 is over-exposed and should be removed.
- Would suggest removing images 522 - 524 as they have excessive cloud coverage.
I'll recheck the delivery when the above issues have been addressed.
comment:9 Changed 10 years ago by tec
RCD
Made changes
Ready for DC
comment:10 Changed 10 years ago by dap
Rechecking RCD Delivery
comment:11 Changed 10 years ago by dap
RCD Delivery Checking complete - delivery at
processing/delivery/RG13_06-288-camera-20150107 ready for delivery.
comment:12 Changed 10 years ago by dap
Beginning LiDAR Processing
comment:13 Changed 10 years ago by tec
- Other processors set to tec
Fenix Processing
Started finding SCT values
comment:14 Changed 10 years ago by dap
Began LiDAR processing on 15/01/2015
comment:15 Changed 10 years ago by dap
Since the delivery check of the RCD delivery for this flight, I have noticed that some of the thumbnails are missing from the kml files in flightline no. 12. Have therefore marked RCD as "in progress".
comment:16 follow-up: ↓ 18 Changed 10 years ago by tec
Fenix Processing
Sent to the grid to process all bands. Line 13 seems to be broke, will look into that
Flightline | FENIX |
---|---|
1 | 0.10 |
2 | 0.90 |
3 | 0.90 |
4 | 0.90 |
5 | 0.90 |
6 | 1.06 |
7 | 0.79 |
8 | 1.13 |
9 | 0.67 |
10 | 1.30 |
11 | 0.53 |
12 | 1.41 |
13 | ? |
comment:17 Changed 10 years ago by dap
Fenix Processing
Taken over from tec. Investigated problem with line 13. Problem was that apl could not find the raw .nav file associated with this line. Have added nav2_args_extra = -nav <path/to/FENIX288-14-13.nav to DEFAULT section of the hyperspectral config and started the processing of line 13.
comment:18 in reply to: ↑ 16 Changed 10 years ago by dap
Replying to tec:
Fenix Processing
Sent to the grid to process all bands. Line 13 seems to be broke, will look into that
Flightline FENIX 1 0.10 2 0.90 3 0.90 4 0.90 5 0.90 6 1.06 7 0.79 8 1.13 9 0.67 10 1.30 11 0.53 12 1.41 13 0.90
Added SCT value for flight line 13 to table
comment:19 Changed 10 years ago by dap
Fenix Processing
Line information xml files were not generated or have been deleted - reprocessing the data to regenerate them.
comment:20 Changed 10 years ago by dap
Fenix Processing
Delivery has been created and read me created. Fenix data now ready for check.
comment:21 Changed 10 years ago by dap
RCD
Found problems with RCD delivery created. Processing has been handed to me. Re-processing the images to create a new delivery.
comment:22 Changed 10 years ago by dap
RCD Processing
Ran rcdimages_in_pdf.py and found that 60 of the thumbnails (the ones beginning with 140143221) were not displayed in the pdf. Have searched the rcd log for the names of the missing files and they are not in there. This suggests that the images that weren't displayed in the pdf file could be from a different flight.
Have emailed Tom Millard to find out what went wrong.
comment:23 Changed 10 years ago by lah
Fenix DC
- Removed fodis folder.
- Converted txt file.
- SCTs mostly look ok, but slightly off between 1-2, 7-8, 10-11 and way off for 11-12 (roads twist in different directions on the different lines). Worth re-checking, though could be slope viewing effects in some places.
- Readme has “0cm” at top of first page. Also not much point of having a table start with only 1 line (pg 2). Many paragraphs start indented, whilst other do not – definitely needs fixing. Line missing between files 2 & 3 in under/overflow table.
- Processing command produces tiffs ok, though does state error 6: “SetColorTable() can only be called on band 1” and “SetColorTable() not supported for multi-sample TIFF files” a lot. Lots of code still complains that it doesn't recognise 'WGS84' because it's in quotes.
- Under/overflows seem reasonable, though some may have been missed, possibly due to the threshold being set too low.
comment:24 Changed 10 years ago by dac
Navigation Processing
Reprocessed base station data.
Latitude | 4 43 30.56087 |
Longitude | 117 36 19.18807 |
El Height | 480.154 m |
comment:25 Changed 10 years ago by dac
Navigation Processing
Reprocessed navigation data in IPAS TC
comment:26 Changed 10 years ago by dap
LiDAR Processing
All flight lines I had found pitch offsets for with the old sol file have now been reprocessed with new sol file. The pitch offsets used for the data processed with the old sol file seem to provide the same result with the new one in terms of pitch and roll. Pitch error offsets found up to now:
Flightline | Pitch Error | Roll Error |
---|---|---|
004022 | 0.000163180 | -0.006375 |
005105 | -0.000921318 | -0.006175 |
010204 | 0.000090032 | -0.006375 |
011317 | -0.000863180 | -0.006175 |
012339 | 0.000443180 | -0.006375 |
013422 | -0.001243180 | -0.005975 |
014456 | 0.000203180 | -0.006375 |
015527 | -0.001203180 | -0.005875 |
020544 | 0.000163180 | -0.006375 |
021635 | -0.001203180 | -0.005875 |
022705 | 0.000263180 | -0.006375 |
023739 | -0.001203180 | -0.005875 |
025033 | 0.000163180 | -0.006375 |
Edit: added rest of pitch error values and all roll error values
comment:27 Changed 10 years ago by dap
RCD Processing
Have received a reply from Tom regarding the spurious RCD images. The spurious images are not noted in the flight log sheet so it is assumed that these were just test images. They have been moved to raw_images/spurious and won't be processed.
comment:28 Changed 10 years ago by dap
RCD Processing
RCD processing with re-processed navigation data now complete. RG13_06-288-camera-20150224 ready for delivery check.
comment:29 Changed 10 years ago by lah
Fenix DC
Suggest new SCT set:
Flightline | FENIX |
---|---|
1 | -0.10 |
2 | 1.00 |
3 | 0.90 |
4 | 0.90 |
5 | 0.90 |
6 | 1.06 |
7 | 0.95 |
8 | 0.99 |
9 | 1.00 |
10 | 0.97 |
11 | 0.93 |
12 | 0.97 |
13 | 0.90 |
comment:30 Changed 10 years ago by dap
LiDAR Processing
Roll and pitch error values for all flight lines have all been found (comment:26 edited accordingly). Now producing LAS1.3 files.
comment:31 Changed 10 years ago by dap
LiDAR Processing
All files have been generated and delivery has been created. Tried to create a read me document, and consistently ran in to the following error:
pdfTeX warning: pdflatex: arithmetic: number too big pdfTeX warning: pdflatex: arithmetic: number too big </tmp/RG13_06-2014_288-intensity.jpg, id=49, --32768.0pt x 0.0pt> <use /tmp/RG13_06-2014_288-intensity.jpg> ! Dimension too large. \Gscale@box ...fdim #1\p@ <\z@ \hb@xt@ -#1\wd \z@ {\kern -#1\wd \z@ \box \tw... l.365 ...5cm]{/tmp/RG13_06-2014_288-intensity.jpg} ? ! Dimension too large. \Gscale@box ...b@xt@ -#1\wd \z@ {\kern -#1\wd \z@ \box \tw@ \hss }\else \wd ... l.365 ...5cm]{/tmp/RG13_06-2014_288-intensity.jpg} ? [6 </tmp/RG13_06-2014_288-intensity.jpg>] pdfTeX warning: pdflatex: arithmetic: number too big pdfTeX warning: pdflatex: arithmetic: number too big </tmp/RG13_06-2014_288-dem.jpg, id=57, --32768.0pt x 0.0pt> <use /tmp/RG13_06-2014_288-dem.jpg> ! Dimension too large. \Gscale@box ...fdim #1\p@ <\z@ \hb@xt@ -#1\wd \z@ {\kern -#1\wd \z@ \box \tw... l.372 ...th=16.5cm]{/tmp/RG13_06-2014_288-dem.jpg} ? ! Dimension too large. \Gscale@box ...b@xt@ -#1\wd \z@ {\kern -#1\wd \z@ \box \tw@ \hss }\else \wd ... l.372 ...th=16.5cm]{/tmp/RG13_06-2014_288-dem.jpg}
Currently investigating this.
comment:32 Changed 10 years ago by dap
LiDAR Processing
Read me document has now been created. Had to reconvert the DEM and intensity images for use in the read me using convert -thumbnail 1024 -density 300 (the create_latex_lidar_readme.py script usually uses convert -thumbnail 1024). Will keep an eye on this to see if this is a problem that occurs frequently.
LiDAR dataset now ready for delivery check.
comment:33 Changed 10 years ago by dac
LiDAR DC
Starting delivery check
comment:34 Changed 10 years ago by dac
LiDAR DC
As with 302a, too many within canopy returns are classified as noise. Suggest redoing classification using a different threshold, then will resume delivery check.
comment:35 Changed 10 years ago by tec
RCD Delivery Check
Starting DC
comment:36 Changed 10 years ago by tec
RCD Delivery Check
Write text in readme about blue tint to images.
Remove photos:
00015
00016
00110
00143
00270
00409 possibly
00476
00477
00532
00542
00574
00575
00634
00635
00642
00656
00657
00658
00821
00824
00836
00837
00838
00839
KML looks ok but images, not showing. Inspected KML and it looks ok. Could be an issue with google earth.
Delivery will be ok once images are removed and preferably check the KML on google earth on windoze machine.
Also theres an old camera delivery, remove it if its not needed.
DC Done
comment:37 Changed 10 years ago by dap
LiDAR Processing
- Reclassified all files using the isolated points algorithm (5 points within 10 metres) and manually and converted the new files to ascii (with Windows line endings).
- Regenerated DEMs with new LAS files.
- Copied new classifications to FW files.
- Regenerated read me with new DEM screenshot and point density information.
LiDAR DC can now be resumed.
comment:38 Changed 10 years ago by dap
RCD Processing
Removed the following images (from the thumbnails and photographs directories):
- 00015
- 00143
- 00270
- 00477
- 00532
- 00574
- 00634
- 00635
- 00642
- 00657
- 00658
- 00821
- 00824
- 00837
The rest have been kept as they have a sufficient number of usable pixels.
Have also renamed files so that they are numbered sequentially (and changed the file name entries in the event file) and regenerated the kml file and read me.
Removed previous deliveries.
RCD ready for delivery.
comment:39 Changed 10 years ago by dac
LiDAR DC
Starting LiDAR Delivery Check.
comment:40 Changed 10 years ago by dac
LiDAR DC
Finished delivery check. Looks fine, and updated classification is much better and ready to send. Couple of minor comments:
- I couldn't get the DEM screenshot to open in eog, it opened in GIMP and on Windows fine though.
- Mean difference between LiDAR and ASTER from dem_compare.py: 13.7 m
- Some of the height differences between lines are larger than normal (as noted in the Readme), there is also a comment explaining why this is so no issue there.
- The differences in the readme should be rounded to the nearest cm.
comment:41 Changed 10 years ago by dap
RCD and LiDAR Data Delivery
Rsyncing deliveries to ftp server for delivery.
comment:42 Changed 10 years ago by dap
Fenix Processing
Processing flight lines in slow mode with all bands using SCT values suggested by Laura. Also had to replace the wavelengths in the header files using the ones from the 2015 calibration. I've moved the header files generated by Terry from the 2014 calibration to old_cal_headers directory.
comment:43 Changed 10 years ago by dap
Fenix Processing
Now complete and ready for delivery check.
comment:44 Changed 10 years ago by dap
Fenix Processing
Fenix processing with new calibration results complete and delivery created. Newer delivery now ready for delivery check (I'll delete the old one when delivery check complete).
comment:45 Changed 10 years ago by tec
Delivery Check
Starting Fenix delivery check
comment:46 Changed 10 years ago by tec
Delivery Check
Delete old delivery
flightlines/mapped/f288083b_mapped_utm50n.bil <-- has no header file
Other than that, all is ok. Once header file is present then it will be ready. You'll need to zip fenix files and then move to /delivery
comment:47 Changed 10 years ago by dap
Fenix Processing
Reprocessing line 8 so that header file is remade.
comment:48 Changed 10 years ago by dap
Fenix Delivery
rsync of data to FTP:arsf1 complete, following Terry saying that the new mapped bil and hdr files for flight line 8 is OK to deliver. PI has not yet been notified of delivery.
comment:49 Changed 10 years ago by lah
PI notified of delivery 05/05/2015 (Fenix, Lidar & RCD)
comment:50 Changed 9 years ago by dac
Fenix
Replaced wavescales in Fenix headers with updated calibration. Processed but locations of peaks and troughs are offset compared to 309 and 219a. Previous analysis indicated minima for this flight were different to other flights from the same campaign so it is possible there is a problem specific to this flight. Will investigate further after processing other data from other days.
comment:51 Changed 9 years ago by dac
Fenix
Still haven't got to the bottom of wavelength shift for this flight - the sensor wasn't stable so we can't be sure wavelengths are consistent for all lines,
Has been on hold for a while now so will process using best available calibration and provide 'as-is' for now.
Replaced wavelengths with those from 2015 calibration (as used for the other data) and set mapping running.
comment:52 Changed 9 years ago by dac
Mapped files. XML generation failed with error:
argument --boresight: expected 3 argument(s)
Also noticed after running mapped files have pixel size of 4.19 m. Will redo with a more sensible pixel size.
comment:53 Changed 9 years ago by dac
Fenix
Re-mapped files with a pixel size of 4 m and generated delivery.
Ready for delivery check.
comment:54 Changed 9 years ago by lah
Fenix DC
Readme:
- Probably worth mentioning that the believed offset is also not consistent for each flightline and may also be non-uniform across the wavelengths.
- We can be more specific and say during the 2014 Malaysia campaign the shift occurred, which could be compensated for with the 2015 calibration, but some instability also occurred which is worst for this data - could guess that environmental conditions might have been a cause or an unknown cause.
- warning: compared to the true value. Again should we mention here that they are unstable and the offset may be different for different flightlines and bands
- copied logsheet into delivery
- xml files have 2014 DQ details, probably should be 2015 if we've used 2015 calibration
comment:55 Changed 9 years ago by dac
- Made suggested changes to readme text
- Updated data quality report details in XML files
Started zipping mapped files (as no changes needed on these).
comment:56 Changed 9 years ago by lah
Fenix DC
Everything else as good as can be, so ready to deliver.
comment:57 Changed 9 years ago by dac
Fenix
Delivered to PI via FTP (arsf10).
comment:58 Changed 8 years ago by dac
Archiving
Owl data in processing directory needs to be tidied up before archiving. Currently have owl, owl1 and owl2.
comment:59 Changed 8 years ago by lah
Removed owl processing directories as these were only for testing the processing chain, not actually related to operational processing.
comment:60 Changed 8 years ago by dac
Archiving
Started uploading data to NEODC.
comment:61 Changed 8 years ago by dac
- Resolution set to fixed
- Status changed from new to closed
Archiving
Data now available from NEODC: http://browse.ceda.ac.uk/browse/neodc/arsf/2014/RG13_06/RG13_06-2014_288_Danum_Valley
Marking as closed.
Navigation Processing
Starting basestation processing.