Opened 9 years ago
Closed 7 years ago
#589 closed flight processing (fixed)
EU15/18, flight day 174, Toulouse
Reported by: | lah | Owned by: | |
---|---|---|---|
Priority: | immediate | Milestone: | |
Component: | Processing: general | Keywords: | |
Cc: | Other processors: |
Description (last modified by dac)
Data location: /users/rsg/arsf/arsf_data/2015/flight_data/france/EUFAR15_18-2015_174_Toulouse
Data arrived from ARSF via SATA disk on 09/07/2015.
Scientific objective: Hyperspectral measurements over agro-forestry areas for landscape assessment of agricultural health, ecophysiology, and satellite product validation.
Priority: Unknown
PI: Jean-Louis Roujean
EUFAR Project ID: AHSPECT
Sensors:
- Fenix (requested)
- Leica LIDAR (not requested)
- Leica FW LIDAR (not requested but flown anyway)
- Owl (not requested but flown anyway)
Change History (54)
comment:1 Changed 9 years ago by stgo
- Description modified (diff)
comment:2 Changed 9 years ago by gej
comment:3 Changed 9 years ago by lah
Unpacking
Removed corrupt webcam images:
- 150623_095016_000.jpg
- 150623_123907_000.jpg
- 150623_124117_000.jpg
comment:4 Changed 9 years ago by asm
Lidar Processing
Started.
comment:5 Changed 9 years ago by gej
Hyperspectral Processing
Started.
comment:6 Changed 9 years ago by dac
Sent notification email to PI with links to KML
comment:7 Changed 9 years ago by asm
Lidar Processing
Found pitch and roll consistent with all lines
Roll | -0.0010109 |
Pitch | -0.0022489 |
comment:8 Changed 9 years ago by asm
Lidar Processing
Classification started.
comment:9 Changed 9 years ago by asm
Lidar Processing
Classification for discrete lidar finished.
comment:10 Changed 9 years ago by asm
Lidar Processing
Classification for lidar full wave form started.
comment:11 Changed 9 years ago by asm
Lidar Processing
Two flightlines are bigger than 50gbs. Process is killed due to that, stgo is splitting the lines on the spare machine.
comment:12 Changed 9 years ago by gej
Hyperspectral Processing
found SCT values
Long flightlines, some had a varying SCT value, have currently gone for best throughout line, but if it looks like they may be too incorrect will change
Flightline | FENIX |
1 | 23.45 |
2 | 3.2 |
3 | 2.90 |
4 | 2.25 |
5 | 3.675 |
6 | 2.75 |
7 | 2.5 |
8 | 1.45 |
comment:13 Changed 9 years ago by asm
Lidar Processing
Classification for full wave form finished. Elevation difference measured, delivery created including Readme file. Ready for Delivery Check'.
comment:14 Changed 9 years ago by gej
Lidar Delivery Check
Read Me
Need to add N/S to UTM projection,
Data qaulity remarks needs reworkding. instead of:
"This might have as a result that the elevation difference on overlapping areas could be not precise",
try:
"This may have resulted in the elevation difference on overlapping areas being less precise"
Navigation
There are only 6 trj files, but 8 flightlines. Not sure if this is due to the flightlines being split or not, if it is should maybe make a comment about it somewhere.
Screenshots
Don't think anything can be done, but maybe make a comment in readme that the screenshots aren't of the best quality due to the length of the flightlines.
comment:15 Changed 9 years ago by dac
- Description modified (diff)
comment:16 Changed 9 years ago by asm
Lidar Processing
All comments has been changed to trj files and Readme.
Please, ready for re-check.
comment:17 Changed 9 years ago by gej
Lidar Delivery Check
All looks good now that changes have been made.
All trj files are now there and comments made in readme on screenshots, and previous comment reworded.
Will be delivered with Fenix Hyperspectral data
comment:18 Changed 9 years ago by gej
Lidar delivered to Jean-Louis Roujean via FTP on 12 October 2015
comment:19 Changed 9 years ago by gej
Fenix Processing
Started creating mapped using files with new calibration
comment:20 Changed 9 years ago by dac
Fenix
Hyperspectral lines are very large producing mapped files 0.5 - 1.5 TB. Emailed PI to check preferred method for splitting data.
comment:21 Changed 9 years ago by dac
Fenix
PI happy to receive data split by scanline.
comment:22 Changed 9 years ago by dac
Fenix
Started testing splitting by scanline by adding cal_args_extra = -lines 0 100 to config file.
comment:23 Changed 9 years ago by gej
Fenix Processing
Lines have now been split, using cal_args_extra. Split line functionality has been added to generate_apl_config.py for future projects.
There was a problem with this flight where the SCT values varied through the flight by about -0.4 for every 10000 scan lines.
This has been fixed by increasing the fps by 0.08 so I am now finding the new scts
comment:24 Changed 9 years ago by gej
Fenix Processing
Found new SCT values, SCT values are consistent through sections so have only put full line values.
Flightline | FENIX |
1 | 27.5 |
2 | 4.3 |
3 | 3.34 |
4 | 2.58 |
5 | 3.92 |
6 | 3.11 |
7 | 4.28 |
8 | 1.85 |
Number of sections per line:
Flightline | Number Sections |
1 | 11 |
2 | 3 |
3 | 4 |
4 | 9 |
5 | 1 |
6 | 1 |
7 | 5 |
8 | 1 |
total | 35 |
Files now being mapped with all bands.
comment:25 Changed 9 years ago by gej
Fenix Processing
Have split line 6 as final filesize was at 200GB.
Has been split into 3 sections. SCT hasn't changed.
comment:26 Changed 9 years ago by gej
Fenix processing
Mapped files processed, creating delivery.
comment:27 Changed 9 years ago by stgo
Fenix processing
Have taken over from gej for this week, some files have been included in the delivery creation which resulted in bad naming of all files. Irritating but fixable, if I can't identify the correct ones (should be easy, they will be the largest files) I will reprocess for simplicity's sake.
comment:28 Changed 9 years ago by stgo
Fenix processing
Have resubmitted to the grid, will recreate delivery once complete.
comment:29 Changed 9 years ago by stgo
All lines still had their dark frames included, have updated end lines to remove these. Also fixed the sct value for line 1 and will be recreating delivery this afternoon.
comment:30 Changed 9 years ago by stgo
Fenix processing
Created readme and placed in the delivery folder. A pre release version of the data has already been provided to the PI via ftp however the size of the delivery means it is not very feasible for them to download it on their own connection. Mapped files are already zipped.
Needs final DC before delivery.
comment:31 Changed 9 years ago by dac
Fenix Delivery Check
- Created ASCII view vectors and converted to Windows line endings.
- Added header to the table at the top of page 3
- Compared spectra to Py6S simulations - look good
Ready to deliver.
Need to remove old deliveries from processing/deliveries.
comment:32 Changed 9 years ago by dac
Archiving
Have started uploading delivered LiDAR data to NEODC - large delivery so will take a while.
comment:33 Changed 9 years ago by stgo
Fenix delivery
Fenix data sent on hard drive 10/12/2015
comment:34 Changed 9 years ago by dac
Archiving
Started archiving
comment:35 Changed 9 years ago by dac
Archiving
Finished upload to NEODC - Notified Wendy.
comment:36 Changed 9 years ago by dac
Delivery
Notification from PI: Received hard drive with Fenix data (22/12/2015)
comment:37 Changed 9 years ago by dac
LiDAR'
Started reprocessing with full range of scan angles.
comment:38 Changed 8 years ago by lah
OWL
found sct for sample flight line 2 = 3.93
Created delivery, ready for checking.
comment:39 Changed 8 years ago by asm
Owl Delivery Check
-check_apl_cmd only created the masked files but I tested the instructions written on readme and everthing looks good.
-proj_tidy do not recognise the owl delivery folder and list everything as unrecognised files. Looks fine however.
-There is only one flightline so both screenshots were the same. Removed mosaic.
-Tried to run py6S with the igm created from apl commands for a given pixel x,y (did not work for ndvi pixel finder). Did not work, error: Unexpected radiance units. May be worth implementing this on the future.
-Spectra looks OK for mapped file on Tuiview.
-As dac pointed out on other project: XML metadata files list Carl as the pilot and Tom as the instrument operator (different to logsheet). This is the same as the Fenix data so not worth blocking the project for but will need fixing in templates.
No major problems found. Owl should be ready to deliver.
comment:40 Changed 8 years ago by dac
Archiving
Processed data available from NEODC (http://browse.ceda.ac.uk/browse/neodc/arsf/2015/EUFAR15_18/EUFAR15_18-2015_174_Toulouse)
comment:41 Changed 8 years ago by dac
- Resolution set to fixed
- Status changed from new to closed
Archiving
Raw data also available from NEODC (http://browse.ceda.ac.uk/browse/neodc/arsf/raw_data/2015/EUFAR15_18-2015_174_Toulouse_raw).
Leaving ticket open for Owl processing (outside EUFAR)
comment:42 Changed 8 years ago by dac
- Resolution fixed deleted
- Status changed from closed to reopened
comment:43 Changed 8 years ago by lah
Owl
Uploaded sample delivery to NEODC.
comment:44 Changed 8 years ago by lah
Owl Processing
Found scts with a combination of wobble minimisation and alignment to fenix. There isn't much overlap between flight lines in this flight. The same sct is good for each split of an entire line, though there is some increased misalignment with the fenix, which does appear to wobble in these regions.
Flightline | OWL |
1 | 2.67 |
2 | 3.89 |
3 | 2.44 |
4 | 3.14 |
5 | 2.60 |
6 | 2.96 |
7 | 2.51 |
Now mapping full band files.
comment:45 Changed 8 years ago by lah
Owl Processing
Created delivery. Scripts deleted all level 1b split lines so recreating. Some of these appear to have been renamed as the full files but the number of lines between these and the mask files are inconsistent so reprocessing level 1 files.
comment:46 Changed 8 years ago by lah
Owl Delivery
Created readme from hyp_config. Manually edited owl sections and removed under/overflow table as there were none. Added in missing lines (files) to delivery content table.
Manually added Owl details to logsheet (FPS & IT).
comment:47 Changed 8 years ago by lah
Owl Delivery
Copied in new level 1b files and renamed. Ready to delivery check.
comment:48 Changed 8 years ago by asm
Owl Delivery Check
Started.
*No xml files for flightline 4.
*Screenshots are currently being generated and will recheck later.
*Project tidy does not like owl names but everything seems fine.
*Everything else looks fine, including checks for apl comands
Once xml for flightline 4 has been created and the screenshorts sorted out, should be ready to deliver.
comment:49 Changed 8 years ago by lah
Owl
Recreated xml files for line 4. These were missing because line 4 had no black body files so used line 5 calibration file. Manually added in software info.
New screenshots have finished. Have renamed these and removed old ones.
comment:50 Changed 8 years ago by asm
Owl Delivery Check
Everything looks fine now. Have removed the tmp_cmd_check and will zip mapped files. Once done, project will be ready to be delivered.
comment:51 Changed 8 years ago by asm
Owl Delivery Check
Mapped files have been zipped. Checked logfiles and everything looks correct. Will mark as Ready to be delivered.
comment:52 Changed 8 years ago by asm
OWL Delivered
By postal mail including a hard drive. Dispatched on 2nd December 2016 to PI. A notification email has been sent as well.
comment:53 Changed 8 years ago by lah
Sent hard disk to Philippe Crebassol with Owl data (and 274) on 24/01/17.
comment:54 Changed 7 years ago by dac
- Resolution set to fixed
- Status changed from reopened to closed
Archiving
Owl data available from CEDA. Closing ticket.
Navigation Processing complete
Basestation:
Lat: 43 38 13.01189
Long: 1 21 58.49814
Ell. Height: 199.663
Position Seperation: Slight spike to -0.06 towards end of flight line, rest below -0.04
Position Accuracy: Lat and long all within +-0.035 Height within +-0.05