Opened 9 years ago
Closed 7 years ago
#602 closed flight processing (fixed)
EUFAR15/48, flight day 249/2015, Hidhaz
Reported by: | asm | Owned by: | |
---|---|---|---|
Priority: | immediate | Milestone: | 2015 data processing completion |
Component: | Processing: general | Keywords: | |
Cc: | Other processors: |
Description (last modified by dac)
Data location: ~arsf/arsf_data/2015/flight_data/iceland/EUFAR15_58-2015_249_Hidhaz
Project acronym: HIDHAZ_N_ICELAND
Data arrived from ARSF via SATA disk 8 on 28/09/2015.
Priority: Unknown
PI: Armelle Decaulne
EUFAR code: E048
Sensors:
- Fenix (requested)
- Owl (requested)
- Leica LIDAR (requested)
- RCD (requested)
Change History (50)
comment:1 Changed 9 years ago by asm
- Milestone set to 2015 data processing completion
comment:2 Changed 9 years ago by asm
- Description modified (diff)
comment:3 Changed 9 years ago by dac
- Description modified (diff)
comment:4 Changed 9 years ago by asm
comment:5 Changed 9 years ago by asm
General Processing
Notification of ARSF data arrival has been sent to PI.
comment:6 Changed 9 years ago by asm
Navigation Processing
Started.
Needed to download base station information for this project. Used IpasTC to download basestation REYK
Basestation verification
Lat: 64 08 19.63049
Lon: -21 57 19.7659
Elipsoidal Height: 93.066 (metres)
Antenna Height: 0.232 (metres)
Could not use Ipas Pro, used best solution from Ipas TC with downloaded basestation.
Settings:
-Switched data to static (default kinematic)
-Computed Position via P3 using Ipas TC
-Needed to specify antena name Leiar25.r4
-Re-calculated all positioning using C/A code
-Elevation Mask: 13.5 degrees
-L2C phase correction: 0
Solution has value 1 on ambiguity fix.
Navigation processing finished.
comment:7 Changed 9 years ago by dac
Marking as blocked - still waiting on GPS data from IMO.
comment:8 Changed 9 years ago by asm
Navigation processing
Navigation reprocessed with GPS data provided by Icelandic Met-office.
Basestation verification for SIFJ:
Latitude 66 08 16.94583
Lon -18 53 57.66368
Ell Height 81.499
Ant Height 1.098
Which corresponds on decimal degrees to the values sent by Met-office: -18.89935061 66.13804033 81.49616202
Basestation verification for AKUR:
Latitude 65 41 07.53592
Lon -18 07 20.93094
Ell Height 134.169
Ant Height 0.165 m, to L1-PC (TRM29659.00, MeasDist 0.055)
Also match data recieved: -18.12248256 65.68542672 134.20199055
Best solution, only using both basestation. Copying files and have renamed old basestation and ipas_honeywell to old_nav_basestation (Please clean after processing, before archiving).
comment:9 Changed 9 years ago by stgo
Lidar processing
started processing
comment:10 Changed 9 years ago by asm
RCD processing
Started. Converting raw files to tif.
comment:11 Changed 9 years ago by dac
Fenix Processing
Started processing - file 1 (not on logsheet) moved to 'bad_lines'
comment:12 Changed 9 years ago by stgo
Lidar processing
Pitch and roll errors seem consistent across all lines. Currently classifying.
pitch | -0.00076 |
roll | -0.00266 |
comment:13 Changed 9 years ago by stgo
Lidar processing
There is only one file of waveform data which appears to be corrupt. There will be no waveform in the lidar delivery.
comment:14 Changed 9 years ago by asm
RCD processing
Tagging pictures.
comment:15 Changed 9 years ago by stgo
Lidar processing
There are two sites, 122822 - 134338 and 135102 - 151115.
122822 - 134338
pitch | -0.00256 |
roll | -0.00066 |
135102 - 151115
pitch | -0.00076 |
roll | -0.00266 |
Recreating delivery
comment:16 Changed 9 years ago by dac
Found SCT values using autoSCT. A couple of lines were out of range and needed to be calculated manually. All lines match well against each other and LiDAR intensity data.
Flightline | SCT |
2 | 0.99 |
3 | 1.01 |
4 | 1.01 |
5 | 1.01 |
6 | 1.03 |
7 | 1.03 |
8 | 1.02 |
9 | 1.08 |
10 | 1.05 |
11 | -0.05 |
12 | 1.00 |
13 | 1.01 |
14 | 1.01 |
15 | 0.99 |
16 | -0.06 |
17 | 1.00 |
18 | 0.99 |
19 | 0.97 |
20 | 0.99 |
21 | 1.09 |
22 | 0.99 |
23 | 0.99 |
24 | 1.0 |
25 | 0.04 |
26 | 0.99 |
27 | 1.05 |
28 | 1.0 |
29 | 0.99 |
comment:17 Changed 9 years ago by asm
RCD
Delivery created. Readme file includes this explanation: The RCD camera is set automatically to an specific gain value depending on light conditions for the recording area. Photos taken over snow areas are brighter than standards but have been included on this delivery if they were not overexposed as they have usable data.
This effect is clearly visible in many pictures but specially on 101 (a bit overexposed) that still have been included as it may be useful (closest pictures were not usable and this one covers that area showing snow).
Ready for delivery check.
comment:18 Changed 9 years ago by stgo
Lidar processing
The first site seems to suffer from range gate issues due to the data being outside the minimum range gate set on the instrument. This can be seen in the large data holes and tapering of the flightlines.
The second site suffers from quite a lot of noisey/hazey points in the first lines. This has been classified as needed.
comment:19 Changed 9 years ago by dac
Fenix Processing
Created delivery - ready for delivery check
comment:20 Changed 9 years ago by stgo
Lidar processing
Created lidar delivery. Ready for check.
comment:21 Changed 9 years ago by asm
Hyperspectral Delivery Check
All checks done, only a few minor comments.
-I suggest to add a note on readme about data quality for fenix saturation over snow. Also would be useful a note explaining that the first flightline is named 02 and not 01.
-Spectra checked with py6s. Looks fine except for some peaks a bit higher than usual but within expecting limits on the overlapping region and they will be masked out.
-Checked apl commands, will remove folder once ready to deliver.
-Have removed last page from logsheet about Eagle and Fenix. Also changed "*_cross" flightline for two different lines for avoiding overwriting next cell.
comment:22 follow-up: ↓ 23 Changed 9 years ago by lah
RCD DC
- I've rephrased the readme text, as many of the snowy images are overexposed in the snow regions, but are still useful:
"The RCD camera is set automatically to a specific gain value depending on light conditions for
the entire recording area. Some photos have snow pixels that appear overexposed, but have been
included if they have usable data."
Everything else is fine, so ready for delivery.
comment:23 in reply to: ↑ 22 ; follow-up: ↓ 25 Changed 9 years ago by dac
Replying to lah:
"The RCD camera is set automatically to a specific gain value depending on light conditions for
the entire recording area.
It will be exposure rather than gain for a camera. In reality just exposure time as the aperture is fixed.
comment:24 Changed 9 years ago by dac
Fenix
I suggest to add a note on readme about data quality for fenix saturation over snow.
Have added:
"Pixels over bright snow-covered areas are overexposed in some wavelengths. These bands have been masked out in the level 3b files."
Also would be useful a note explaining that the first flightline is named 02 and not 01.
Have added:
"Note file numbering starts at 2 as the first file was saved prior to acquisition and contains no usable data."
Started zipping mapped files.
If asm is happy will changes to readme will mark as ready to deliver.
comment:25 in reply to: ↑ 23 Changed 9 years ago by lah
Replying to dac:
Replying to lah:
"The RCD camera is set automatically to a specific gain value depending on light conditions for
the entire recording area.
It will be exposure rather than gain for a camera. In reality just exposure time as the aperture is fixed.
changed gain value to exposure time in readme
comment:26 Changed 9 years ago by asm
Hyperspectral Delivery Check
Happy with the changes made on readme. Mapped files have been zipped and everything is correct on their logfiles. Removed an aux file from the mapped directory. Fenix is ready to deliver.
comment:27 Changed 9 years ago by dac
Delivery
To be delivered via hard drive to Costanza Morino - waiting on LiDAR. Have uploaded Fenix and RCD to FTP server (arsf14) for PI and other collaborators.
Also copy VM for training course to hard drive.
comment:28 Changed 9 years ago by lah
Lidar DC
- Roll & pitch look fine
- Classification fine
- changed las1.0 to 1.2
- removed eagle & hawk pages from logsheet (actually copied from HS delivery, then removed fodis folder from delivery!)
- changed the readme name to exclude _01
- removed extra sol file
- demcompare:
Min: -105.999249023437
Max: 128.388448730469
Sum: -2256662.91263792
Mean: -7.19989443460396
Median: -6.14898
Absolute mean: 10.1892045178341
Std deviation: 11.7393376564784
Total cells: 4776398
Non-null cells: 313430
- only problem is in readme - you've spelt noisy and hazy wrong. Apart from those typos it's ready for delivery
comment:29 Changed 9 years ago by stgo
Lidar delivery
Fixed readme, always get confused by ey's!
comment:30 Changed 9 years ago by dac
Delivery and Archiving
Data copied to hard drive ready to deliver tomorrow.
Started uploading processed data to NEODC.
comment:31 Changed 9 years ago by dac
Delivery
Sent on hard drive to Costanza Morino (Open University) 04/03/2016
Processed data uploaded to NEODC. Raw Owl data needs to be unpacked before raw data are uploaded.
comment:32 Changed 9 years ago by lah
OWL
Found sct = -2.78 for Owl line 007 (identified from time on Owl folder name). Only unpacked (copied and renamed) this line as only processing a single line for now. GPS timestamps copied from Fenix data, and used T1 calibration file for dark frames.
Now mapping all bands.
comment:33 Changed 9 years ago by dac
Archiving
Processed Camera, LiDAR and Fenix data have been archived to http://browse.ceda.ac.uk/browse/neodc/arsf/2015/EUFAR15_48/EUFAR15_48-2015_249_Hidhaz
comment:34 Changed 9 years ago by lah
OWL
Created delivery with 1 line. Strangely, the level 1b was overwritten during delivery creation with a mask file of more lines than the created mask file, so no idea what happened there. Have recreated the level 1b file again and copied across.
Again, proj_inf xml is not created. I believe this is because we decided to stop it being created automatically if there are missing items (calibration stuff for owl?).
Also ascii FOV vectors failed.
comment:35 Changed 9 years ago by lah
OWL
Reprocessed with corrected processing chain as file sizes were incorrect and couldn't run autoqc.
Created new complete delivery. Ready for checking.
comment:36 Changed 9 years ago by dac
Owl delivery check
Looks fine apart from would change figure caption in readme from 'mosaic' as it is only a single line. Also can remove mosaic screenshot.
Other than that ready to go.
comment:37 Changed 9 years ago by lah
OWL
Made suggested changes and zipped mapped file.
comment:38 Changed 8 years ago by lah
Owl delivery uploaded to ftp (arsf 14) and email sent to Susan Conway (requested data).
comment:39 Changed 8 years ago by lah
Owl renaming
- Ran new script to rename original raw files.
- Overwrote line 9 with modified raw data used to process sample data (labelled line 7). Calibration data used to process this line now named 1404.
- There is still an originally named folder with no data in which can probably be deleted.
comment:40 Changed 8 years ago by lah
Owl Processing
Appended T1 files acquired immediately after each flight line to raw data for lines 1-16 (requested South site) to allow processing with Specim's tool (no dark frames for this flight).
comment:41 Changed 8 years ago by lah
Owl Processing
Have processed to level 1b (radiometric calibration) and checked files can be mapped (Had to add gps stop times to raw header files). Apart from a few lines being outside the dem area and the sct offsets very large, everything look ok.
Will wait until finances approved before proceeding with manual sct corrections. (sct offsets too large for automated scripts to speed up process)
comment:42 Changed 8 years ago by dac
Archiving
Archiving of processed data on-hold until Owl processing is complete.
comment:43 Changed 8 years ago by lah
Owl Processing
Found scts. Still some across track deviation between different flightlines. Lines 4 and 8 have not been included as they are failed lines (errors mentioned in the logsheet). Large offset is due to GPS timestamp being automatically added during unpacking.
Flightline | OWL |
1 | -3767.15 |
2 | -3758.59 |
3 | -3763.04 |
5 | -3749.18 |
6 | -3759.54 |
7 | -3749.80 |
9 | -2.86 |
10 | -3755.24 |
11 | -3757.75 |
12 | -3703.51 |
13 | -3760.89 |
14 | -3740.81 |
15 | -3772.50 |
16 | -3749.00 |
17 | -3819.20 |
comment:44 Changed 8 years ago by lah
Owl Processing
Delivery created and ready for checking. Added comment to end of page 2 in the readme that only the South site data is included.
comment:45 Changed 8 years ago by mark1
Owl delivery check
Have all level1 files but:
- No XML files for lines: 04, 08
- No navigation for lines: 04 08
- No mapped for lines: 04 08
- No screenshots for lines: 04 08
Level 1 files for 04 and 08 are small.
No Owl information in the logsheet - we should enter FPS and integration time (manually if need be).
Table in read me does not list the matching owl line names.
Will check example commands and view data next
comment:46 Changed 8 years ago by lah
Owl delivery
Have removed level 1b lines 4 and 8 as they shouldn't be there (no usable data). This is probably what messed up the readme table. Have added in the lines manually and made a comment about lines 4 and 8 not being there, but all lines being captured in subsequent files.
Looks like missing Owl info in the logsheet is a recurring problem so probably needs the logsheet generation altering. I'll raise an issue, but might not be a quick fix, so manual input might be best for this project.
comment:47 Changed 8 years ago by mark1
Owl delivery check
Have tested all level1b and level3 images and they open and show good data.
All example commands worked-
comment:48 Changed 8 years ago by dac
Owl
Manually added Owl information to logsheet. Updated readme to NERC-ARF (with note facility was NERC ARSF when data were flown).
Zipped mapped files.
Delivered via FTP (arsf14).
comment:49 Changed 8 years ago by lah
Owl Archiving
Uploaded Owl delivery to NEODC 21/09/16.
comment:50 Changed 7 years ago by asm
- Resolution set to fixed
- Status changed from new to closed
Archiving
Archived and it everything available from CEDA. Processed data at http://browse.ceda.ac.uk/browse/neodc/arsf/2015/EUFAR15_48/EUFAR15_48-2015_249_Hidhaz
Plymouth 12th May 2017
General Processing
-Logsheet generated.
-Send a copy of the notifications to: Dr. Susan Conway (s.j.conway.00@…)
-Landing and taking off times are approximate.