#595 closed flight processing (fixed)
RANNIS15/28, flight day 241a/2015, Hekla.
Reported by: | asm | Owned by: | |
---|---|---|---|
Priority: | immediate | Milestone: | 2015 data processing completion |
Component: | Processing: general | Keywords: | |
Cc: | Other processors: |
Description (last modified by dac)
Data location: ~arsf/arsf_data/2015/flight_data/iceland/RANNIS15_28-2015_241a_Hekla
Data arrived from ARSF via SATA disk 8 on 28/09/2015.
Priority: Unknown
PI: Gro Pedersen
CAFAM code: R028
Sensors:
- Fenix (requested)
- Owl (requested - currently unfunded)
- Leica LIDAR (requested)
- RCD (requested)
Change History (90)
comment:1 Changed 9 years ago by dac
- Milestone set to 2015 data processing completion
comment:2 Changed 9 years ago by asm
- Description modified (diff)
comment:3 Changed 9 years ago by dac
- Description modified (diff)
comment:4 Changed 9 years ago by asm
comment:5 Changed 9 years ago by asm
General Processing
Sent to PI ARSF data arrival notification for RANNIS15/28.
comment:6 Changed 9 years ago by asm
Navigation Processing,
Started. Needed to download base station information for this project. Used REYK
comment:7 Changed 9 years ago by asm
Basestation verification
Needed to download base station information for this project. Used REYK
Lat: 64 08 19.63063
Lon: -21 57 19.7628
Elipsoidal Height: 93.223 (metres)
Antenna Height: 0.057 (metres)
comment:8 Changed 9 years ago by asm
Navigation Processing
Could not find a suitable solution on Ipas TC. Will try using Ips Pro.
comment:9 Changed 9 years ago by gej
RCD processing
Have produced tif files, waiting for navigation.
comment:10 Changed 9 years ago by asm
Navigation Processing
Navigation was done using Ipas Pro. It is located on a different folder on ipas_honeywell since it will be tested during processing chain (specially Lidar). If no further problems appear, navigation is finished.
comment:11 Changed 9 years ago by lah
Lidar Processing
Processed with 20150829_090311_ipaspro.sol. No obvious problems yet.
comment:12 Changed 9 years ago by lah
Lidar Processing
Found pitch and roll. Seems stable for all lines, though later lines don't have much overlap. No buildings, so had to use rock formations.
Roll = -0.0007
Pitch = -0.0025
comment:13 Changed 9 years ago by lah
Lidar Processing
All classified. Volcanic ash cloud required substantial manual classification.
comment:14 Changed 9 years ago by lah
Lidar Processing
Delivery made. Ready for checking. Alspp would not process first line, so it is omitted. FW is included even though not requested.
A screenshot from this flight has been uploaded to twitter with PIs approval.
comment:15 Changed 9 years ago by asm
Lidar DC
Started
-Readme is not present at delivery but was created and had a look at it on processing directory. Flightline 028 recorded in logsheet is not on delivery, a note should be added to the readme.
-proj_tidy is complaning about every single file name but I can't see anything wrong, I suspect it is caused by a longer than usual project_code.
comment:16 follow-up: ↓ 17 Changed 9 years ago by lah
Lidar
Have moved readme into delivery. Missing line 28 is mentioned at the end of page 2 with the line info. Should it be somewhere else? Not sure why alspp wouldn't process it though.
Pretty sure I previously ran proj tidy before the delivery and didn't get any naming problems. Looks like RANNIS is to long for the regex (2-5 chars allowed), so fails everything in the delivery directory.
comment:17 in reply to: ↑ 16 Changed 9 years ago by asm
Replying to lah:
Lidar
Have moved readme into delivery. Missing line 28 is mentioned at the end of page 2 with the line info. Should it be somewhere else? Not sure why alspp wouldn't process it though.
Everything looks good now. My mistake, I did not see those additional lines before, I usually write it on the data quality section, that is why I've missed it but I think is better this way.
comment:18 Changed 9 years ago by asm
Lidar DC
Navigation is currently being re-doing after some issues found on Ipas Pro. Will check again if line 28 could be processed with the new navigation.
comment:19 Changed 9 years ago by stgo
Navigation processing
I've reprocessed in Ipas TC with ppp, separation of 2 cm so will be using this over the Ipas pro solution. Gej is copying off the windows machine now.
comment:20 Changed 9 years ago by gej
Navigation Processing
Files have been copied to posatt directory, in ipas_Honeywell.
RCD processing
Event file recreated with new sol files. No Problems anymore (was previously giving nav values of 0 for the first 50 images).
Tagging now.
comment:21 Changed 9 years ago by stgo
Navigation processing
That suggests the Lidar file that wouldn't process should be fine now
comment:22 Changed 9 years ago by asm
Fenix Processing
Started. Generated config file and found that Owl header files are missing all starting and ending GPS times and points. We will have to discuss how to process the Owl data.
comment:23 Changed 9 years ago by lah
Lidar Processing
Line 028 has been processed with new sol and aligns well with adjacent line. Is now classified, so just needs adding to delivery.
Should we include both sol files in delivery?
comment:24 Changed 9 years ago by stgo
Lidar Processing
The PPP result is considerably better quality than the differential so I don't think we should be sending the differential. It also doesn't cover the full survey site (hence the problems processing the first lidar line), have the other lines been reprocessed with the new nav file?
comment:25 Changed 9 years ago by lah
Lidar Processing
Needs reprocessing as original nav quality not good enough.
comment:26 Changed 9 years ago by asm
Fenix Processing
Found Sct values.
Flightline | FENIX |
1 | 0.88 |
2 | 1.10 |
3 | 0.91 |
4 | 1.08 |
5 | 0.94 |
6 | 1.03 |
7 | 0.93 |
8 | 1.04 |
9 | 1.02 |
10 | 1.03 |
11 | 0.94 |
12 | 1.01 |
13 | 1.04 |
14 | 0.96 |
15 | 0.97 |
16 | 0.98 |
17 | 1.04 |
18 | 1.01 |
Will split lines since they are fairly long.
comment:27 Changed 9 years ago by lah
Lidar Processing
Have reprocessed data with new sol file, copied in classifications and made new delivery
Ready for delivery check.
comment:28 Changed 9 years ago by gej
RCD ready for delivery check
comment:29 Changed 9 years ago by asm
Fenix Processing
Fenix delivery created included readme file. Flightlines have been split and are still around 40Gb a note should be added to the notification mail in case they are having problems with them. Ready for Delivery Check.
comment:30 Changed 9 years ago by dac
Fenix Delivery Check
Starting delivery check
comment:31 Changed 9 years ago by asm
Fenix Processing
Notice that the script for creating screenshots is failing to name split flightlines and were not named according to its part. Have solved the issue manually.
comment:32 Changed 9 years ago by dac
Fenix Delivery Check
- Missing ASCII view vectors
- On page 7 of readme undefined LaTeX reference (see ?? section)
- Config file / APL commands contain the wrong name for the DEM
- Screenshots still have SCT values in name.
- proj_tidy reports 100s of errors (likely due to name)
- Checked spectral with Py6S - look fine.
- Logsheet has Al HOwlad as the pilot - suggest fix capitalisation
- Part numbering starts at 0 not 1 - could cause confusion. This is the same as 174 but I hadn't realised the script did this - starting at 1 might be better.
- Need to remove fodis directory.
Other than this all look OK.
comment:33 Changed 9 years ago by asm
Fenix Processing
All issues have been checked. A couple of comments:
-Readme undefined LaTeX reference - It was referenced before being defined, it took me sometime to realise that nothing was wrong and it had to be compiled at least twice.
-Missing ASCII view vectors - The file "fenix_fov_fullccd_vectors.bil" is there; please, let me know if is missing something else.
comment:34 Changed 9 years ago by dac
Fenix
Also need ASCII version of view vectors. Created using get_ascii_fov_for_level1.py the script wasn't finding the files from the directory so had to specify an individual line.
check_apl_cmd works now and generated files look fine.
Ready to deliver
comment:35 Changed 9 years ago by dac
Fenix Delivery
Started zipping mapped files.
comment:36 Changed 9 years ago by dac
RCD Delivery Check
Starting delivery check
comment:37 Changed 9 years ago by dac
RCD Delivery Check
No Read me. Everything else looks OK. Will continue when readme is present.
comment:38 Changed 9 years ago by dac
LiDAR Delivery Check
Starting delivery check.
comment:39 Changed 9 years ago by gej
RCD Delivery
Have generated readme and moved to delivery directory.
Ready to carry on DC.
comment:40 Changed 9 years ago by dac
LiDAR Delivery Check
Couple of minor points found and fixed:
- In overlap table file number not line name used (have fixed)
- The pilots name is wrong in the logsheet - it should be Al Howland not Al Howlad (I think). Have corrected and in hyperspectral delivery as well.
- demcompare.py wouldn't run. Checked patched DEM instead, LiDAR and ASTER line up well.
Ready to deliver.
comment:41 Changed 9 years ago by dac
RCD Delivery
Readme look fine. Ready to deliver.
comment:42 Changed 9 years ago by dac
Delivery
Send RCD, Fenix and LiDAR to Gro Pederson (University of Iceland) on a hard drive (21/12/2015). Also placed on FTP server (arsf4) for early access.
comment:43 Changed 9 years ago by lah
OWL
Used this flight for the missing dark frame tests and example imagery for the Owl report. Only used 5 lines, so project still requires proper unpacking. scts found are in table below, based on GPS times in Fenix headers (owl and fenix line numbers are different due to the way data has been stored - asm is working on automated unpacking to deal with this):
Flightline (Fenix) | Flightline (Owl) | OWL |
2 | 5 | 501.83 |
3 | 8 | -3.08 |
4 | 10 | -2.98 |
5 | 12 | -3.10 |
8 | 18 | 9.35 |
Owl line 5 is much shorter than fenix line 2 so probably failed to start at the same time and explains the large sct.
comment:44 Changed 9 years ago by asm
Navigation processing
Navigation reprocessed with GPS data provided by Icelandic Met-office to compare data quality.
Basestation verification for FEDG:
Latitude 64 01 30.93075
Lon -19 41 22.05467
Ell Height 503.758
Ant Height 1.00 m, to L1-PC (TRM411249.00, MeasDist 0.929)
Which corresponds on decimal degrees to the values sent by Met-office: -19.68946006 64.02525821 503.73149445
Basestation verification for HESA:
Latitude 64 02 48.62706
Lon -19 33 38.08790
Ell Height 529.112
Ant Height 1.090 m, to L1-PC (TRM411249.00, MeasDist 1.019)
Also match data recieved: -19.56057992 64.04684073 529.08620580 HESA deltaz=0.244
Basestation verification for MJSK:
Latitude 63 55 58.59960
Lon -19 40 20.52995
Ell Height 770.561
Ant Height 1.050 m, to L1-PC (TRM411249.00, MeasDist 0.979)
Best solution found using FEDG and HESA. Copied files and renamed old basestation and ipas_honeywell to old_nav_basestation (Please clean after processing, before archiving).
comment:45 Changed 9 years ago by asm
RCD Processing
Tagging pictures with the new navigation. There are two different ImageEvents1 and PhotoId1, the correct one is:
150829-013924
comment:46 Changed 9 years ago by asm
RCD Processing
Have created a new delivery folder with the new event file. Tagging all files again. New event file has 8 pictures out of bonds that will be removed on this delivery.
comment:47 Changed 9 years ago by asm
RCD Processing
Finished tagging, creating delivery. Noticed that 8 pictures outside bonds where also outside bonds in the first event and therefore there is no problem.
comment:48 Changed 9 years ago by asm
RCD Processing
Delivery created and readme file added. Ready for Delivery Check.
comment:49 Changed 9 years ago by gej
RCD delivery check
No Problems with delivery,
Ready to deliver
comment:50 Changed 9 years ago by asm
Delivery
RCD data dispatched via hard drive on 23/02/2016.
comment:51 Changed 9 years ago by lah
Hyperspectral reprocessing
Started reprocessing with new navigation. There were a lot of files still in the hyperspectral processing directory, dated after the previous delivery so I've temporarily moved them to hyperspectral_old as I don't know what they're for. Please delete or move elsewhere if still in use.
comment:52 Changed 9 years ago by lah
Lidar reprocessing
Started reprocessing lidar with new nav. Also corrected problems from boresight file which were used in previous delivery:
- Filter range bank B deselected
- max angle set to 45 instead of 7
comment:53 Changed 9 years ago by dac
Hyperspectral
The hyperspectral directory lah moved to 'hyperspectral_old' contained gej's tests comparing the old and new navigation data. No longer needed so have removed.
comment:54 Changed 9 years ago by lah
Lidar reprocessing
New files created. Reclassifying as more points than previously.
comment:55 Changed 9 years ago by lah
Hyperspectral reprocessing
New delivery created with new nav and old scts. Ready for DC.
comment:56 Changed 9 years ago by lah
Lidar reprocessing
New delivery created with new nav and new manual classification. Quickly checked the first few elevation differences and they still seem reasonable.
Ready for DC
comment:57 Changed 9 years ago by stgo
Lidar delivery check
comment:58 Changed 9 years ago by asm
Hyperspectral Delivery Check
Everything looks fine.
Spectra was checked using Py6s and it could not always find vegetation pixel but spectra is OK.
Will zip files and should be ready for delivery tomorrow morning.
comment:59 follow-up: ↓ 60 Changed 9 years ago by stgo
Lidar delivery check
Everything looks fine, waiting on demcompare to confirm.
Proj tidy really hates the hyperspectral delivery, we should probably update it to accept the "part_xxx" files
comment:60 in reply to: ↑ 59 Changed 9 years ago by asm
Replying to stgo:
Lidar delivery check
Everything looks fine, waiting on demcompare to confirm.
Proj tidy really hates the hyperspectral delivery, we should probably update it to accept the "part_xxx" files
And larger project names.
comment:61 Changed 9 years ago by stgo
Lidar delivery check
demcompare output:
Min: -759.149876953125
Max: 78.0656688232422
Sum: -3121742.19369611
Mean: -7.44021286604393
Median: -7.57575
Absolute mean: 9.56823590658078
Std deviation: 11.5387215208943
Total cells: 6486792
Non-null cells: 419577
Mean height difference looks fine for the masked area, ready for delivery.
comment:62 Changed 9 years ago by asm
Hyperspectral Delivery Check
Mapped files have been zipped. Project is ready to go.
comment:63 Changed 9 years ago by asm
Delivery
Fenix and LiDAR reprocessed with new navigation dispatched via hard drive on 03/03/2016. No more reprocessing pending.
comment:64 Changed 9 years ago by lah
OWL Reprocessing
Started reprocessing 5 flightlines with improved navigation data using existing scts. Manually renamed raw files as previous copies had been deleted and added GPS times into renamed raw header files.
comment:65 Changed 9 years ago by lah
OWL
Created a partial delivery with the 5 flightlines. A couple of things have failed (project info & ascii FOV vectors), but may not be required for sample data.
The duplicated raw files need deleting when this is 'delivered'.
comment:66 Changed 8 years ago by lah
OWL
Forced creation of ascii FOV vectors by bypassing getting sensor from level 1 header ID (lines 276 - 286 of get_ascii_fov_for_level1.py). Should only be xml info missing now.
comment:67 Changed 8 years ago by lah
OWL
Alas, the delivery scripts overwrote the level 1 data with the mask data so it will have to be reprocessed.
comment:68 Changed 8 years ago by lah
OWL
Created new delivery with all problems resolved. Ready for a delivery check.
comment:69 Changed 8 years ago by dac
Owl delivery check
Starting delivery check.
comment:70 Changed 8 years ago by dac
Owl delivery check
Ran through procedure for hyperspectral data and ran into the following issues:
- Wasn't sure if check_apl_cmd works so didn't run this. Did test APL commands from readme and worked OK though.
- Proj_tidy has problems with project name so didn't run
- 1b files are very faint in fastQC and produce error as sensor isn't recognised. Look OK in TuiView though.
Only problem found is the XML metadata files list Carl as the pilot and Tom as the instrument operator (different to logsheet). This is the same as the Fenix data so not worth blocking the project for but will need fixing in templates.
Ready to deliver.
comment:71 Changed 8 years ago by lah
Owl Delivery
- Changed readme GDAL translate command to use owl bands and a size within flightline.
- Also added file descriptions to delivery contents table
Only other difference is that metadata files don't have processing chain / software info since apl was run only in meta stage to create the xml files after data were already processed.
comment:72 Changed 8 years ago by lah
OWL Delivery
Uploaded to ftp arsf4 on 7/4/16 and notification sent to Gro, Susan and Graham.
Dispatched hard disk to Gro on 11/4/16.
comment:73 Changed 8 years ago by dac
RCD Processing
Reprocessed RCD images to locate missing files.
comment:74 Changed 8 years ago by dac
RCD Processing
Looks like images were removed due to overexposure and a red tint (not recorded in ticket). Will work out out which files were previously missing and generate new delivery with these.
comment:75 Changed 8 years ago by asm
RCD Processing
The red tint was caused by a problem with calibration. Once solved revealed pictures were not delivered due to overexposure only (still not recorded on ticket).
Delivered on 12/May/2016 using ftp 4
comment:76 Changed 8 years ago by lah
OWL renaming
- Ran new renaming script on original owl raw files
- Replaced some of these with the modified raw files used in processing (which have now been manually renamed to the correct number)
- Identified T1 & T2 used in processing by new name before deleting copy
Processed line | New raw file | Calibration file used in processing |
5 | 1 | 1113 |
8 | 3 | 1120 |
10 | 4 | 1133 |
12 | 5 | 1146 |
18 | 8 | 1225 |
i.e. raw files 1,3,4,5, and 8 have been modified, all others have been automatically timestamped but have no dark frames appended.
comment:77 Changed 6 years ago by dac
Archiving
Data uploaded to CEDA at request of PI.
comment:78 Changed 6 years ago by dac
- Resolution set to fixed
- Status changed from new to closed
Archiving
Data archived at CEDA. Closing ticket.
comment:79 Changed 5 years ago by wja
Owl Processing
Starting Owl processing.
comment:80 Changed 5 years ago by wja
Owl Processing
- Calibration files are in separate directories to line data. These have been symlinked to the nearest flightline capture directory (by timestamps) and a config file has been created to match each flightline to a calibration file.
- Line 5 has no timestamp, it has been matched to the calibration file for flightline 4.
- Darkframes missing on lines 2, 6, 7, 9, 10, 11, 12 and no separate darkframe files are present. These flightlines have been stitched to their closest calibration file. Flightlines which do not need stitching have been symlinked into the stitched directory for processing.
Level1b data processed.
The start and end times from hardcopy logsheet have been added to the stitched headers where they are missing (lines 2, 6, 7, 9, 10, 11 & 12).
Error being thrown regarding the timestamp of flightline 2 when trying to generate processing config file. Investigating.
comment:81 Changed 5 years ago by wja
Owl Processing
Config generation has not been successful with lines that have been stitched (2, 6, 7, 9, 10, 11, 12).
Have noticed that the start times of stiched headers were one hour ahead, which I have corrected. However, the following needs to added at the bottom of stitched headers manually.
GPS Stop Time = UTC TIME: hh:mm:ss.ssss
comment:82 Changed 5 years ago by wja
Owl Processing
Stop times calculated using owl_hdr_fix.py.
Flightline 2 has been omitted from further processing as it is considerably short (only 669 lines).
SCT Values found:
Flightline | SCT |
1 | 503.80 |
2 | N/A |
3 | -1.12 |
4 | -0.96 |
5 | -1.10 |
6 | 2.08 |
7 | 2.44 |
8 | 11.25 |
9 | 3.20 |
10 | 2.60 |
11 | 2.44 |
12 | 2.96 |
XMLs are being produced by processing full lines and changing the 'sensorid' in the level1b headers to match that found in the hyperspectral parameters CSV.
Need to process split lines for delivery.
comment:83 Changed 5 years ago by wja
Owl Processing
Reprocessed to level3 with the closest calibration times for each lines using the flightline times from owl_hdr_fix.py and accounting for the calibration file times being 1 hour ahead:
Line | Cal File |
1 | T1108 |
2 | T1120 |
3 | T1120 |
4 | T1133 |
5 | T1146 |
6 | T1200 |
7 | T1212 |
8 | T1225 |
9 | T1239 |
10 | T1252 |
11 | T1305 |
12 | T1318 |
All lines apart from line 1 have been split into 3 parts to be similar to the delivered hyperspectral data.
Ready for Delivery Check
comment:84 Changed 5 years ago by asm
Owl Delivery Check
Started.
comment:85 Changed 5 years ago by asm
Owl Delivery Check
-Renamed screenshots from "mapped_utm27n.bil.jpg" to ".jpg"
-Screenshots for overlapping parts look a bit odd due to the stretch but that shouldn't be aproblem
-Realised that Fenix line 02 overlaps with owl line 01 (a bit inconvenient but this is OK). Then owl line 02 is not delivered and line 03 overlaps with fenix 03 (and so on). This format is consistent with other deliveries.
-APL commands work fine.
-All others checks were made with positive results
Mapped files have been zipped and will mark project as ready to be delivered
comment:86 Changed 5 years ago by wja
Owl Delivered
Data placed on FTP slot 11.
Notification e-mail sent.
Delivery finalised.
comment:87 Changed 5 years ago by wja
Archiving Owl
Started archiving.
comment:88 Changed 5 years ago by wja
Owl Archiving
Added to PostGIS.
Starting rsync to CEDA (15:18 31/05/2019).
comment:89 Changed 5 years ago by wja
Owl Archiving
rsync to CEDA complete (3/6/19).
E-mail sent to confirm it has been received.
comment:90 Changed 5 years ago by wja
Owl Archiving
CEDA has confirmed recepit of the reprocessed Owl data.
All data is now archived.
General Processing
-Logsheet generated.
-Landing and taking off times are approximate.