Opened 7 years ago

Last modified 6 years ago

#624 new flight processing

ES17/106, flight day 168/2017, Valencia

Reported by: asm Owned by:
Priority: immediate Milestone:
Component: Processing: general Keywords:
Cc: Other processors:

Description

Data location: ~arsf/arsf_data/2017/flight_data/spain/ES17_106-2017_168_Valencia

Data arrived from NERC-ARF via hard drive 2017-06-30

Scientific objective: Validation of vegetation products derived from Sentinel-2 and -3.
Priority: Urgent

PI: Jadunandan Dash

Sensors:

  • Fenix (flown, requested)
  • Leica LIDAR (flown, requested, failed during acquisition)
  • OWL (flown, not requested)
  • PhaseOne (flown, not requested)

Change History (33)

comment:1 Changed 7 years ago by asm

Unpacking

Project has been unpacked. LiDAR failed and data is not usable for this flight season. There are 22 fenix flightlines and 22 owl lines.

Have run unpacked checks:
-Less than usual timestamps for owl nav file 22
-All owl last timestamps are recorded sooner than expected (most of them in line/frame number 9901), some are also incorrect and nav file 1 has the last timestap in line 0.
-Most of owl files have 1 to 4 dropped frames. There is one line with 11 dropped frames.
-2 Empty files: RawLaser/170617_100601/LDR170617_100601_0000000019.scn and RawLaser/170617_085920/LDR170617_085920_0000000018.scn

comment:2 Changed 7 years ago by dac

Gary provided the height of the basestation from floor to marker (where tape measure slots in) as approximately 1.23 m (from memory). The distance from this to antenna centre is 0.36 m (from previous log sheet) so antenna height to be used for processing is 1.59 m.

comment:3 Changed 7 years ago by asm

Ipas20 data was named as '167-17_dash.dat' and have renamed to '168-17_dash.dat' as the data is indeed correct for that day.

comment:4 Changed 7 years ago by asm

There were too many days in the basestation data. Used the the lastest survey (corresponds with the data):

teqc -st 170617070517 -e 170617120542 +nav short_teqc.14n Default_9488_0815_075927.m01 > short_teqc.14o

comment:5 Changed 7 years ago by asm

Navigation Processing

Did the basestation verification
Lat: 38 56 42.00249
Lon: -1 52 50.51963
Ell. Height: 749.894
Use antenna Height: 1.59 m

Used IMU boresight angles of 0, 0,0 and have to changed aircraft frame from 0, 0, -180 to 0, 0, 0. IpasTC was showing the following error during processing of the data:
Forward: 555416.000: $$$ GNSS data errors detected--will try and reject measurements, baselines or satellites
This was preventing to get a better result but still the output was better than using other stations. Lat and lon separation is better than 0.05 m and ambiguity fix is 2 for most of the flight but it changes around the last 1/3 of flight with separation can get to 1.2m and ambiguity fix drops to 1. Since we do not have LiDAR to process for this flight, this accuracy is enough as it will be lower than the Fenix pixel size.

Files copied, navigation processing is finished.

Last edited 7 years ago by asm (previous) (diff)

comment:6 Changed 7 years ago by dac

Digital camera

Converted raw Phase One data and generated events file (time diff +2). Starting image tagging.

comment:7 Changed 7 years ago by dac

Digital camera

Images tagged - ready for delivery generation.

comment:8 Changed 7 years ago by dac

Owl processing

Started calibrating files.

comment:9 Changed 7 years ago by asm

Fenix Processing

Have found SCTs
22 → 2
21 → 2
20 →2
19 → 1
18 →2
17 →2
16 → 2
15→ 2
14→ 1
13→ 2
12→ 1
11→ 2
10→ 2
9 → 1
8 → 1
7→ 2
6→ 2
5 → 2
4 → 1
3 → 1
2 → 1
1→ 2

asm will be away for 3 weeks, will resume this project then if it is not completed.

Last edited 7 years ago by asm (previous) (diff)

comment:10 follow-up: Changed 7 years ago by dac

Owl Processing

Default SCT of 2 looks good for all lines. Looks like some distortion in crossline but not due to SCT. Proceeding to map all bands.

comment:11 Changed 7 years ago by dac

Fenix Processing

Started mapping all bands on grid.

comment:12 in reply to: ↑ 10 Changed 7 years ago by lah

Replying to dac:

Owl Processing

Default SCT of 2 looks good for all lines. Looks like some distortion in crossline but not due to SCT. Proceeding to map all bands.

Ticket says Owl is not requested. Is this incorrect or are we just processing everything?

comment:13 Changed 7 years ago by dac

Fenix Processing

Set processing with correct FOV file for borrowed Fenix and bad pixel mask.

comment:14 Changed 7 years ago by dac

Fenix processing

Delivery created - ready for checking.

comment:15 Changed 7 years ago by asm

Hyperspectral Delivery Check

Started
There are 22 flightlines delivered but only 21 in logsheet! (Also 21 in the assignment readme flineame identification section) Flightline 12 is not noted on logsheet but the data is there. The flightline is an aborted flightline and the data is duplicated so it could be removed (and files need to be renamed) or it can be delivered making changes to logsheet and readme.

comment:16 Changed 7 years ago by dac

Hyperspectral

Removed line 12 and updated line names and readme.

comment:17 Changed 7 years ago by asm

Fenix Delivery Check

-There is no picture of the plane in the front page (I know that we have changed the plane so it is fine and can be delivered as it is but it looks a bit poor so we should select a new picture and change the template).
-Logsheet: Transit hours is negative (-1:43) as the one time was not missing, have deleted the info
-Scripts for checking spectra using Py6S could not find vegetation pixels except for a few lines. For those, it look fine. Checked spectra for all mapped flightlines in tuiview, everything looks fine.
-The APL commands works (still need to solve the image creation but everything is fine).

Run all other checks but not obvious errors found. Starting zipping mapped files

Last edited 7 years ago by asm (previous) (diff)

comment:18 Changed 7 years ago by asm

Fenix Delivery Check

Mapped files have been zipped successfully, project is ready to be delivered.

comment:19 Changed 7 years ago by dac

Fenix Delivery

Delivered data via FTP and copied to hard drive - will post Monday.

comment:20 Changed 7 years ago by dac

Owl Processing

Started mapping reprocessed Owl data on grid.

comment:21 Changed 6 years ago by dac

Owl Processing

Delivery created - ready for checking.

comment:22 Changed 6 years ago by asm

Owl Delivery

All checks have been made. Checking apl commands did not work as aplmap had as argument --pixelsize ? ?. Changed to a valid value and everything was correct.

Dispatched via FTP (arsf7) on 05/October/2017

Last edited 6 years ago by asm (previous) (diff)

comment:23 Changed 6 years ago by dac

Digital Camera

Started creating delivery with new scripts.

comment:24 Changed 6 years ago by asm

Digital Camera DC

Dan created delivery as part of the new processing chain implementation, I took over the DC:

-Google Earth displays the kml fine (and can click in photographs) and the eventfile also looks good on qgis
-No quality reports yet
Readme says that the first 4 photos could not be geolocated but they are included (pic 1 is a photograph showing only black so it could be manually removed from photogrhaps directory). They are not found in the eventfile and the kml so everything fine there.
-3 Pic is outside flightlines coverage ((127, 168, 358). )

comment:25 Changed 6 years ago by dac

Digital Camera

  • Regenerated delivery with updates scripts and removed areas outside coverage as part of this.
  • Waiting on updated data quality report before checking again.

comment:26 Changed 6 years ago by dac

Digital Camera

Added data quality report. Needs final check before delivering.

comment:27 Changed 6 years ago by asm

Digital camera Delivery check

Started.

comment:28 Changed 6 years ago by asm

Digital camera Delivery Check

Black picture has been removed and everything updated accordingly to the new numeration. The 3 photos outside coverage can be useful and can be delivered. Have removed from readme:

The following files could not be GPS tagged:
PHASEONE-IXU-ES17 106-2017168-00001.tif
PHASEONE-IXU-ES17 106-2017168-00002.tif
PHASEONE-IXU-ES17 106-2017168-00003.tif
PHASEONE-IXU-ES17 106-2017168-00004.tif

As they are now GPS tagged. Project is ready to be delivered

Last edited 6 years ago by asm (previous) (diff)

comment:29 Changed 6 years ago by asm

Camera Delivery

Dispatched via FTP (arsf7) on 18/October/2017

comment:30 Changed 6 years ago by dac

Fenix

For atmospheric correction, made copy and replaced FWHM in header with ones Specim provide using command:

for hdr_file in `ls /users/rsg/dac/scratch_network/nerc-arf-user_requests/brown_atm_correction_nov17/level1_hdr_replacement/*hdr`
 do 
     combine_arsf_specim_wls_files.py --arsf_wls /users/rsg/arsf/calibration/2017/fenix/wlcalFENIX.wls \
                  --specim_vnir /users/rsg/arsf/calibration/loan_fenix_2017/wlcal2b_fwhm.wls \
                  --specim_swir /users/rsg/arsf/calibration/loan_fenix_2017/wlcal1b_fwhm.wls \
                   --out_wvl /tmp/specim_tmp_wvl.txt --update_header --hdr $hdr_file
done

comment:31 Changed 6 years ago by dac

Archiving

Tidied files ready for archiving.

comment:32 Changed 6 years ago by dac

Archiving

Files uploaded to CEDA.

comment:33 Changed 6 years ago by dac

Archiving

Mapped owl files weren't zipped before delivery. Zipping now and will re-upload to CEDA.

Note: See TracTickets for help on using tickets.