Ferrier Reprocessing Archived WM06_13 Airborne Data

[asm]

Reprocessing of Eagle and Hawk data from 2006. Project code is WM06_13, area is Rodalquilar, Spain.

Flight was undertaken on 153 2006

[asm]

Eagle and Hawk

Data was archived at CEDA and not processed at NEODAAS (PML). The RAW data was not available, downloaded level1b and hdf files. Extracted navigation from hdf files and reprocessed the data using the level1b files with APL. There was missing info that needed to be added to the level1b to allow APL processing chain, that data was extracted from hdf files and added to the level1b hdr files including:
-Acquisition date
-Start time
-Stop time
-Binning
-x start (starting at 27 for the Eagle sensor)

Also corrected the Hawk sensor id and sensor name (Eagle was displaying) and gave correct radiance units rather than reflectance percentage (as it is not the case).

[asm]

Eagle and Hawk

Created a delivery, quite a few things were sorted manually. Specially creation of xml files, mosaics and a few other steps. Created a readme as well. The script for readme creation was failing trying to get overflows and underflows from the mask files as those are not present. I copied some files for another project and then removed the files and sections in the readme as well all references to masking the data. Also edited manually the Readme file quite a bit.

After all the workarounds, I think this project is now ready and awaiting DC.

[dac]

Delivery Check

Readme

• In delivery contents doesn't have any file names listed for mapped files (can't remember of we normally do)
• Hadn't noticed this in the readme before that ARSF-DAN is mentioned as a predecseosr of NEODAAS, I know what is meant here but how it is written isn't correct. Suggest change to "As the data processed prior to 2007 predate the involvement of NEODAAS, and the ARSF Data Analysis Node operated at PML..."
• "Hence the Eagle and Hawk deliveries have reduced information in particular on data quality." change to "Hence, the Eagle and Hawk deliveries have reduced information, in particular on data quality."
• Change "It is important to be aware that usual quality control is more limited with these data as the raw datasets are usually not available." to "It is important to be aware that usual quality control is more limited with these data as the raw datasets were not available" (be specific for this delivery)
• Filename for DEM and nav data are wrong in example aplcorr command

Delivery

• Missing ASCI view vectors file for hawk
• In line info 'Captured by NCEO' should be 'Captured by ARSF', not sure I spotted this on previous delivery. Should also change there before archiving to CEDA
• Couldn't run check apl_cmd, I think this is due to the lack of mask files to run aplmap
• Aligment looks OK based on screenshots, couldn't open the data to check (too large)
• Eagle data is noisy but not much we can do about this

Once these problems are resolved good to go. You can start zipping up the mapped files now.

[asm]

Eagle and Hawk

Realised that there was a missing line, eagle flightline 9. The commands failed to process this flightline as the number of lines noted in the hdr file and hdf were 10079 but the size was 10080 lines. Correcting for it to add the line to the delivery and will re-create the screenshots.

[asm]

Eagle and Hawk

Have created flightline 9 for eagle. Re-created the screenshots, updated the readme with all suggestions and zipped files again. I have also created the hawk ASCII view vectors file. Updated the xml files with ARSF mention. Indeed the run check apl_cmd does not work as there are no masked files.

I think this delivery is ready again for a final delivery check and should be ready to be delivered.

[dac]

Delivery Check

All looks good apart from two minor points:

• Remove fodis folder
• Viewvectors should have Windows line endings (use unix2dos)

Delivery is 105 GB so note this in delivery as might be better to send on a hard drive.

[asm]

Eagle and Hawk Delivery

All changes suggested were made. Data was delivered to PI via FTP on 13/11/2021. As the data is over 100 Gb, sending a hard drive over the mail has as well been suggested.

[wja]

Wrong Data!

Frustratingly, the data appears to be labeled wrong at CEDA. We believe this flights' data is actually 2006 138, which is labeled as WM06-16, San Marcos. This is not as simple as having the wrong project code as the readme, documentation and filenames all reference 2006 153 WM06_13 Rodalquillar.

2006 138 WM06-16 San Marcos data is on CEDA, but only ATM and CASI. PI has been if it's possible to provide data, Wendy at CEDA also asked whether they have these data anywhere.

CEDA link for 2006 138 WM06-16 San Marcos ​https://data.ceda.ac.uk/neodc/arsf/2006/WM06_16/WM06_16-2006_138_San_Marcos

[asm]

Eagle and Hawk

The data on CEDA was mislabelled and did not corresponded to this dataset. PI uploaded the data that was delivered to him and needed to be reprocessed.

This dataset needed a lot of manual checks and wrappers for all apl commands had to be tailored for this processing. The level1b files were missing a lot of info that had to be extracted from the hdf and written into the BIL hdr files.
-Acquisition date
-Start time
-Stop time
-Binning
-x start (starting at 27 for the Eagle sensor)

However, the acquisition date was badly formatted in the hdf files and showed time as hmmss rather than hhmmss or hh-mm-ss. This was causing some scripts to break and took some time to figure out as initially we believed it was seconds of the week.

The Hawk also had incorrect sensor name and sensorid that had to be manually fixed.

Contrary to the project previously processed, this dataset have information recorded for boresight angles: 0.1, 0.2, -1.1
Same to what we have in our config files from that year.

However, flightlines present offset likely caused for a wrong boresight value applied to the hdf files (from where the navigation was extracted). I tried to process aplmap with boresight values 0.1, 0.2, -1.1 and variations in negative symbols but the best fit still for boresight 0, 0, 0 rather than any other meaning the above values were already applied to the hdf files (and the extracted nav files as it was expected).

In the case of Eagle line 5 there is a large SCT error causing wobbly geocorrection.

[asm]

Eagle and Hawk

Used aplshift in a loop for creating different SCT corrections to find the best fit that accounts for wobbly arreas (in combination with aplcorr, apltran and aplmap). After lots of work and visual inspection, the best fit was found to be SCT=7.88 (eagle line 5).

The Eagle line 5 with SCT=7.88 also overlaps nicely with Hawk line 5 with no correction (which is very surprising and nice news for a change).

[asm]

Eagle and Hawk

Have created a delivery. Tailored a Readme file and created a project xml file. Edited manually xml files for some flightlines.

In this case, I am not including the logfile as what we have looks for another project.

Project should be ready for DC.

[wja]

Delivery Check

Issues:

• Navigation file for Eagle line 1 is missing
• Rename mapped data to include projection (UTM zone 30)
• Some updates to make to the ReadMe:
  • No overflow information. Need to say that they overflows are evident in Eagle data. There are underflows in Hawk also, but mainly the usual wavelengths, the ocean
  • Mosaic screenshots need resizing
  • There are two gaps (new paragraph or LaTex 'skip'?) within a paragraph in the "Quality and issues with data" section
• Eagle line 5 XML doesn't open (XML Parsing Error: mismatched tag). The rest are fine
• Lots of overflows in all Eagle files

Files missing, please check these are due to lack of data and flight information:

• Navigation quality files (*b_nav_post_processed.bil)
• FODIS data
• Logsheet
• Level1b mask (*1b_mask.bil)
• Level1b badpixel mask (*mask-badpixelmethod.bil)

No hyperspectral config file. Cannot check commands using check_apl_cmd. Tested ReadMe APL commands manually with:

aplcorr -lev1file flightlines/level1b/e153031b.bil -igmfile $OUTDIR/e153033b.igm -vvfile sensor_FOV_vectors/eagle_fov_fullccd_vectors.bil -navfile flightlines/navigation/e153031b_nav_post_processed.bil -dem dem/WM06_13-2006_153-ASTER.dem

apltran -inproj latlong WGS84 -igm $OUTDIR/e153033b.igm -output /users/rsg/arsf/arsf_data/2006/flight_data/spain/WM06_13-2006_153_Rodalquilar/processing/wja_cmd_check/e153033b_utm_wgs84N30.igm -outproj utm_wgs84N 30

aplmap -igm $OUTDIR/e153033b_utm_wgs84N30.igm -lev1 flightlines/level1b/e153031b.bil -mapname $OUTDIR/e153033b_mapped_utm30n.bil -pixelsize 1 1 -bandlist ALL

I agree that offsets appear to be due to boresight, not timing. Boresight values are unknown for this flight but this is explained well in the ReadMe.

[asm]

Hyperspectral Delivery

Addressing all comments from DC.
-There are no mask files as we don't have the calibration file.
-As this flight was mislabelled, I can't see any logsheet that matches this delivery. I hope the PI already has it.
-No fodis data either
-Boresight values are those that were encoded in the hdf. They are known but there is still an offset if lines were flown in different direction.
-Renamed mapped files to include utm region.
-There is only the nav files extracted from hdf files. Added missing nav file.
-Eagle line 5 XML, sorted now. I added manually two processing steps that broke the file but fixed now.
-ReadMe

-Resized the screenshots
-Sorted spacing in "Quality and issues with data"
-I still have not included information about overflowing pixels as this is done with the mask and the calibration file that we don't have. We don't have the information about how overflowing and underflowing pixels were masked but the ReadMe file points to the data quality report that explains this issues.

If these changes are accepted, the mapped files could be zipped and project marked as ready to be delivered.

[wja]

Ready to Delivery

[asm]

Hyperspectral Delivery

Delivered to PI via FTP and notification sent (07/02/2022).