CEH2007/01 (GB2007/12), flight day 240/2007, Buckingham

[mggr]

Data location: ~arsf/arsf_data/in_progress/2007/flight_data/uk/CEH2007_01-2007_240_Buckingham

Data arrived from ARSF via second SATA disk transfer at the start of Sept.

Scientific details:

• Spectral analysis of UK peatland fires via airborne remote sensing: detection of 'flaming' k-emission lines and mapping of plume trace gases
• ~/arsf_data/in_progress/2007/ARSF_Applications-GB_2007/GB07-12_Application_and_science_case.pdf

[mggr]

Mailed Andrew Wilson and asked if he just wants the raw data.

[mggr]

Raw data and processed applanix only for this one too.

[anee]

Combined operation unacceptably high – need to sort

[anee]

Processing completed to end of applanix.

[mggr]

Synced back into main repository.

[mggr]

Sent on disk 3 to Andrew.

[mggr]

Andrew noted that Eagle data is missing from this delivery. On checking, we never received this data from Kidlington (due to it needing to be recovered from the QC machine?). Phil has found the data and will forward it to Andrew and to us.

This should have been caught prior to delivery. As only raw data was sent, we didn't attempt to process the Eagle data and this didn't notice it was missing. I've amended the procedure so that we should verify we have all data mentioned on the logsheet at the point of receipt.

[mggr]

Updating code per Gary's instructions:

GB07/12 was an ambiguous code that referred to eight different jobs, many of which were commissioned by CEH. To reduce confusion these were modified (in consultation with Andrew) to a CEH07 code. I know the log sheets still holds the code GB07/12 but I suggest that for the purposes of reporting these can be entered as 'CEH07/01 (GB07/12)'  for Buckingham (240), 'CEH07/06 (GB07/12)'  for Didcot (192b).  

[mggr]

Found that the CASI data is also missing while checking 171a (#58). Forwarding to AKW on DVD.

[mggr]

Following visiting Kidlington, it turns out that the eagle data was on a disk that disconnected during flight (on the last line) due to cables not being glued in. Consequently, the data structures on disk are corrupt and, additionally, there are actual bad sectors on the disk.

Kidlington have attempted recovery using standard tools with no success. I've taken an image (ie. removed the bad sector issue) and attempted to recover that with the same tools - no luck.

The PI (Andrew, and Shelley) has now informed of the problem and told that, while we'll continue to attempt recovery, that the prospects are slim.

The next steps are to search the disk image directly for the .hdr files to see if they're intact and present. If so, we'll need to write a small application and search the disk manually for imagery.

The first step has been completed successfully (found all 10 .hdr files).

[mggr]

Disk image is in two places currently:

• pmpc974:/home/buckingham_disk_dup.raw-DO_NOT_DELETE-ask_mgg
• pmpc886:/local0/data/arsf/buckingham_disk_dup.raw-DO_NOT_DELETE-ask_mgg

Attaching grep of the image to this ticket (should give approximate on-disk locations of hdr files, which might be near the image files).

The half-written disk viewer code is at ~mggr/arsf/diskviewer/ This is a heavily-hacked geotiff viewer that displays 'pages' of data from the image. It needs some interface features added to make it usable (basic back/forwards a page, small hops by <disk block size> bytes, etc).

[mggr]

Output from grep of disk image for hdr files

[mggr]

After a bunch of custom coding, searching the disk, manual comparison and other work, all lines have been recovered, though lines 8, 4 & 1 are partially damaged (mostly missing small bits at the start or end).

Attaching a pile of notes to this ticket, though they won't make much sense to anyone else.

line1:
aline.bil  aline.hdr

line10:
eagle_header_file-125367681025.bil  eagle_header_file-125367681025.hdr

line2:
eagle_header_file-4390125568.bil  eagle_header_file-4390125568.hdr

line3:
eagle_header_file-4390060033.bil  eagle_header_file-4390060033.hdr

line4:
eagle_header_file-4389994497.bil  eagle_header_file-4389994497.hdr  eagle_header_file-4389994497.hdr-original

line5:
eagle_header_file-4388880385.bil  eagle_header_file-4388880385.hdr

line6:
eagle_header_file-4388814849.bil  eagle_header_file-4388814849.hdr

line7:
eagle_header_file-4388749312.bil  eagle_header_file-4388749312.hdr

line8:
eagle_header_file-4394516479.bil  eagle_header_file-4394516479.hdr  eagle_header_file-4394516479.hdr.original

line9:
eagle_header_file-129659568129.bil  eagle_header_file-129659568129.hdr

line_test:
eagle_header_file-121956597761.bil  eagle_header_file-121956597761.hdr

[mggr]

notes of locations of data on disk

[mggr]

Line 8: (Eagle) Truncated on disk -no autodark, use dark lines from another file, header edited.

Line 4: Possibly front-truncated, needed to edit header.

Line 1: Data truncated about halfway through. No header file (totally reconstructed by hand). No autodark (destroyed by truncation).

final line "test": No attempt to recover.

[mggr]

Note only .hdr and .raw files have been retrieved - .log & .nav are too hard to find. Processing will need to be done with azspec -sYS option and probably hdr file adjustments.

[mggr]

Extracted dark frames from files surrounding the damaged ones for convenience.

Synced into main data repository.

[mggr]

Notified AKW and asked how he wants it delivered / whether he wants it tested first.

[mggr]

Prepared disk for posting on Monday per AKW's request. Included a sample script.

With any luck, this job is now done :)

[mggr]

Andrew confirmed he's successfully worked with this data in the Dec steering committee.

[mggr]

Data has been sent to NEODC for archiving.

[mggr]

Archived at NEODC.