Opened 12 years ago

Closed 11 years ago

Last modified 10 years ago

#53 closed flight processing (fixed)

CEH2007/01 (GB2007/12), flight day 240/2007, Buckingham

Reported by: mggr Owned by: mggr
Priority: alpha 4 high Milestone: 2007 data processing completion
Component: Processing: general Keywords: GB2007/12, CEH2007/01
Cc: benj Other processors:

Description

Data location: ~arsf/arsf_data/in_progress/2007/flight_data/uk/CEH2007_01-2007_240_Buckingham

Data arrived from ARSF via second SATA disk transfer at the start of Sept.

Scientific details:

  • Spectral analysis of UK peatland fires via airborne remote sensing: detection of 'flaming' k-emission lines and mapping of plume trace gases
  • ~/arsf_data/in_progress/2007/ARSF_Applications-GB_2007/GB07-12_Application_and_science_case.pdf

Attachments (2)

aisa_grep.txt (261.2 KB) - added by mggr 11 years ago.
Output from grep of disk image for hdr files
images.txt (9.0 KB) - added by mggr 11 years ago.
notes of locations of data on disk

Download all attachments as: .zip

Change History (22)

comment:1 Changed 12 years ago by mggr

Mailed Andrew Wilson and asked if he just wants the raw data.

comment:2 Changed 12 years ago by mggr

  • Owner changed from mggr to anee

Raw data and processed applanix only for this one too.

comment:3 Changed 12 years ago by anee

Combined operation unacceptably high – need to sort

comment:4 Changed 12 years ago by anee

Processing completed to end of applanix.

comment:5 Changed 12 years ago by mggr

Synced back into main repository.

comment:6 Changed 12 years ago by mggr

Sent on disk 3 to Andrew.

comment:7 Changed 12 years ago by mggr

  • Owner changed from anee to mggr
  • Status changed from new to assigned

Andrew noted that Eagle data is missing from this delivery. On checking, we never received this data from Kidlington (due to it needing to be recovered from the QC machine?). Phil has found the data and will forward it to Andrew and to us.

This should have been caught prior to delivery. As only raw data was sent, we didn't attempt to process the Eagle data and this didn't notice it was missing. I've amended the procedure so that we should verify we have all data mentioned on the logsheet at the point of receipt.

comment:8 Changed 12 years ago by mggr

  • Summary changed from GB2007/12 (and CEH2007/01?), flight day 240/2007, Buckingham to CEH2007/01 (GB2007/12), flight day 240/2007, Buckingham

Updating code per Gary's instructions:

GB07/12 was an ambiguous code that referred to eight different jobs, many of which were commissioned by CEH. To reduce confusion these were modified (in consultation with Andrew) to a CEH07 code. I know the log sheets still holds the code GB07/12 but I suggest that for the purposes of reporting these can be entered as 'CEH07/01 (GB07/12)'  for Buckingham (240), 'CEH07/06 (GB07/12)'  for Didcot (192b).  

comment:9 Changed 12 years ago by mggr

Found that the CASI data is also missing while checking 171a (#58). Forwarding to AKW on DVD.

comment:10 Changed 12 years ago by mggr

Following visiting Kidlington, it turns out that the eagle data was on a disk that disconnected during flight (on the last line) due to cables not being glued in. Consequently, the data structures on disk are corrupt and, additionally, there are actual bad sectors on the disk.

Kidlington have attempted recovery using standard tools with no success. I've taken an image (ie. removed the bad sector issue) and attempted to recover that with the same tools - no luck.

The PI (Andrew, and Shelley) has now informed of the problem and told that, while we'll continue to attempt recovery, that the prospects are slim.

The next steps are to search the disk image directly for the .hdr files to see if they're intact and present. If so, we'll need to write a small application and search the disk manually for imagery.

The first step has been completed successfully (found all 10 .hdr files).

comment:11 Changed 11 years ago by mggr

  • Cc benj added

Disk image is in two places currently:

  • pmpc974:/home/buckingham_disk_dup.raw-DO_NOT_DELETE-ask_mgg
  • pmpc886:/local0/data/arsf/buckingham_disk_dup.raw-DO_NOT_DELETE-ask_mgg

Attaching grep of the image to this ticket (should give approximate on-disk locations of hdr files, which might be near the image files).

The half-written disk viewer code is at ~mggr/arsf/diskviewer/ This is a heavily-hacked geotiff viewer that displays 'pages' of data from the image. It needs some interface features added to make it usable (basic back/forwards a page, small hops by <disk block size> bytes, etc).

Changed 11 years ago by mggr

Output from grep of disk image for hdr files

comment:12 Changed 11 years ago by mggr

After a bunch of custom coding, searching the disk, manual comparison and other work, all lines have been recovered, though lines 8, 4 & 1 are partially damaged (mostly missing small bits at the start or end).

Attaching a pile of notes to this ticket, though they won't make much sense to anyone else.

line1:
aline.bil  aline.hdr

line10:
eagle_header_file-125367681025.bil  eagle_header_file-125367681025.hdr

line2:
eagle_header_file-4390125568.bil  eagle_header_file-4390125568.hdr

line3:
eagle_header_file-4390060033.bil  eagle_header_file-4390060033.hdr

line4:
eagle_header_file-4389994497.bil  eagle_header_file-4389994497.hdr  eagle_header_file-4389994497.hdr-original

line5:
eagle_header_file-4388880385.bil  eagle_header_file-4388880385.hdr

line6:
eagle_header_file-4388814849.bil  eagle_header_file-4388814849.hdr

line7:
eagle_header_file-4388749312.bil  eagle_header_file-4388749312.hdr

line8:
eagle_header_file-4394516479.bil  eagle_header_file-4394516479.hdr  eagle_header_file-4394516479.hdr.original

line9:
eagle_header_file-129659568129.bil  eagle_header_file-129659568129.hdr

line_test:
eagle_header_file-121956597761.bil  eagle_header_file-121956597761.hdr

Changed 11 years ago by mggr

notes of locations of data on disk

comment:13 Changed 11 years ago by mggr

Line 8: (Eagle) Truncated on disk -no autodark, use dark lines from another file, header edited.

Line 4: Possibly front-truncated, needed to edit header.

Line 1: Data truncated about halfway through. No header file (totally reconstructed by hand). No autodark (destroyed by truncation).

final line "test": No attempt to recover.

comment:14 Changed 11 years ago by mggr

Note only .hdr and .raw files have been retrieved - .log & .nav are too hard to find. Processing will need to be done with azspec -sYS option and probably hdr file adjustments.

comment:15 Changed 11 years ago by mggr

Extracted dark frames from files surrounding the damaged ones for convenience.

Synced into main data repository.

comment:16 Changed 11 years ago by mggr

Notified AKW and asked how he wants it delivered / whether he wants it tested first.

comment:17 Changed 11 years ago by mggr

Prepared disk for posting on Monday per AKW's request. Included a sample script.

With any luck, this job is now done :)

comment:18 Changed 11 years ago by mggr

Andrew confirmed he's successfully worked with this data in the Dec steering committee.

comment:19 Changed 11 years ago by mggr

  • Resolution set to fixed
  • Status changed from assigned to closed

Data has been sent to NEODC for archiving.

comment:20 Changed 10 years ago by mggr

Archived at NEODC.

Note: See TracTickets for help on using tickets.