Version 32 (modified by mggr, 16 years ago) (diff)

--

Data Delivery

Preparation

What should be included

  1. level 1 data that has been mapped to level 3 successfully and verified as correctly geolocated (as far as possible).
  2. a readme file describing the contents of the delivery
  3. flight logsheet
  4. screenshots (level 3)
    • one of each line (no vectors)
    • mosaic of all lines for a single sensor, one with and one without vectors overlaid
    • each screenshot should try to show typical or noteworthy characteristics of the image - e.g. any distortion or error, or a typical part of the scene
  5. (2008+?) a DEM file where we have one and can give it to the user
  6. any ancillary files (spheroid separation grids for non-UK flights, OS GB correction factors, etc)

There are delivery template directories available (copy these and populate):

2006
~arsf/arsf_data/in_progress/2006/delivery_template/
2007
~arsf/arsf_data/in_progress/2007/delivery/template/
2008
~arsf/arsf_data/in_progress/2008/delivery/template/

The delivery data should be created in the project directory named delivery/YYYYMMDD/PROJECTCODE/... (e.g. delivery/20071215/GB07-05 for a delivery of GB07-05 created on 15/Dec/2007).

Necessary comments to make in the readme file

  1. Update anything marked TODO in the template
  2. Appropriate pixel size
  3. Appropriate bands for the sensor
  4. Comments on the quality of the data (accuracy vs vectors, any bad bands, etc) and any specific problems encountered
  5. Include a tested example command to convert the data from level 1 to level 3
  6. Ensure the text file is in Windows format (run unix2dos on it if necessary)
  7. (list of other things to change)

Verification

Once a dataset has been prepared for burning and delivery, a second person should go over it and:

  1. Check against the logsheet that all data that should be present is.
  2. Check for typos, etc in the documentation.
  3. Ensure the readme and other text files are Windows compatible (run unix2dos ReadMe.txt)
  4. Test out the level 3 processing command for one or more of the level 1 files (definitely one per sensor).
  5. All level 1 files should be looked at visually (ideally in all bands, though this is only practical for ATM and casi).

Final steps

Burning the data

  1. create an iso image with mkisofs -R -udf -verbose -o /tmp/dvd.iso -V <PROJECTCODE-JULIANDAY> <delivery_directory>
  2. test the iso with sudo mount -o ro,loop /tmp/dvd.iso /mnt/tmp - mounts to /mnt/tmp, go look at it, then unmount with sudo umount /mnt/tmp (yes, that is umount not unmount)
  3. if it's ok, burn to CD/DVD with sudo cdrecord dev=/dev/dvd -sao fs=100m driveropts=burnfree -v /tmp/dvd.iso
  4. clean up with rm /tmp/dvd.iso
  5. check the dvd on another machine (ideally Windows)

Lightscribe

  1. open the CD template from ~arsf/arsf_data/in_progress/2007/delivery/cd_covers/arsf_cd_label_gimp_editable.xcf in gimp
  2. click on the text button in the toolbox, then click the GB... text and edit to reflect the correct project code
  3. save as a .bmp file somewhere, e.g. ~/ipy0708214ef.bmp
  4. on the lightscribe machine (currently pmpc974), run 4L-cli print /dev/sr0 ~/ipy0708214ef.bmp

If it appears to be stuck on "Starting up", it may still be burning - give it time to complete. For a dense full disk picture, wait at least 45mins!

If the drive seems wrong, try 4L-cli enumerate to list attached lightscribe compatible drives.

Preparing hard disks for delivery

You will need to have root permissions before you start.

To create a partition:

Unmount disk run dmesg to find device name (listed at the bottom if just plugged in)

run fdisk /dev/device name (probably sdb)

enter 'p' to print partition table and check you have selected the correct disk
enter 'd' to delete the current partition
enter 'n' to create a new partition
enter 'p' to make new partition the primary partition
enter 'p' to print new partition table – if all seems fine enter 'w' to write

To format the disk:

ensure disk is unmounted again

to be on the safe side, run dmesg again to make sure device name hasn't changed

run mke2fs -j /dev/partition (probably sdb1)

Change permissions:

make writable for everyone – chmod a+rwx /media/disk

Finalising hard disk

Set permissions and owner:

chmod a+rX,a-w -R .
chown root.root -R .

Cover letter

There should be a cover letter with all deliveries. Templates can be found in ~arsf/arsf_data/in_progress/YYYY/delivery/

Trac and website updating

  1. Add a comment on the trac ticket for this project that this data has been delivered (describe what was sent and when).
  2. Also add a quicklook to the ticket.
  3. Update the status database to reflect the delivered data (​http://www.npm.ac.uk/rsg/projects/arsf/status/addflight/editflight.php)

Email the PI

  1. email them that you're putting the disk in the post and ask for confirmation of receipt 
    Subject: Notification of ARSF data delivery for <PROJECT CODE>

Dear <PI>,

This is to notify you that we have just dispatched some of your ARSF data and that you should receive it shortly.  Please let us know when the data arrives so we know not to resend.

The data is sent on USB hard disk (formatted with the Linux ext3 filesystem) or DVD depending on total size.  If we sent a hard disk, please copy the data onto your system and return the disk to us (see below for address) for reuse.

The delivery comprises:
 - ATM flight lines taken on <DATE>
 - CASI flight lines taken on <DATE>
 - Specim Eagle flight lines taken on <DATE>
 - Specim Hawk flight lines taken on <DATE>

<NOTES - any other notes, including what data is held back>

Information about the processing of the project is included with the delivery, but our internal notes are also available at:
 http://www.npm.ac.uk/rsg/projects/arsf/trac/ticket/<TICKETNO>

Regards,

ARSF Data Analysis Node.
Plymouth Marine Laboratory,
Prospect Place,
Plymouth.
PL1 3DH.
UK.

Email: arsf-processing@pml.ac.uk
Tel: +44 (0)1752 633432
Fax: +44 (0)1752 633101
Web (NERC): http://arsf.nerc.ac.uk/
Web (processing wiki): http://www.npm.ac.uk/rsg/projects/arsf/trac/