Version 94 (modified by knpa, 15 years ago) (diff)

--

Data Delivery

Preparation

Hyperspectral deliveries

What should be included

  1. level 1 data that has been mapped to level 3 successfully and verified as correctly geolocated (as far as possible).
  2. a readme file describing the contents of the delivery
  3. flight logsheet
  4. screenshots (level 3)
    • one of each line (no vectors)
    • mosaic of all lines for a single sensor, one with and one without vectors overlaid
    • each screenshot should try to show typical or noteworthy characteristics of the image - e.g. any distortion or error, or a typical part of the scene
  5. (2008+?) a DEM file where we have one and can give it to the user
  6. any ancillary files (spheroid separation grids for non-UK flights, OS GB correction factors, etc)

Scripted Approach

  1. Run the script to convert the level 3 Tiff files to jpgs.
    • This will convert all tiff files in directory to jpegs (keeping the tiffs as well)
      • gtiff2jpg.py -d <dir containg tiffs> -a
    • This will only convert the file named filename
      • gtiff2jpg.py -f <filename> -a
  2. Create the delivery directory using make_delivery_folder.sh
    • make_delivery_folder.sh <main project dir> <year> <proj-code>
    • e.g. make_delivery_folder.sh ~mark/scratch/GB08_35-2008_Example 2008 GB08-35
    • This will create the delivery directory and populate it with the your lev1 directory files and the other usual directories. It will not copy the misc directory or separation grids nor a DEM as these are not supplied as default in the delivery. These can be manually copied from the below template directories. The lev1 files will be renamed as follows. If the filename is xxx_gpt0.1.zzz the renamed file will be xxx.zzz. This is based on the default file name structure as example e129a121b_gpt0.0.bil. File names with more than 1 '_' will be renamed incorrectly. (I'll make this more robust at some point).
  3. Copy the misc directory or other files (such as the jpgs) if required for the delivery.
  4. Ensure that an ASCII version of the logsheet exists in the project admin/ directory. To create one, start open office and save logsheet as .txt format. Also ensure no other file in the admin dir as a .txt extension. Run the Read Me generation script
    • `create_readme.py -d <main project directory> -D <DEM-TYPE>
    • <DEM-TYPE> the type of DEM used (in capitals).
    • This should create a file named ReadMe.txt in your delivery folder. It WILL still need to be edited. Flight line numbers will need to be added to the table at the top of the read me, also any reference to vectors will have to changed as required. Please still read through the final read me to check for errors.

Manual Approach

There are delivery template directories available (copy these and populate):

2006
~arsf/arsf_data/in_progress/2006/delivery_template/
2007
~arsf/arsf_data/in_progress/2007/delivery/template/
2008
~arsf/arsf_data/in_progress/2008/delivery/template/

The delivery data should be created in the project directory named delivery/YYYYMMDD/PROJECTCODE/... (e.g. delivery/20071215/GB07-05 for a delivery of GB07-05 created on 15/Dec/2007).

Necessary comments to make in the readme file

  1. Update anything marked TODO in the template
  2. Appropriate pixel size
  3. Appropriate bands for the sensor
  4. Comments on the quality of the data (accuracy vs vectors, any bad bands, etc) and any specific problems encountered
  5. Include a tested example command to convert the data from level 1 to level 3
  6. Ensure the text file is in Windows format (run unix2dos on it if necessary)
  7. (list of other things to change)

LIDAR deliveries

What should be included

  1. ASCII LIDAR pointcloud data
  2. flight logsheet
  3. readme file describing the data set + copyright notice
  4. screenshot of mosaic of all lines (full resolution) and zoomed image with vector overlay
  5. data quality report and further processing scripts
  6. DEM of the LIDAR data - usually gridded to 2m resolution. Use SRTM to fill holes and extend lines.
  7. Screenshot of DEM

Procedure for creating a LIDAR data set delivery

  1. Copy the template directory over to the project directory. Template directory at ~arsf/arsf_data/2009/delivery/lidar/template
  2. Convert the LAS binary files into ASCII files, ensuring to output all the appropriate information
    • run las2txt --parse txyzicra <lidarfilename> <outputfilename> for each file, outputting to the ascii_laser directory (may not work with LAS 1.1).
    • REMEMBER THAT THESE LAS FILES SHOULD HAVE BEEN QC'ED AND CLASSIFIED FOR NOISY POINTS
    • If not already done, rename the files in the convention "LDR-PPPP_PP-yyyydddnn.txt" (details in readme).
  3. Create a full resolution JPEG of a mosaic of all the LIDAR lines + a separate one with vector overlays and put in screenshot directory (with make_lidardem_or_intensity.sh?)
  4. Go through the readme file and edit as required
    • Enter all the project details into the table at the top
    • Fill in the contents. Add the lidar filenames on the left and line numbers (from logsheet) on the right.
    • Enter the projection and datum details - get these from ALS PP output dialog when processing.
      • UK flights should be - Horizontal Datum: "ETRF89" Vertical Datum: "Newlyn" Projection: "OSTN02 British National Grid"
    • Insert statistics for each line: lasinfo --check <lidarfilename> then cut/paste the required information
    • Check the accuracy of data vs vectors and estimate the average offset, also the average elevation offset between neighbouring lines
    • Search for 'TODO' and change anything that remains to be done
  5. Ensure readme file is in Windows format (run unix2dos on it)
  6. Make sure correct upto date data quality report (pdf version) is included in docs directory
  7. Include the flight logsheet with the delivery
  8. If DEM was created for processing then include in delivery. Else create one (with make_lidardem_or_intensity.sh) and include in delivery. Create a screenshot also.

Verification

Once a dataset has been prepared for burning and delivery, a second person should go over it and:

For Hyperspectral Data

  1. Verify we have the correct PI (check against application, call Gloucester if unsure) and address (application, google, Gloucester)
  2. Check against the logsheet that all data that should be present is.
  3. Check for typos, etc in the documentation.
  4. Ensure the readme and other text files are Windows compatible (use the 'file' command on the .txt files: they should be ASCII with CRLF terminators) (run unix2dos ReadMe.txt)
  5. Test out the readme level 3 processing command for one or more of the level 1 files (definitely one per sensor).
  6. All level 1 files should be looked at visually (ideally in all bands, though this is only practical for ATM and casi).

For LIDAR Data

  1. Verify we have the correct PI (check against application, call Gloucester if unsure) and address (application, google, Gloucester)
  2. Check against the logsheet that all data that should be present is.
  3. Check for typos, etc in the documentation.
  4. Ensure the readme and other text files are Windows compatible (use the 'file' command on the .txt files: they should be ASCII with CRLF terminators) (run unix2dos ReadMe.txt)
  5. Open all the ASCII LIDAR files into a GIS and check they open/load ok
  6. Check overlaps with neighbouring lines for obvious errors
  7. Check against vectors if available
  8. use tail/head to check the ASCII LIDAR files have all the required columns in the order specified in the readme.txt
    • head <lidarfilename>

Once the delivery has been checked (and corrected if necessary), the project directory (all of it, not just the delivery) should be rsynced back to the repository:

  1. su to arsf
  2. cd to the relevant project directory under ~arsf/arsf_data/
  3. Run rsync -av --dry-run <processing_directory> ./
    • This should give you a list of files that it thinks have been updated - check this looks sensible
  4. If there are any files in the list that shouldn't be rsynced back (eg. screenshots in the wrong place, test scripts), take them out of the rsync by adding --exclude <exclude_pattern>. Then re-run the dry-run rsync and check that the files to be excluded are no longer in the list.
  5. Once you're happy with the list of files to be copied to the repository, re-run the rsync command without the --dry-run argument - this will actually copy the files.
  6. Put a comment in the ticket to say where you've rsynced the files back to.

Final steps

If you wish to burn the data to DVD please see notes at the bottom of this page.

Preparing hard disks for delivery

Plug in the external hard drive.

You will need to have root permissions before you start ('sudo su' and then type in your password).

Run dmesg to find device name. Listed at the bottom if just plugged in.

e.g 

SSELinux: initialized (dev sdb, type fuseblk), uses genfs_contexts

Device path is then /dev/sdb

If you see: 

 sdb: sdb1

then sdb1 is the name to use.

Create a partition:

Unmount the disk with umount /dev/device_name, e.g. umount /dev/sdb, and plug back in again.

Run fdisk /dev/device_name

enter 'p' to print partition table and check you have selected the correct disk
enter 'd' to delete the current partition
enter 'n' to create a new partition
enter 'p' to make new partition the primary partition. If it asks you to give a partition number press 1 and enter. Then press enter twice.
enter 'p' to print new partition table – if all seems fine enter 'w' to write

Format the disk:

Ensure disk is unmounted and plugged back in again.

To be on the safe side, run dmesg again to make sure device name hasn't changed. You should now see the partition listed as device_name1, e.g. sdb1

Run mke2fs -j /dev/partition

Change permissions:

Unplug the disk again and plug back in. The disk space will be located under /media/disk.

Make writable for everyone using chmod a+rwx /media/disk


Copy over your data onto the disk

Top directory should be the one with the project code (from within the delivery directory). Copy the data to /media/disk.


Finalising hard disk

Set permissions and owner:

chmod a+rX,a-w -R .
chown root.root -R .

NOTE: Be sure to record the number of the hard disk (or giving it one if it does not yet have one) before packing it.

Cover letter

There should be a cover letter with all deliveries. Templates can be found in ~arsf/arsf_data/YYYY/delivery/

Data quality report

There should be a hard copy of the most recent data quality report included with all deliveries. Print off the most recent one from ~arsf/doc

Trac and website updating

  1. Add a comment on the trac ticket for this project that this data has been delivered (describe what was sent and when).
  2. Update the status database to reflect the delivered data (​http://www.npm.ac.uk/rsg/projects/arsf/status/addflight/editflight.php)
  3. Update the page saying who has our disks? (if sent on disk)

Email the PI

  1. email them that you're putting the disk in the post and ask for confirmation of receipt and CC to arsf-processing@… to keep others informed

Subject:

Notification of ARSF data delivery for <PROJECT CODE>

Body:

(note you need to fill in PI, DATE, NOTES, TICKETNO, also mention if you aren't sending all their data at once) 

Dear <PI>,

--PICK ONE OF THE NEXT TWO SENTENCES (DVD or HDD)--
This is to notify you that we have just dispatched your ARSF data on a USB hard disk formatted with the Linux ext3 filesystem.  Please let us know when the data arrive so we know not to resend.  We'd appreciate it if you could copy the data onto your system and return the disk to us (see below for address) for reuse.  
This is to notify you that we have just dispatched your ARSF data on DVDs.  Please let us know when the data arrive so we know not to resend.

The delivery comprises:
 - ATM flight lines taken on <DATE>
 - CASI flight lines taken on <DATE>
 - Specim Eagle flight lines taken on <DATE>
 - Specim Hawk flight lines taken on <DATE>

<NOTES - any other notes, including what data is held back>

Information about the processing of the project is included with the delivery, but our internal notes are also available at:
 http://www.npm.ac.uk/rsg/projects/arsf/trac/ticket/<TICKETNO>

Regards,


ARSF Data Analysis Node.
Plymouth Marine Laboratory,
Prospect Place,
Plymouth.
PL1 3DH.
UK.

Email: arsf-processing@pml.ac.uk
Tel: +44 (0)1752 633432
Fax: +44 (0)1752 633101
Web (NERC): http://arsf.nerc.ac.uk/
Web (processing wiki): http://www.npm.ac.uk/rsg/projects/arsf/trac/

This section relates to burning data to DVD

Burning the data

  1. create an iso image with mkisofs -R -udf -verbose -o /tmp/dvd.iso -V <PROJECTCODE-JULIANDAY> <delivery_directory>
  2. test the iso with sudo mount -o ro,loop /tmp/dvd.iso /mnt/tmp - mounts to /mnt/tmp, go look at it, then unmount with sudo umount /mnt/tmp (yes, that is umount not unmount)
  3. if it's ok, burn to CD/DVD with sudo cdrecord dev=/dev/dvd -sao fs=100m driveropts=burnfree -v /tmp/dvd.iso
  4. clean up with rm /tmp/dvd.iso
  5. check the dvd on another machine (ideally Windows)

Lightscribe

  1. open the CD template from ~arsf/arsf_data/in_progress/2007/delivery/cd_covers/arsf_cd_label_gimp_editable.xcf in gimp
  2. click on the text button in the toolbox, then click the GB... text and edit to reflect the correct project code
  3. save as a .bmp file somewhere, e.g. ~/ipy0708214ef.bmp
  4. on the lightscribe machine (currently pmpc974), run 4L-cli print /dev/sr0 ~/ipy0708214ef.bmp

If it appears to be stuck on "Starting up", it may still be burning - give it time to complete. For a dense full disk picture, wait at least 45mins!

If the drive seems wrong, try 4L-cli enumerate to list attached lightscribe compatible drives.