File structure and naming

A single site/logical group of flight lines/sortie should be contained in a single directory. If there are multiple sites from different projects, significantly different times of overflight or large time gaps, each should have its own directory.

The directory name should look like: PROJECTCODE-YYYY_JJJxx_SITENAME, e.g. GB07_07-2007_102a_Inverclyde, boresight-2007_198, etc.

There should be:

• no spaces, brackets or other Unix-upsetting characters in filenames
• execute permission removed from anything non-executable
• no empty directories? (need to think about this, maybe only do after checking it's not a stub for later data)

The structure under the project directory should look like:

Top level dir	2nd level dir	Files	Comments
admin			Contains the flight logsheet and any miscellaneous information (readme's, copies of trac tickets, etc)
applanix			Raw and processed Applanix navigation (raw files in applanix/raw/DCALM*, final processed data in applanix/*sbet or applanix/Proc/sbet*.out)
	auto_applanix (optional)	auto.bat	If navigation was processed using auto_applanix script then this dir will contain additional output from that script
		combine_seperation_bestguess_values
		normal_first_convertbase.log
		normal_first_convertflight.log
		*.bat
		*GNSS.cmb
		*GNSS.dat
		*GNSS_gnuplot
		*GNSS.gnv
		*GNSS.log
		*GNSS.png
		*GNSS.sum
	base (deprecated)		?
	eo (possibly optional and often empty)		?
	extract	*.dat	Plane gps data after extracted from raw files by applanix
		dephem_01.dat
		extract_01.log
		gps_prim_01.dat
		hwconf_01.out
		iinr_01.out
		iinz_01.out
		imu_01.dat
		mgps_01.epp
		mgps_01.gpb
		mgps_01.nov
		mgps_01.sta
		navclk_prim_01.dat
		obs_prim_01.dat
		rers_01.out
		rinv_01.out
		rmrs_01.out
		rrms_01.out
		rtstat_01.txt
		tm_01.dat
		vnav_01.out
		vrms_01.out
		*GNSS.dat
	gps	*.fml	?
		*.fss
		*.fwd
		*.his
		*.rev
		*.rml
		*.rss
	proc	ers_01.out	Processed applanix data
		idx_01.txt
		inv_01.out
		iin_01.out
		mrs_01.out
		rms_01.out
		rmsg_01.out
		sbet_01.out
		smers_01.out
		smhkm_01.dat
		smindex_01.dat
		smout_01.out
		smparm_01.txt
		smphi_01.dat
		smpx_01.dat
		smreset_01.dat
		smrms_01.out
		smrmsg_01.out
	raw	DCALM.???	Raw gps data from plane
atm			Raw ATM data files (az???LLL.JJJ, where ??? = disk id, LLL = 3 digit file number (000 typically contains navigation), JJJ = Julian day of capture), pre-2009 only
calibration			Rypically a symlink to the appropriate calibration directory
casi			Raw CASI data files (ca??LLL.JJJ, where ?? = disk id, LLL = 3 digit file number, JJJ = Julian day of capture), pre-2008 only
delivery			Contains copies of data delivered to end users
	(see below)		Structure described below
dem			Contains azgcorr compatible DEMs
eagle			Raw Eagle data files (.raw = image data, .hdr = ENVI compatible header, .nav = navigation synchronisation data)
	orignal_headers (optional)		If the header files have been altered for any reason, the originals should be in here
grid_logs (deprecated)			Logs from hyperspectral script runs on gridnode (one log per sct run)
hawk			Raw Hawk data files (.raw = image data, .hdr = ENVI compatible header, .nav = navigation synchronisation data)
	original_headers (optional)		If the header files have been altered for any reason, the originals should be in here
leica			Leica LIDAR data
	(see below)		Structure described below
lev1			Processed hyperspectral level 1 data (note this will be the most current, not necessarily what was delivered), see below for filename info
lev3			Processed hyperspectral level 3 data (note this will be the most current, not necessarily what was delivered), see below for filename info
lidar (deprecated)			Raw and processed ULM LIDAR (raw LIDAR looks like "surveyXXX.range", processed is typically in a strips/ directory)
logs			Logs from hyperspectral script runs on gridnode (one log per flightline)
photos (deprecated)			RC10 scanned aerial photographs
rinex			GPS basestation data appropriate to the site
runa			ATM processing scripts
runc			CASI processing scripts
rune			Eagle processing scripts
runh			Hawk processing scripts
tabi			Raw TABI data
vectors			(2007/8 only) - symlink to appropriate vectors

The lev1 and lev3 data filenames are structured as "sdddfnn1b.{hdf|tif}", where:

• s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
• ddd is the Julian day of the data acquisition
• f distinguishes between multiple flights on the same day (“a” being the first flight)
• nn is the flight line number (note this is not necessarily the same as the flight line identifier as specified by the PI - these numbers are the number of the flight in order of when it was taken, as on the logsheet)

Leica LIDAR data

subdir of "leica"	2nd level dir	Files	Comments
aux_files		*.reg	Reg file used, any logs produced, intensity mosaics, boresight values)
		*.tif
		*.txt
ipas
	base
	eo
	extract
	gps
	proc		Processed GPS/IMU data (SBETs)
	raw		Raw GPS/IMU data
logs			Contains the flight/ops log files
proc_laser			Contains the finalised processed LAS files only
raw_laser			Contains the raw laser data
rawWFD			Raw waveform data
rcd
	raw_images
	logs
webcam

Delivered data

As data may be delivered multiple times (in the event of a problem requiring reprocessing) or in stages (if particular sensors are completed first and others look like they'll be delayed), we take copies of any data sent to the end user for future reference (e.g. in support). The lev1/ and lev3/ directories will typically contain the latest processed version.

Delivered data is stored delivery/YYYYMMDD/PROJECTCODE/, where YYYYMMDD is the date the delivery was created. The directory structure of a delivery looks like:

Hyperspectral data

bin

latest azgcorr and azexhdf at the time of delivery

dem (optional)

a dem to help the user with processing to lev3. Can be from their lidar data, from SRTM or a combination.

doc

azgcorr manual and other documentation

lev1

processed level 1 data

logsheet

logsheet as stored in admin/ above

misc

any additional files needed for level 3 processing (e.g. OS National Grid correction file, geoid-spheroid separation grid file, etc)

screenshots

images of the level 3 output we managed when testing the data

COPYRIGHT.txt

copyright notice!

Read_Me.txt

Instructions for use and description of dataset, plus any problems encountered.

The image filenames in the lev1 directory are structured as sdddfnnLL.hdf, where:

• s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
• ddd is the Julian day of the data acquisition
• f distinguishes between multiple flights on the same day ("a" being the first flight)
• nn is the flight line number
• LL is the processing level
  • 1b = ungeocorrected data
  • 3a = geocorrected without a DEM
  • 3b = geocorrected with a DEM

Lidar data

ascii_laser

classified ascii lidar point clouds

bin

latest versions of the pt_cloud_filter for linux and windows

dem

dem created from the lidar data

doc

lidar quality report and processing information

logsheet

logsheet as stored in admin/ above

screenshots

images of the dem and intensity mosaic

COPYRIGHT.txt

copyright notice!

Read_Me.txt

Instructions for use and description of dataset, plus any problems encountered.

Photographic data

doc

camera data quality report

eventfile (may be missing if camera crashed)

csv file containing pos/att info per photograph event

photographs

tagged tif files of each photograph in area of interest

thumbnails

jpeg thumbnails of each photograph in area of interest

COPYRIGHT.txt

copyright notice!

proj_code.kml (unavailable if no eventfile)

google earth file containing eagle/hawk/photo positions

Read_Me.txt

Instructions for use and description of dataset, plus any problems encountered.