File structure and naming

A single site/logical group of flight lines/sortie should be contained in a single directory. If there are multiple sites from different projects, significantly different times of overflight or large time gaps, each should have its own directory.

The directory name should look like: PROJECTCODE-YYYY_JJJxx_SITENAME, e.g. GB07_07-2007_102a_Inverclyde, boresight-2007_198, etc.

There should be:

• no spaces, brackets or other Unix-upsetting characters in filenames
• execute permission removed from anything non-executable
• no empty directories? (need to think about this, maybe only do after checking it's not a stub for later data)

The structure under the project directory should look like:

Top Level Dirs	2nd Level Dirs	Example File Formats	Comments
		0532010.cfg	Config file used in current processing proceedure
		(eagle|hawk).mos	Eagle and Hawk mosaics of lev3 data
		qsub_comm.sh (deprecated)	Script for submitting processing to gridnodes, used in earlier processing proceedure
admin			Contains the flight logsheet and any miscellaneous information (readme's, copies of trac tickets, etc)
		GB08_02-2009_078a_Delamere.(doc|txt|pdf)	Logsheet
		GB08_02-2009_078a_Delamere.fpd	Flight planning document
applanix		*.ppc	Raw and processed Applanix navigation (raw files in applanix/raw/DCALM*, final processed data in applanix/*sbet or applanix/Proc/sbet*.out)
	auto_applanix (optional)	auto.bat	If navigation was processed using auto_applanix script then this dir will contain additional output from that script
		combine_seperation_bestguess_values
		normal_first_convertbase.log
		normal_first_convertflight.log
		*.bat
		*GNSS.cmb
		*GNSS.dat
		*GNSS_gnuplot
		*GNSS.gnv
		*GNSS.log
		*GNSS.png
		*GNSS.sum
	base (deprecated)		?
	eo (possibly optional and often empty)		?
	extract	dephem_*.dat	Plane gps data after extracted from raw files by applanix
		extract_*.log
		gps_prim_*.dat
		hwconf_*.out
		iinr_*.out
		iinz_*.out
		imu_*.dat
		mgps_*.epp
		mgps_*.gpb
		mgps_*.nov
		mgps_*.sta
		navclk_prim_*.dat
		obs_prim_*.dat
		rers_*.out
		rinv_*.out
		rmrs_*.out
		rrms_*.out
		rtstat_*.txt
		tm_*.dat
		vnav_*.out
		vrms_*.out
		*GNSS.dat
	gps	*.cmb	?
		*.fbv
		*.fml
		*.fss
		*.fwd
		*.his
		*.rbv
		*.rev
		*.rml
		*.rss
	proc	ers_*.out	Processed applanix data
		idx_*.txt
		inv_*.out
		iin_*.out
		mrs_*.out
		rms_*.out
		rmsg_*.out
		sbet_*.out
		smers_*.out
		smhkm_*.dat
		smindex_*.dat
		smout_*.out
		smparm_*.txt
		smphi_*.dat
		smpx_*.dat
		smreset_*.dat
		smrms_*.out
		smrmsg_*.out
	raw	DCALM.001	Raw gps data from plane
atm			Raw ATM data files (az???LLL.JJJ, where ??? = disk id, LLL = 3 digit file number (000 typically contains navigation), JJJ = Julian day of capture), pre-2009 only
calibration			Typically a symlink to the appropriate calibration directory
casi			Raw CASI data files (ca??LLL.JJJ, where ?? = disk id, LLL = 3 digit file number, JJJ = Julian day of capture), pre-2008 only
delivery			Contains copies of data delivered to end users
	(see below)		Structure described below
dem			Contains azgcorr compatible DEMs
eagle			Raw Eagle data files
		VNIR078a-09-1.hdr	ENVI compatible header
		VNIR078a-09-1.log	Log files
		VNIR078a-09-1.nav	Navigation synchronisation data
		VNIR078a-09-1.raw	Uncalibrated image data in .bil format
	orignal_headers (optional)		If the header files have been altered for any reason, the originals should be in here
grid_logs (deprecated)			Logs from hyperspectral script runs on gridnode (one log per sct run)
hawk			Raw Hawk data files (.raw = image data, .hdr = ENVI compatible header, .nav = navigation synchronisation data)
		SWIR078a-09-1.hdr	ENVI compatible header
		SWIR078a-09-1.log	Log files
		SWIR078a-09-1.nav	Navigation synchronisation data
		SWIR078a-09-1.raw	Uncalibrated image data in .bil format
	original_headers (optional)		If the header files have been altered for any reason, the originals should be in here
leica			Leica LIDAR data
	(see below)		Structure described below
lev1		(see below)	Processed hyperspectral level 1 data (should be empty if data has been moved into delivery)
lev3			Processed hyperspectral level 3 data
		e078a013b_n_sct-0.03_p_gpt0.0.tif
lidar (deprecated)			Raw and processed ULM LIDAR (raw LIDAR looks like "surveyXXX.range", processed is typically in a strips/ directory)
logs		*.logs	Logs from hyperspectral script runs on gridnode (one log per flightline)
photos (deprecated)			RC10 scanned aerial photographs
rinex			GPS basestation data appropriate to the site
runa (deprecated)			ATM processing scripts
runc (deprecated)			CASI processing scripts
rune (deprecated)			Eagle processing scripts
runh (deprecated)			Hawk processing scripts
tabi			Raw TABI data
vectors			(2007/8 only) - symlink to appropriate vectors

The lev1 and lev3 data filenames are structured as "sdddfnn1b.{hdf|tif}", where:

• s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
• ddd is the Julian day of the data acquisition
• f distinguishes between multiple flights on the same day (“a” being the first flight)
• nn is the flight line number (note this is not necessarily the same as the flight line identifier as specified by the PI - these numbers are the number of the flight in order of when it was taken, as on the logsheet)

Leica LIDAR data

subdir of "leica"	2nd level dir	Files	Comments
aux_files		*.reg	Reg file used, any logs produced, intensity mosaics, boresight values)
		*.tif
		*.txt
ipas
	base
	eo
	extract	*.fbv
		*.rbv
		*.RNV
		*.RTG
	gps	*.cfg
		*.fml
		*.fss
		*.fwd
		*.gpj
		*.his
		*.lat
		*.lat.bin
		*.rev
		*.rml
		*.rss
	proc		Processed GPS/IMU data (SBETs)
		sbet_20091025_from_sol.out	sbet created from sol file
	raw		Raw GPS/IMU data
logs		*.csv	Contains the flight/ops log files
		*.mdb
		*.txt
		*.xml
proc_laser			Contains the finalised processed LAS files only
raw_laser (deprecated)			Contains the raw laser data
RawLaser			Contains the raw laser data (and works with FW processing)
	090601_111411		Contains raw laser for one contiguous recording
		LDR090601_111411_0000000001.scn	Raw LIDAR scan files
RawWFD			Raw waveform data
	100706_094620
		WFD100706_094620_0000000001A.LWV
rcd
	raw_images
	logs
webcam

Delivered data

As data may be delivered multiple times (in the event of a problem requiring reprocessing) or in stages (if particular sensors are completed first and others look like they'll be delayed), we take copies of any data sent to the end user for future reference (e.g. in support). The lev1/ and lev3/ directories will typically contain the latest processed version.

Delivered data is stored delivery/YYYYMMDD/PROJECTCODE/, where YYYYMMDD is the date the delivery was created. The directory structure of a delivery looks like:

Hyperspectral data

Dirs	Example Files	Comments
	Read_Me-20100202.(txt|pdf)	Instructions for use and description of dataset, plus any problems encountered.
bin (deprecated)		Instructions for downloading azgcorr
dem (optional)		A dem to help the user with processing to lev3. Can be from their lidar data, from SRTM or a combination (not from NEXTMAP data).
doc		azgcorr manual and other documentation
	azgcorr_geocorrection.pdf
	azgcorr_manual.pdf
	data_quality-20100525.pdf
	README-manual_outdated.txt
lev1		Processed level 1 data
	(e|h)078a031b.bil	Level 1 image data
	(e|h)078a031b.bil.hdr	Header file for image data
	(e|h)078a031b.hdf	Metadata for image
logsheet		Copy of pdf version of logsheet
	FW10_01-2010_187_Dennys_wood.pdf
misc		Any additional files needed for level 3 processing (e.g. OS National Grid correction file, geoid-spheroid separation grid file, etc)
	osgb02
	sphsep15lx.grd
screenshots		Images of the level 3 output we managed when testing the data
	(e|h)078a013b.jpg	Individual line screenshot
	(eagle|hawk)_mosaic.jpg	Screenshot of all level 3 data
	(eagle|hawk_mosaic_with_vectors.jpg	Screenshot of all level 3 data with overlaid vectors

The image filenames in the lev1 directory are structured as sdddfnnLL.hdf, where:

• s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
• ddd is the Julian day of the data acquisition
• f distinguishes between multiple flights on the same day ("a" being the first flight)
• nn is the flight line number
• LL is the processing level
  • 1b = ungeocorrected data
  • 3a = geocorrected without a DEM
  • 3b = geocorrected with a DEM

Lidar data

Dirs	Example Files	Comments
	COPYRIGHT.txt	Copyright notice
	Read_Me-20100202.(txt|pdf)	Instructions for use and description of dataset, plus any problems encountered
ascii_laser		Classified ascii lidar point clouds
bin		Latest versions of the pt_cloud_filter for linux and windows
dem	GB08_19-2010-254b.dem	DEM created from the lidar data
discrete_laser (FW only)	(same as ascii_laser)	(same as ascii_laser)
doc		Lidar quality report and processing information
fw_extractions (FW only)		Full waveform area extractions
fw_laser (FW only)		Full waveform laser data
logsheet		Logsheet as stored in admin/ above
screenshots		Images of the dem and intensity mosaic
	GB08_19-2010-254b_intensity.jpg
	GB08_19-2010-254b_intensity_vectors.jpg
	GB08_19-2010-254b_dem.jpg

Photographic data

Dirs	Example Files	Comments
	GB08_02-2009_078a_Delamere.kml	Google earth file containing eagle/hawk/photo positions
	Read_Me-20100202.(txt|pdf)	Instructions for use and description of dataset, plus any problems encountered
doc		Camera data quality report
eventfile		CSV file containing pos/att info per photograph event (may be missing if camera crashed)
photographs		Tagged tif files of each photograph in area of interest
thumbnails		JPEG thumbnails of each photograph in area of interest