File structure and naming

The old structure is located here.

A single site/logical group of flight lines/sortie should be contained in a single directory. If there are multiple sites from different projects, significantly different times of overflight or large time gaps, each should have its own directory.

The directory name should look like: PROJECTCODE-YYYY_JJJxx_SITENAME, e.g. GB07_07-2007_102a_Inverclyde, boresight-2007_198, etc.

There should be:

• no spaces, brackets or other Unix-upsetting characters in filenames
• execute permission removed from anything non-executable
• no empty directories? (need to think about this, maybe only do after checking it's not a stub for later data)

The structure under the project directory should look like:

|

Top Level Dirs	2nd Level Dirs	3rd Level Dirs	Example File Formats	Comments
		Config file used in current processing procedure
		(eagle|hawk).mos	Eagle and Hawk mosaics of lev3 data
		qsub_comm.sh (deprecated)	Script for submitting processing to gridnodes, used in earlier processing procedure
admin			Contains the flight logsheet and any miscellaneous information (readme's, copies of trac tickets, etc)
		GB08_02-2009_078a_Delamere.(doc|txt|pdf)	Logsheet
		GB08_02-2009_078a_Delamere.fpd	Flight planning document
	delivery_readmes	Read_Me-20100809.(aux|log|tex)	Contains intermediate files used to create the ReadMe(s)
		(cam|hyp|lidar)_genreadme.cfg	Configuration files generated by generate_readme_config.py
atm (deprecated)			Raw ATM data files (az???LLL.JJJ, where ??? = disk id, LLL = 3 digit file number (000 typically contains navigation), JJJ = Julian day of capture), pre-2009 only
calibration			Typically a symlink to the appropriate calibration directory
camera
	rcd
			090803-100306ImageEvents1-processed.csv	Processed events file
		logs
			camera_delivery.log
			090803-100306ImageEvents(1|2).csv
			090803-145404PhotoId(1|2).csv
			090803-100306SensorLog.csv
			090803-145404SensorStat(1|2).csv
		proc_images	16093417100001G(3|4).tif
		proc_thumbnails	16093417100001G(3|4).jpg
		raw_images	16093417100001G(3|4).raw
		webcam		100715_124101_000.jpg	Webcam image
				20100715_105036_evt.csv	Webcam event file
casi (deprecated)			Raw CASI data files (ca??LLL.JJJ, where ?? = disk id, LLL = 3 digit file number, JJJ = Julian day of capture), pre-2008 only
delivery			Contains copies of data delivered to end users
	(see below)		Structure described below
dem			Contains azgcorr compatible DEMs
		VNIR078a-09-1.hdr	ENVI compatible header
		VNIR078a-09-1.log	Log files
		VNIR078a-09-1.nav	Navigation synchronisation data
		VNIR078a-09-1.raw	Uncalibrated image data in .bil format
	orignal_headers (optional)	VNIR078a-GB08-02.hdr-orig	If the header files have been altered for any reason, the originals should be in here
grid_logs (deprecated)			Logs from hyperspectral script runs on gridnode (one log per sct run)
		dataCube.raw
		SWIR078a-09-1.hdr	ENVI compatible header
		SWIR078a-09-1.log	Log files
		SWIR078a-09-1.nav	Navigation synchronisation data
		SWIR078a-09-1.raw	Uncalibrated image data in .bil format
	original_headers (optional)	SWIR078a-GB08-02.hdr-orig	If the header files have been altered for any reason, the originals should be in here
			e078a013b_n_sct-0.03_p_gpt0.0.tif
hyperspectral
	eagle			Raw Eagle data files
	hawk			Raw Hawk data files (.raw = image data, .hdr = ENVI compatible header, .nav = navigation synchronisation data)
lidar
	als50
		aux_files		Reg file used, any logs produced, intensity mosaics, boresight values
			lidar_2009078a.mos	ENVI mosaic of lidar lines intensity
			lidar_2009078a.mos.hdr	ENVI mosaic header
			2009078a.reg	Contains ALSPP settings
			lidar_2009078a.tif	Lidar intensity image
			line01_2009078a.tif	Individual flightline intensity image
		logs		Contains the flight/ops log files
			100411_084840_FlightLineLog.csv
			20091123212501_LogData.mdb
			091005_LogFile.txt
			090812_DLLogFile.txt
			20091123212501_LogDataSysDesc.xml
	proc_laser			Contains the finalised processed LAS files
			LDR091026_060148_1.LAS	LAS file
			pplog_20100226_122027.txt	log output from ALSPP
			stats.txt	info from ALSPP about the LAS files
	raw_laser (deprecated)			Contains the raw laser data
		090601_111411
			LDR090601_111411_0000000001.scn
	RawLaser			Contains the raw laser data (and works with FW processing)
		090601_111411		Contains raw laser for one contiguous recording
			LDR090601_111411_0000000001.scn	Raw LIDAR scan files
	RawWFD			Raw waveform data
		100706_094620
			WFD100706_094620_0000000001A.LWV
logs		*.logs	Logs from hyperspectral script runs on gridnode (one log per flightline)
posatt			position/attitude data
	applanix_litton		data from applanix using the litton imu
		2009078a.ppc	POSPac project file
	base (deprecated)		?
	eo (possibly optional and often empty)		?
	extract		Plane gps data after extracted from raw files by applanix
		dephem_2009078a.dat
		extract_2009078a.log
		gps_prim_2009078a.dat
		hwconf_2009078a.out
		iinr_2009078a.out
		iinz_2009078a.out
		imu_2009078a.dat
		mgps_2009078a.epp
		mgps_2009078a.gpb	POSGNSS friendly file format
		mgps_2009078a.nov
		mgps_2009078a.sta
		navclk_prim_2009078a.dat
		obs_prim_2009078a.dat
		rers_2009078a.out
		rinv_2009078a.out
		rmrs_2009078a.out
		rrms_2009078a.out
		rtstat_2009078a.txt
		tm_2009078a.dat
		vnav_2009078a.out
		vrms_2009078a.out
		*GNSS.dat
		gps	Files associated with POSGNSS
		2009078a.cmb	Combined
		2009078a.fbv
		2009078a.fml
		2009078a.fsl	PPP log file
		2009078a.fss
		2009078a.fwd
		2009078a.gnv	POSGNSS Project file
		2009078a.his
		2009078a.rbv
		2009078a.rev
		2009078a.rml
		2009078a.rsl	PPP log file
		2009078a.rss
	proc		Processed applanix data
		ers_2009078a.out
		idx_2009078a.txt
		inv_2009078a.out
		iin_2009078a.out
		mrs_2009078a.out
		rms_2009078a.out
		rmsg_2009078a.out
		sbet_2009078a.out	Main output file
		sbet_2009078.out
		smhkm_2009078a.dat
		smindex_2009078a.dat
		smout_2009078a.out
		smparm_2009078a.txt
		smphi_2009078a.dat
		smpx_2009078a.dat
		smreset_2009078a.dat
		smrms_2009078a.out
		smrmsg_2009078a.out
	raw	DCALM.001	Raw gps data from plane
	basestation		basestation data
		oxfr154g30.09n
		oxfr154g30.09o
		oxfr154g30.epp
		oxfr154g30.gpb	POSGNSS friendly file format
		oxfr154g30.sta	Text file with info about basestation
	ipas_honeywell		data from ipas using the honeywell imu
	*.gpj	IPAS Pro Project File
		*.itp	IPAS TC Project File
	base		Base station data
		oxfr154g30.09n
		oxfr154g30.09o
		oxfr154g30.epp
		oxfr154g30.gpb	Gravnav friendly file format
		oxfr154g30.sta	Text file with info about basestation
	eo
	extract
		*.epp
		*.EVT
		*_extractLog.txt
		*.fbv
		*.rbv
		*.gpb	Grafnav friendly file format
		*.GPS
		*.IMU
		*.RNV
		*.RTG
		*_RTMsg.txt
		*.sta
		*.SUP
		*.tm
	gps
		*.cfg	Grafnav project file
		*.fml
		*.fss
		*.fsl	PPP log file
		*.fwd
		*.his
		*.lat	grafnav final output which is used by IPAS (text version)
		*.lat.bin	grafnav final output which is used by IPAS
		*.rev
		*.rml
		*.rsl	PPP log file
		*.rss
	proc		Processed ipas nav data
		20090319_142319.sol	Final output file
		sbet_20090319_from_sol.out	applanix sbet created from sol file
		20090319_142319.ers
		20090319_142319.sup
		2009078a_procLog.txt
		20090319_142319.iqc	IPAS TC quality control file
	raw	20090319_111251.001	Raw GPS/IMU data
processing		0532010.cfg	config file
	hyperspectral
	lidar
	camera
tabi			Raw TABI data

The lev1 and lev3 data filenames are structured as "sdddfnn1b.{hdf|tif}", where:

• s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
• ddd is the Julian day of the data acquisition
• f distinguishes between multiple flights on the same day (“a” being the first flight)
• nn is the flight line number (note this is not necessarily the same as the flight line identifier as specified by the PI - these numbers are the number of the flight in order of when it was taken, as on the logsheet)

Delivered data

As data may be delivered multiple times (in the event of a problem requiring reprocessing) or in stages (if particular sensors are completed first and others look like they'll be delayed), we take copies of any data sent to the end user for future reference (e.g. in support). The lev1/ and lev3/ directories will typically contain the latest processed version.

Delivered data is stored delivery/YYYYMMDD/PROJECTCODE/, where YYYYMMDD is the date the delivery was created. The directory structure of a delivery looks like:

Hyperspectral deliveries

Dirs	Example Files	Comments
	COPYRIGHT.txt (deprecated - now in Readme)	Copyright notice
	Read_Me-20100202.(txt|pdf)	Instructions for use and description of dataset, plus any problems encountered.
bin (deprecated - now in readme)		Instructions for downloading azgcorr
dem (optional)	GB08_19-2010_254b-lidar-ASTER.dem	A dem to help the user with processing to lev3. Can be from their lidar/SRTM/ASTER or a combination (not from NEXTMAP data - license restrictions).
doc		azgcorr manual and other documentation
	azgcorr_geocorrection.pdf
	azgcorr_manual.pdf
	data_quality-20100525.pdf
	README-manual_outdated.txt
lev1		Processed level 1 data
	(e|h)078a031b.bil	Level 1 image data
	(e|h)078a031b.bil.hdr	Header file for image data
	(e|h)078a031b.hdf	Metadata for image
logsheet		Copy of pdf version of logsheet
	FW10_01-2010_187_Dennys_wood.pdf
misc (optional)		Any additional files needed for level 3 processing (e.g. OS National Grid correction file, geoid-spheroid separation grid file, etc)
	osgb02
	sphsep15lx.grd
screenshots		Images of the level 3 output we managed when testing the data
	(e|h)078a013b.jpg	Individual line screenshot
	(eagle|hawk)_mosaic.jpg	Screenshot of all level 3 data
	(eagle|hawk_mosaic_with_vectors.jpg	Screenshot of all level 3 data with overlaid vectors

The image filenames in the lev1 directory are structured as sdddfnnLL.hdf, where:

• s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
• ddd is the Julian day of the data acquisition
• f distinguishes between multiple flights on the same day ("a" being the first flight)
• nn is the flight line number
• LL is the processing level
  • 1b = ungeocorrected data
  • 3a = geocorrected without a DEM
  • 3b = geocorrected with a DEM

Lidar deliveries

Dirs	Sub Dirs	Example Files	Comments
		COPYRIGHT.txt (deprecated - now in Readme)	Copyright notice
		Read_Me-20100202.(txt|pdf)	Instructions for use and description of dataset, plus any problems encountered
bin			Latest versions of the pt_cloud_filter for linux and windows
dem		GB08_19-2010_254b-lidar_ASTER-wgs84_utm.dem	DEM created from the lidar data. Can be from their lidar/SRTM/ASTER or a combination (not from NEXTMAP data - license restrictions). DEM is in same co-ordinate system as processed LAS files.
		GB08_19-2010_254b-lidar_ASTER-wgs84_latlong.dem	Above LiDAR dem converted to wgs84_latlong (compatible with apl)

doc		lidar-quality-report_20110519.pdf	Lidar quality report
flightlines			Classified lidar point clouds
	ascii	LDR-FW10_02-201010101.txt	ASCII point clouds
	las1.0	LDR-FW10_02-201010101.LAS	LAS1.0 point clouds
	las1.3 (FW only)	LDR-FW-FW10_02-201009801.LAS	LAS1.3 point clouds
fw_extractions (FW only)			Full waveform area extractions
		FW10_02-2010_101_extractions.jpg
		FW10_02-2010_101_extractions.txt
	area1
		waveform_42437_649348_only.txt ?
logsheet			Logsheet as stored in admin/ above
screenshots			Images of the dem and intensity mosaic
		GB08_19-2010_254b_intensity.jpg
		GB08_19-2010_254b_intensity_vectors.jpg
		GB08_19-2010_254b_dem.jpg

Photographic deliveries

Dirs	Example Files	Comments
	COPYRIGHT.txt (deprecated - now in Readme)	Copyright document
	GB08_02-2009_078a.kml	Google earth file containing eagle/hawk/photo positions
	Read_Me-20100202.(txt|pdf)	Instructions for use and description of dataset, plus any problems encountered
doc	data_quality-20091202.pdf	Camera data quality report
	RCD105_Calibration_Certificate.pdf (2010 only)	Camera Calibration Certificate
eventfile	RCD105-GB08_02-2009078a-ImageEvents.csv	CSV file containing pos/att info per photograph event (may be missing if camera crashed)
photographs	RCD105-GB08_02-2009078a-00001.tif	Tagged tif files of each photograph in area of interest
thumbnails	RCD105-GB08_02-2009078a-00001.jpg	JPEG thumbnails of each photograph in area of interest