File structure and naming

This page denotes the 2011 project structure. The current structure can be found here.

A single site/logical group of flight lines/sortie should be contained in a single directory. If there are multiple sites from different projects, significantly different times of overflight or large time gaps, each should have its own directory.

The directory name should look like: PROJECTCODE-YYYY_JJJxx_SITENAME, e.g. GB07_07-2007_102a_Inverclyde, boresight-2007_198, etc.

There should be:

• no spaces, brackets or other Unix-upsetting characters in filenames
• execute permission removed from anything non-executable
• no empty directories? (need to think about this, maybe only do after checking it's not a stub for later data)

The structure under the project directory should look like:

Top Level Dirs	2nd Level Dirs	3rd Level Dirs	4th Level Dirs	5th Level Dirs	6th Level Dirs	Example File Formats	Comments
admin							Contains the flight logsheet and any miscellaneous information (readme's, copies of trac tickets, etc)
						GB08_02-2009_078a_Delamere.(doc|txt|pdf)	Logsheet
						GB08_02-2009_078a_Delamere.fpd	Flight planning document
	delivery_readmes					Read_Me-20100809.(aux|log|tex)	Intermediate files used to create the ReadMe(s)
						(cam|hyp|lidar)_genreadme.cfg	Configuration files generated by generate_readme_config.py
calibration							Typically a symlink to the appropriate calibration directory
camera							Camera-type sensors
	rcd						RCD digital camera
						090803-100306ImageEvents1-processed.csv	Processed events file
		logs
						camera_delivery.log
						090803-100306ImageEvents(1|2).csv
						090803-145404PhotoId(1|2).csv
						090803-100306SensorLog.csv
						090803-145404SensorStat(1|2).csv
		raw_images				16093417100001G(3|4).raw
		webcam				100715_124101_000.jpg	Webcam image
						20100715_105036_evt.csv	Webcam event file
hyperspectral
	eagle						Raw Eagle data files
						dataCube.raw
						VNIR078a-09-1.hdr	ENVI compatible header
						VNIR078a-09-1.log	Log files
						VNIR078a-09-1.nav	Navigation synchronisation data
						VNIR078a-09-1.raw	Uncalibrated image data in .bil format
	hawk						Raw Hawk data files
						dataCube.raw
						SWIR078a-09-1.hdr	ENVI compatible header
						SWIR078a-09-1.log	Log files
						SWIR078a-09-1.nav	Navigation synchronisation data
						SWIR078a-09-1.raw	Uncalibrated image data in .bil format
lidar	als50
		aux_files					Reg file used, any logs produced, intensity mosaics, boresight values
						lidar_2009078a.mos	ENVI mosaic of lidar lines intensity
						lidar_2009078a.mos.hdr	ENVI mosaic header
						2009078a.reg	Contains ALSPP settings
						lidar_2009078a.tif	Lidar intensity image
						line01_2009078a.tif	Individual flightline intensity image
		logs					Contains the flight/ops log files
						100411_084840_FlightLineLog.csv
						20091123212501_LogData.mdb
						091005_LogFile.txt
						090812_DLLogFile.txt
						20091123212501_LogDataSysDesc.xml
		RawLaser					Contains the raw laser data
			090601_111411				Contains raw laser for one contiguous recording
						LDR090601_111411_0000000001.scn	Raw LIDAR scan files
		RawWFD					Raw waveform data
			100706_094620
						WFD100706_094620_0000000001A.LWV
logfiles						EU11_06_2011-098_eagle_-3.o2810538	Logs from hyperspectral script runs on gridnode (one log per flightline)
posatt							Position/attitude data
	applanix_litton						Data from applanix using the litton imu
						2009078a.ppc	POSPac project file
		extract					Plane gps data after extracted from raw files by applanix
						dephem_2009078a.dat
						extract_2009078a.log
						gps_prim_2009078a.dat
						hwconf_2009078a.out
						iinr_2009078a.out
						iinz_2009078a.out
						imu_2009078a.dat
						mgps_2009078a.epp
						mgps_2009078a.gpb	POSGNSS friendly file format
						mgps_2009078a.nov
						mgps_2009078a.sta
						navclk_prim_2009078a.dat
						obs_prim_2009078a.dat
						rers_2009078a.out
						rinv_2009078a.out
						rmrs_2009078a.out
						rrms_2009078a.out
						rtstat_2009078a.txt
						tm_2009078a.dat
						vnav_2009078a.out
						vrms_2009078a.out
						*GNSS.dat
		gps					Files associated with POSGNSS
						2009078a.cmb	Combined
						2009078a.fbv
						2009078a.fml
						2009078a.fsl	PPP log file
						2009078a.fss
						2009078a.fwd
						2009078a.gnv	POSGNSS Project file
						2009078a.his
						2009078a.rbv
						2009078a.rev
						2009078a.rml
						2009078a.rsl	PPP log file
						2009078a.rss
		proc					Processed applanix data
						ers_2009078a.out
						idx_2009078a.txt
						inv_2009078a.out
						iin_2009078a.out
						mrs_2009078a.out
						rms_2009078a.out
						rmsg_2009078a.out
						sbet_2009078a.out	Main output file
						sbet_2009078.out
						smhkm_2009078a.dat
						smindex_2009078a.dat
						smout_2009078a.out
						smparm_2009078a.txt
						smphi_2009078a.dat
						smpx_2009078a.dat
						smreset_2009078a.dat
						smrms_2009078a.out
						smrmsg_2009078a.out
		raw				DCALM.001	Raw gps data from plane
	basestation						Basestation data
						oxfr154g30.09n
						oxfr154g30.09o
						oxfr154g30.epp
						oxfr154g30.gpb	POSGNSS friendly file format
						oxfr154g30.sta	Text file with info about basestation
	ipas_honeywell						Data from ipas using the honeywell imu
						*.gpj	IPAS Pro Project File
						*.itp	IPAS TC Project File
		extract
						*.epp
						*.EVT
						*_extractLog.txt
						*.fbv
						*.rbv
						*.gpb	Grafnav friendly file format
						*.GPS
						*.IMU
						*.RNV
						*.RTG
						*_RTMsg.txt
						*.sta
						*.SUP
						*.tm
		gps
						*.cfg	Grafnav project file
						*.fml
						*.fss
						*.fsl	PPP log file
						*.fwd
						*.his
						*.lat	Grafnav final output which is used by IPAS (text version)
						*.lat.bin	Grafnav final output which is used by IPAS
						*.rev
						*.rml
						*.rsl	PPP log file
						*.rss
		proc					Processed ipas nav data
						20090319_142319.sol	Final output file
						sbet_20090319_from_sol.out	Applanix sbet created from sol file
						20090319_142319.ers
						20090319_142319.sup
						2009078a_procLog.txt
						20090319_142319.iqc	IPAS TC quality control file
		raw				20090319_111251.001	Raw GPS/IMU data
processing
	als50
		las					Contains the finalised processed LAS files
						LDR091026_060148_1.LAS	LAS file
						pplog_20100226_122027.txt	Log output from ALSPP
						stats.txt	Info from ALSPP about the LAS files
		las-classified					Contains the classified LAS files for delivery
						LDR091026_060148_1.LAS	LAS file
	delivery						Contains copies of data delivered to end users
		EUFAR11_06-098-hyperspectral-20110621	Delivery structures described below				Delivery folder
	hyperspectral					0532010.cfg	Config file
		dem
		flightlines
			georeferencing
				igm
				mapped		e078a013b_n_sct-0.03_p_gpt0.0.tif	Geo-rectified images
			level 1b
			line information
			navigation
				interpolated
					post_processed
					real_time
	rcd
		proc_images				16093417100001G(3|4).tif
		thumbnails				16093417100001G(3|4).jpg
tabi							Raw TABI data

The lev1 and lev3 data filenames are structured as "sdddfnn1b.{hdf|tif}", where:

• s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
• ddd is the Julian day of the data acquisition
• f distinguishes between multiple flights on the same day (“a” being the first flight)
• nn is the flight line number (note this is not necessarily the same as the flight line identifier as specified by the PI - these numbers are the number of the flight in order of when it was taken, as on the logsheet)

Delivered data

As data may be delivered multiple times (in the event of a problem requiring reprocessing) or in stages (if particular sensors are completed first and others look like they'll be delayed), we take copies of any data sent to the end user for future reference (e.g. in support). The lev1/ and lev3/ directories will typically contain the latest processed version.

Delivered data is stored delivery/YYYYMMDD/PROJECTCODE/, where YYYYMMDD is the date the delivery was created. The directory structure of a delivery looks like:

Hyperspectral deliveries

Top level Dir	2nd level Dir	Example Files	Comments
		Read_Me-20100202.pdf	Instructions for use and description of dataset, plus any problems encountered.
dem			A dem to help the user with processing to lev3. Should be ASTER.
		GB08_19-2010-254b-ASTER.dem	ASTER binary dem
		GB08_19-2010-254b-ASTER.hdr	Header file for DEM
doc			data quality report and other documentation
		data_quality-20100525.pdf	data quality report
flightlines
	fodis		Data for the FODIS instrument
		e220041b_FODIS.bil
		e220041b_FODIS.bil.hdr
	level1b		Level 1 image data
		(e|h)078a031b.bil	ENVI bil file of level 1 data
		(e|h)078a031b.bil.hdr	Header file for image data
		e223041b_mask.bil	An image showing masked pixels
		e223041b_mask.bil.hdr	Header file for mask image
		h223041b_badpixels.txt	List of bad pixels (Hawk only)
	line_information	(e|h)221021b.xml	metadata for each line
	navigation		Navigation data for each line
		e098101b_nav_post_processed.bil	Envi bil file with navigation data
		e098101b_nav_post_processed.bil.hdr	Header file for navigation
		e098101b_nav_post_processed_quality.bil	Bil file indicating quality of navigation file
		e098101b_nav_post_processed_quality.bil.hdr	Header file for quality file
logsheet		FW10_01-2010_187_Dennys_wood.pdf	Copy of pdf version of logsheet
misc (optional)			Any additional files needed for level 3 processing (e.g. OS National Grid correction file, geoid-spheroid separation grid file, etc)
		osgb02
		sphsep15lx.grd
project_information		BB11_01-2011_244a-project.xml	Meta data for the project as a whole
screenshots			Images of the level 3 output we created when testing the data
		(e|h)078a013b.jpg	Individual line screenshot
		(eagle|hawk)_mosaic.jpg	Screenshot of all level 3 data
		(eagle|hawk_mosaic_with_vectors.jpg	Screenshot of all level 3 data with overlaid vectors (UK only)
sensor_FOV_vectors
		(eagle|hawk)_fov_fullccd_vectors.bil
		(eagle|hawk)_fov_fullccd_vectors.hdr

The image filenames in the lev1 directory are structured as sdddfnnLL.hdf, where:

• s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
• ddd is the Julian day of the data acquisition
• f distinguishes between multiple flights on the same day ("a" being the first flight)
• nn is the flight line number
• LL is the processing level
  • 1b = ungeocorrected data
  • 3a = geocorrected without a DEM
  • 3b = geocorrected with a DEM

Lidar deliveries

Dirs	Sub Dirs	Example Files	Comments
		Read_Me-20100202.pdf	Instructions for use and description of dataset, plus any problems encountered
bin			Latest versions of the pt_cloud_filter for linux and windows
dem			DEMs created from the LiDAR
		GB08_19-2010-254b-lidar_ASTER-wgs84_utm.dem	LiDAR dem patched with ASTER data (do not use NEXTMAP data - license restrictions). DEM is in same co-ordinate system as processed LAS files.
		GB08_19-2010-254b-lidar_ASTER-wgs84_utm.hdr
		GB08_19-2010-254b-lidar_ASTER-wgs84_latlong.dem	Above LiDAR dem converted to wgs84_latlong (compatible with apl)
		GB08_19-2010-254b-lidar_ASTER-wgs84_latlong.hdr
		README_DEM_MAY_NOT_BE_ACCURATE.txt	Warning to end user that there are potential flaws with the provided DEMs
doc		lidar-quality-report_20110519.pdf	Lidar quality report
flightlines			Classified lidar point clouds
	ascii	LDR-EU11_03-2011-102-10.txt	ASCII point clouds
	las1.0	LDR-EU11_03-2011-102-10.LAS	LAS1.0 point clouds
	las1.3 (FW only)	LDR-FF-EU11_03-201110201.LAS	LAS1.3 point clouds
fw_extractions (FW only)			Full waveform area extractions
		FW10_02-2010_101_extractions.jpg
		FW10_02-2010_101_extractions.txt
	area1
		waveform_42437_649348_only.txt ?
logsheet		GB08_02-2009_078a_Delamere.pdf	Flight logsheet, typed up or created by generate_logsheet.py
screenshots			Images of the dem and intensity mosaic
		GB08_19-2010-254b_intensity.jpg
		GB08_19-2010-254b_intensity_vectors.jpg	UK only
		GB08_19-2010-254b_dem.jpg

Photographic deliveries

Dirs	Example Files	Comments
	GB08_02-2009_078a.kml	Google earth file containing eagle/hawk/photo positions
	Read_Me-20100202.pdf	Instructions for use and description of dataset, plus any problems encountered
doc		Useful documents
	data_quality-20091202.pdf	Camera data quality report
	RCD105_Calibration_Certificate.pdf	Camera Calibration Certificate
eventfile	RCD105-GB08_02-2009078a-ImageEvents.csv	CSV file containing pos/att info per photograph event (may be missing if camera crashed)
photographs	RCD105-GB08_02-2009078a-00001.tif	Tagged tif files of each photograph in area of interest
thumbnails	RCD105-GB08_02-2009078a-00001.jpg	JPEG thumbnails of each photograph in area of interest