Version 196 (modified by emca, 13 years ago) (diff)

--

File structure and naming

A single site/logical group of flight lines/sortie should be contained in a single directory. If there are multiple sites from different projects, significantly different times of overflight or large time gaps, each should have its own directory.

The directory name should look like: PROJECTCODE-YYYY_JJJxx_SITENAME, e.g. GB07_07-2007_102a_Inverclyde, boresight-2007_198, etc.

There should be:

  • no spaces, brackets or other Unix-upsetting characters in filenames
  • execute permission removed from anything non-executable
  • no empty directories? (need to think about this, maybe only do after checking it's not a stub for later data)

The structure under the project directory should look like:

Top Level Dirs 2nd Level Dirs Example File Formats Comments
0532010.cfg Config file used in current processing procedure
(eagle|hawk).mos Eagle and Hawk mosaics of lev3 data
qsub_comm.sh (deprecated) Script for submitting processing to gridnodes, used in earlier processing procedure
admin Contains the flight logsheet and any miscellaneous information (readme's, copies of trac tickets, etc)
GB08_02-2009_078a_Delamere.(doc|txt|pdf) Logsheet
GB08_02-2009_078a_Delamere.fpd Flight planning document
delivery_readmes Read_Me-20100809.(aux|log|tex) Contains intermediate files used to create the ReadMe(s)
(cam|hyp|lidar)_genreadme.cfg Configuration files generated by generate_readme_config.py
applanix Raw and processed Applanix navigation (raw files in applanix/raw/DCALM*, final processed data in applanix/*sbet or applanix/Proc/sbet*.out)
2009078a.ppc POSPac project file
auto_applanix (optional) auto.bat If navigation was processed using auto_applanix script then this dir will contain additional output from that script
combine_seperation_bestguess_values
normal_first_convertbase.log
normal_first_convertflight.log
2009078a.bat
GNSS.cmb
*GNSS.dat
*GNSS_gnuplot
*GNSS.gnv
*GNSS.log
*GNSS.png
*GNSS.sum
base (deprecated) ?
eo (possibly optional and often empty) ?
extract Plane gps data after extracted from raw files by applanix
dephem_2009078a.dat
extract_2009078a.log
gps_prim_2009078a.dat
hwconf_2009078a.out
iinr_2009078a.out
iinz_2009078a.out
imu_2009078a.dat
mgps_2009078a.epp
mgps_2009078a.gpb POSGNSS friendly file format
mgps_2009078a.nov
mgps_2009078a.sta
navclk_prim_2009078a.dat
obs_prim_2009078a.dat
rers_2009078a.out
rinv_2009078a.out
rmrs_2009078a.out
rrms_2009078a.out
rtstat_2009078a.txt
tm_2009078a.dat
vnav_2009078a.out
vrms_2009078a.out
*GNSS.dat
gps Files associated with POSGNSS
2009078a.cmb Combined
2009078a.fbv
2009078a.fml
2009078a.fsl PPP log file
2009078a.fss
2009078a.fwd
2009078a.gnv POSGNSS Project file
2009078a.his
2009078a.rbv
2009078a.rev
2009078a.rml
2009078a.rsl PPP log file
2009078a.rss
proc Processed applanix data
ers_2009078a.out
idx_2009078a.txt
inv_2009078a.out
iin_2009078a.out
mrs_2009078a.out
rms_2009078a.out
rmsg_2009078a.out
sbet_2009078a.out Main output file
sbet_2009078.out
smhkm_2009078a.dat
smindex_2009078a.dat
smout_2009078a.out
smparm_2009078a.txt
smphi_2009078a.dat
smpx_2009078a.dat
smreset_2009078a.dat
smrms_2009078a.out
smrmsg_2009078a.out
raw DCALM.001 Raw gps data from plane
atm (deprecated) Raw ATM data files (az???LLL.JJJ, where ??? = disk id, LLL = 3 digit file number (000 typically contains navigation), JJJ = Julian day of capture), pre-2009 only
calibration Typically a symlink to the appropriate calibration directory
casi (deprecated) Raw CASI data files (ca??LLL.JJJ, where ?? = disk id, LLL = 3 digit file number, JJJ = Julian day of capture), pre-2008 only
delivery Contains copies of data delivered to end users
(see below) Structure described below
dem Contains azgcorr compatible DEMs
eagle Raw Eagle data files
VNIR078a-09-1.hdr ENVI compatible header
VNIR078a-09-1.log Log files
VNIR078a-09-1.nav Navigation synchronisation data
VNIR078a-09-1.raw Uncalibrated image data in .bil format
orignal_headers (optional) VNIR078a-GB08-02.hdr-orig If the header files have been altered for any reason, the originals should be in here
grid_logs (deprecated) Logs from hyperspectral script runs on gridnode (one log per sct run)
hawk Raw Hawk data files (.raw = image data, .hdr = ENVI compatible header, .nav = navigation synchronisation data)
dataCube.raw
SWIR078a-09-1.hdr ENVI compatible header
SWIR078a-09-1.log Log files
SWIR078a-09-1.nav Navigation synchronisation data
SWIR078a-09-1.raw Uncalibrated image data in .bil format
original_headers (optional) SWIR078a-GB08-02.hdr-orig If the header files have been altered for any reason, the originals should be in here
leica Leica LIDAR data
(see below) Structure described below
lev1 (see below) Processed hyperspectral level 1 data (should be empty if data has been moved into delivery)
lev3 Processed hyperspectral level 3 data
e078a013b_n_sct-0.03_p_gpt0.0.tif
lidar (deprecated) Raw and processed ULM LIDAR (raw LIDAR looks like "surveyXXX.range", processed is typically in a strips/ directory)
logs *.logs Logs from hyperspectral script runs on gridnode (one log per flightline)
photos (deprecated) RC10 scanned aerial photographs
rinex GPS basestation data appropriate to the site
oxfr154g30.09n
oxfr154g30.09o
oxfr154g30.epp
oxfr154g30.gpb POSGNSS friendly file format
oxfr154g30.sta Text file with info about basestation
runa (deprecated) ATM processing scripts
runc (deprecated) CASI processing scripts
rune (deprecated) Eagle processing scripts
runh (deprecated) Hawk processing scripts
tabi Raw TABI data
vectors (deprecated) (2007/8 only) - symlink to appropriate vectors

The lev1 and lev3 data filenames are structured as "sdddfnn1b.{hdf|tif}", where:

  • s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
  • ddd is the Julian day of the data acquisition
  • f distinguishes between multiple flights on the same day (“a” being the first flight)
  • nn is the flight line number (note this is not necessarily the same as the flight line identifier as specified by the PI - these numbers are the number of the flight in order of when it was taken, as on the logsheet)

Leica Directory

subdir of "leica" 2nd level dir Files Comments
aux_files Reg file used, any logs produced, intensity mosaics, boresight values
lidar_2009078a.mos ENVI mosaic of lidar lines intensity
lidar_2009078a.mos.hdr ENVI mosaic header
2009078a.reg Contains ALSPP settings
lidar_2009078a.tif Lidar intensity image
line01_2009078a.tif Individual flightline intensity image
ipas
*.gpj IPAS Pro Project File
*.itp IPAS TC Project File
base Base station data
oxfr154g30.09n
oxfr154g30.09o
oxfr154g30.epp
oxfr154g30.gpb Gravnav friendly file format
oxfr154g30.sta Text file with info about basestation
eo
extract
*.epp
*.EVT
*_extractLog.txt
*.fbv
*.rbv
*.gpb Grafnav friendly file format
*.GPS
*.IMU
*.RNV
*.RTG
*_RTMsg.txt
*.sta
*.SUP
*.tm
gps
*.cfg Grafnav project file
*.fml
*.fss
*.fsl PPP log file
*.fwd
*.his
*.lat grafnav final output which is used by IPAS (text version)
*.lat.bin grafnav final output which is used by IPAS
*.rev
*.rml
*.rsl PPP log file
*.rss
proc Processed ipas nav data
20090319_142319.sol Final output file
sbet_20090319_from_sol.out applanix sbet created from sol file
20090319_142319.ers
20090319_142319.sup
2009078a_procLog.txt
20090319_142319.iqc IPAS TC quality control file
raw 20090319_111251.001 Raw GPS/IMU data
logs Contains the flight/ops log files
100411_084840_FlightLineLog.csv
20091123212501_LogData.mdb
091005_LogFile.txt
090812_DLLogFile.txt
20091123212501_LogDataSysDesc.xml
proc_laser Contains the finalised processed LAS files
LDR091026_060148_1.LAS LAS file
pplog_20100226_122027.txt log output from ALSPP
stats.txt info from ALSPP about the LAS files
raw_laser (deprecated) Contains the raw laser data
090601_111411
LDR090601_111411_0000000001.scn
RawLaser Contains the raw laser data (and works with FW processing)
090601_111411 Contains raw laser for one contiguous recording
LDR090601_111411_0000000001.scn Raw LIDAR scan files
RawWFD Raw waveform data
100706_094620
WFD100706_094620_0000000001A.LWV
rcd
090803-100306ImageEvents1-processed.csv Processed events file
logs
camera_delivery.log
090803-100306ImageEvents(1|2).csv
090803-145404PhotoId(1|2).csv
090803-100306SensorLog.csv
090803-145404SensorStat(1|2).csv
proc_images 16093417100001G(3|4).tif
proc_thumbnails 16093417100001G(3|4).jpg
raw_images 16093417100001G(3|4).raw
webcam 100715_124101_000.jpg Webcam image
20100715_105036_evt.csv Webcam event file

Delivered data

As data may be delivered multiple times (in the event of a problem requiring reprocessing) or in stages (if particular sensors are completed first and others look like they'll be delayed), we take copies of any data sent to the end user for future reference (e.g. in support). The lev1/ and lev3/ directories will typically contain the latest processed version.

Delivered data is stored delivery/YYYYMMDD/PROJECTCODE/, where YYYYMMDD is the date the delivery was created. The directory structure of a delivery looks like:

Hyperspectral deliveries

Dirs Example Files Comments
COPYRIGHT.txt (deprecated - now in Readme) Copyright notice
Read_Me-20100202.(txt|pdf) Instructions for use and description of dataset, plus any problems encountered.
bin (deprecated - now in readme) Instructions for downloading azgcorr
dem (optional) GB08_19-2010_254b-lidar-ASTER.dem A dem to help the user with processing to lev3. Can be from their lidar/SRTM/ASTER or a combination (not from NEXTMAP data - license restrictions).
doc azgcorr manual and other documentation
azgcorr_geocorrection.pdf
azgcorr_manual.pdf
data_quality-20100525.pdf
README-manual_outdated.txt
lev1 Processed level 1 data
(e|h)078a031b.bil Level 1 image data
(e|h)078a031b.bil.hdr Header file for image data
(e|h)078a031b.hdf Metadata for image
logsheet Copy of pdf version of logsheet
FW10_01-2010_187_Dennys_wood.pdf
misc (optional) Any additional files needed for level 3 processing (e.g. OS National Grid correction file, geoid-spheroid separation grid file, etc)
osgb02
sphsep15lx.grd
screenshots Images of the level 3 output we managed when testing the data
(e|h)078a013b.jpg Individual line screenshot
(eagle|hawk)_mosaic.jpg Screenshot of all level 3 data
(eagle|hawk_mosaic_with_vectors.jpg Screenshot of all level 3 data with overlaid vectors

The image filenames in the lev1 directory are structured as sdddfnnLL.hdf, where:

  • s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
  • ddd is the Julian day of the data acquisition
  • f distinguishes between multiple flights on the same day ("a" being the first flight)
  • nn is the flight line number
  • LL is the processing level
    • 1b = ungeocorrected data
    • 3a = geocorrected without a DEM
    • 3b = geocorrected with a DEM

Lidar deliveries

Dirs Sub Dirs Example Files Comments
COPYRIGHT.txt (deprecated - now in Readme) Copyright notice
Read_Me-20100202.(txt|pdf) Instructions for use and description of dataset, plus any problems encountered
bin Latest versions of the pt_cloud_filter for linux and windows
dem GB08_19-2010_254b-lidar_ASTER-wgs84_utm.dem DEM created from the lidar data. Can be from their lidar/SRTM/ASTER or a combination (not from NEXTMAP data - license restrictions). DEM is in same co-ordinate system as processed LAS files.
GB08_19-2010_254b-lidar_ASTER-wgs84_latlong.dem Above LiDAR dem converted to wgs84_latlong (compatible with apl)
doc lidar-quality-report_20110519.pdf Lidar quality report
flightlines Classified lidar point clouds
ascii LDR-FW10_02-201010101.txt ASCII point clouds
las1.0 LDR-FW10_02-201010101.LAS LAS1.0 point clouds
las1.3 (FW only) LDR-FW-FW10_02-201009801.LAS LAS1.3 point clouds
fw_extractions (FW only) Full waveform area extractions
FW10_02-2010_101_extractions.jpg
FW10_02-2010_101_extractions.txt
area1
waveform_42437_649348_only.txt ?
logsheet Logsheet as stored in admin/ above
screenshots Images of the dem and intensity mosaic
GB08_19-2010_254b_intensity.jpg
GB08_19-2010_254b_intensity_vectors.jpg
GB08_19-2010_254b_dem.jpg

Photographic deliveries

Dirs Example Files Comments
COPYRIGHT.txt (deprecated - now in Readme) Copyright document
GB08_02-2009_078a.kml Google earth file containing eagle/hawk/photo positions
Read_Me-20100202.(txt|pdf) Instructions for use and description of dataset, plus any problems encountered
doc data_quality-20091202.pdf Camera data quality report
RCD105_Calibration_Certificate.pdf (2010 only) Camera Calibration Certificate
eventfile RCD105-GB08_02-2009078a-ImageEvents.csv CSV file containing pos/att info per photograph event (may be missing if camera crashed)
photographs RCD105-GB08_02-2009078a-00001.tif Tagged tif files of each photograph in area of interest
thumbnails RCD105-GB08_02-2009078a-00001.jpg JPEG thumbnails of each photograph in area of interest

Rogue File Types

Anyone know what these are?

GB08_06-2009_182_Bleaklow/leica/logs/RTC090701_141724.las