Version 76 (modified by knpa, 14 years ago) (diff)

--

File structure and naming

A single site/logical group of flight lines/sortie should be contained in a single directory. If there are multiple sites from different projects, significantly different times of overflight or large time gaps, each should have its own directory.

The directory name should look like: PROJECTCODE-YYYY_JJJxx_SITENAME, e.g. GB07_07-2007_102a_Inverclyde, boresight-2007_198, etc.

There should be:

  • no spaces, brackets or other Unix-upsetting characters in filenames
  • execute permission removed from anything non-executable
  • no empty directories? (need to think about this, maybe only do after checking it's not a stub for later data)

The structure under the project directory should look like:

Top Level Dirs 2nd Level Dirs Example Files Comments
0532010.cfg Config file used in current processing proceedure
qsub_comm.sh (deprecated) Script for submitting processing to gridnodes, used in earlier processing proceedure
admin Contains the flight logsheet and any miscellaneous information (readme's, copies of trac tickets, etc)
GB08_02-2009_078a_Delamere.(doc|txt|pdf) Logsheet
applanix *.ppc Raw and processed Applanix navigation (raw files in applanix/raw/DCALM*, final processed data in applanix/*sbet or applanix/Proc/sbet*.out)
auto_applanix (optional) auto.bat If navigation was processed using auto_applanix script then this dir will contain additional output from that script
combine_seperation_bestguess_values
normal_first_convertbase.log
normal_first_convertflight.log
*.bat
*GNSS.cmb
*GNSS.dat
*GNSS_gnuplot
*GNSS.gnv
*GNSS.log
*GNSS.png
*GNSS.sum
base (deprecated) ?
eo (possibly optional and often empty) ?
extract dephem_*.dat Plane gps data after extracted from raw files by applanix
extract_*.log
gps_prim_*.dat
hwconf_*.out
iinr_*.out
iinz_*.out
imu_*.dat
mgps_*.epp
mgps_*.gpb
mgps_*.nov
mgps_*.sta
navclk_prim_*.dat
obs_prim_*.dat
rers_*.out
rinv_*.out
rmrs_*.out
rrms_*.out
rtstat_*.txt
tm_*.dat
vnav_*.out
vrms_*.out
*GNSS.dat
gps *.cmb ?
*.fbv
*.fml
*.fss
*.fwd
*.his
*.rbv
*.rev
*.rml
*.rss
proc ers_*.out Processed applanix data
idx_*.txt
inv_*.out
iin_*.out
mrs_*.out
rms_*.out
rmsg_*.out
sbet_*.out
smers_*.out
smhkm_*.dat
smindex_*.dat
smout_*.out
smparm_*.txt
smphi_*.dat
smpx_*.dat
smreset_*.dat
smrms_*.out
smrmsg_*.out
raw DCALM.??? Raw gps data from plane
atm Raw ATM data files (az???LLL.JJJ, where ??? = disk id, LLL = 3 digit file number (000 typically contains navigation), JJJ = Julian day of capture), pre-2009 only
calibration Rypically a symlink to the appropriate calibration directory
casi Raw CASI data files (ca??LLL.JJJ, where ?? = disk id, LLL = 3 digit file number, JJJ = Julian day of capture), pre-2008 only
delivery Contains copies of data delivered to end users
(see below) Structure described below
dem Contains azgcorr compatible DEMs
eagle Raw Eagle data files (.raw = image data, .hdr = ENVI compatible header, .nav = navigation synchronisation data)
orignal_headers (optional) If the header files have been altered for any reason, the originals should be in here
grid_logs (deprecated) Logs from hyperspectral script runs on gridnode (one log per sct run)
hawk Raw Hawk data files (.raw = image data, .hdr = ENVI compatible header, .nav = navigation synchronisation data)
original_headers (optional) If the header files have been altered for any reason, the originals should be in here
leica Leica LIDAR data
(see below) Structure described below
lev1 Processed hyperspectral level 1 data (note this will be the most current, not necessarily what was delivered), see below for filename info
lev3 Processed hyperspectral level 3 data (note this will be the most current, not necessarily what was delivered), see below for filename info
lidar (deprecated) Raw and processed ULM LIDAR (raw LIDAR looks like "surveyXXX.range", processed is typically in a strips/ directory)
logs *.logs Logs from hyperspectral script runs on gridnode (one log per flightline)
photos (deprecated) RC10 scanned aerial photographs
rinex GPS basestation data appropriate to the site
runa ATM processing scripts
runc CASI processing scripts
rune Eagle processing scripts
runh Hawk processing scripts
tabi Raw TABI data
vectors (2007/8 only) - symlink to appropriate vectors

The lev1 and lev3 data filenames are structured as "sdddfnn1b.{hdf|tif}", where:

  • s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
  • ddd is the Julian day of the data acquisition
  • f distinguishes between multiple flights on the same day (“a” being the first flight)
  • nn is the flight line number (note this is not necessarily the same as the flight line identifier as specified by the PI - these numbers are the number of the flight in order of when it was taken, as on the logsheet)

Leica LIDAR data

subdir of "leica" 2nd level dir Files Comments
aux_files *.reg Reg file used, any logs produced, intensity mosaics, boresight values)
*.tif
*.txt
ipas
base
eo
extract *.fbv
*.rbv
*.RNV
*.RTG
gps *.cfg
*.fml
*.fss
*.fwd
*.gpj
*.his
*.lat
*.lat.bin
*.rev
*.rml
*.rss
proc Processed GPS/IMU data (SBETs)
raw Raw GPS/IMU data
logs *.csv Contains the flight/ops log files
*.mdb
*.txt
*.xml
proc_laser Contains the finalised processed LAS files only
raw_laser Contains the raw laser data
rawWFD Raw waveform data
rcd
raw_images
logs
webcam

Delivered data

As data may be delivered multiple times (in the event of a problem requiring reprocessing) or in stages (if particular sensors are completed first and others look like they'll be delayed), we take copies of any data sent to the end user for future reference (e.g. in support). The lev1/ and lev3/ directories will typically contain the latest processed version.

Delivered data is stored delivery/YYYYMMDD/PROJECTCODE/, where YYYYMMDD is the date the delivery was created. The directory structure of a delivery looks like:

Hyperspectral data

Dirs Files Comments
Read_Me-20100202.(txt|pdf) Instructions for use and description of dataset, plus any problems encountered.
dem A dem to help the user with processing to lev3. Can be from their lidar data, from SRTM or a combination (not from NEXTMAP data).
doc azgcorr manual and other documentation
lev1 Processed level 1 data
logsheet Logsheet as stored in admin/ above
misc Any additional files needed for level 3 processing (e.g. OS National Grid correction file, geoid-spheroid separation grid file, etc)
screenshots Images of the level 3 output we managed when testing the data

The image filenames in the lev1 directory are structured as sdddfnnLL.hdf, where:

  • s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
  • ddd is the Julian day of the data acquisition
  • f distinguishes between multiple flights on the same day ("a" being the first flight)
  • nn is the flight line number
  • LL is the processing level
    • 1b = ungeocorrected data
    • 3a = geocorrected without a DEM
    • 3b = geocorrected with a DEM

Lidar data

Dirs Files Comments
COPYRIGHT.txt Copyright notice
Read_Me-20100202.(txt|pdf) Instructions for use and description of dataset, plus any problems encountered
ascii_laser Classified ascii lidar point clouds
bin Latest versions of the pt_cloud_filter for linux and windows
dem GB08_19-2010-254b.dem DEM created from the lidar data
doc Lidar quality report and processing information
logsheet Logsheet as stored in admin/ above
screenshots Images of the dem and intensity mosaic
GB08_19-2010-254b_intensity.jpg
GB08_19-2010-254b_intensity_vectors.jpg
GB08_19-2010-254b_dem.jpg

Photographic data

Dirs Example Files Comments
GB08_02-2009_078a_Delamere.kml Google earth file containing eagle/hawk/photo positions
Read_Me-20100202.(txt|pdf) Instructions for use and description of dataset, plus any problems encountered
doc Camera data quality report
eventfile CSV file containing pos/att info per photograph event (may be missing if camera crashed)
photographs Tagged tif files of each photograph in area of interest
thumbnails JPEG thumbnails of each photograph in area of interest