Changes between Initial Version and Version 1 of Processing/FilenameConventionsOld


Ignore:
Timestamp:
Jul 4, 2011, 1:20:25 PM (11 years ago)
Author:
knpa
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Processing/FilenameConventionsOld

    v1 v1  
     1= File structure and naming =
     2[[PageOutline]]
     3
     4This page details the old (pre apl suite) structure. The new structure is [wiki:Processing/FilenameConventions here].
     5
     6A single site/logical group of flight lines/sortie should be contained in a single directory.  If there are multiple sites from different projects, significantly different times of overflight or large time gaps, each should have its own directory.
     7
     8The directory name should look like: PROJECTCODE-YYYY_JJJxx_SITENAME, e.g. GB07_07-2007_102a_Inverclyde, boresight-2007_198, etc.
     9
     10There should be:
     11 * no spaces, brackets or other Unix-upsetting characters in filenames
     12 * execute permission removed from anything non-executable
     13 * no empty directories? (need to think about this, maybe only do after checking it's not a stub for later data)
     14
     15The structure under the project directory should look like:
     16||'''Top Level Dirs '''|| '''2nd Level Dirs''' || '''Example File Formats''' || '''Comments'''||
     17|| || || 0532010.cfg || Config file used in current processing procedure ||
     18|| || || (eagle|hawk).mos || Eagle and Hawk mosaics of lev3 data ||
     19|| || || qsub_comm.sh (deprecated) || Script for submitting processing to gridnodes, used in earlier processing procedure ||
     20|| admin ||         || || Contains the flight logsheet and any miscellaneous information (readme's, copies of trac tickets, etc) ||
     21|| || || GB08_02-2009_078a_Delamere.(doc|txt|pdf) || Logsheet ||
     22|| || || GB08_02-2009_078a_Delamere.fpd || Flight planning document ||
     23|| || delivery_readmes || Read_Me-20100809.(aux|log|tex) || Contains intermediate files used to create the !ReadMe(s) ||
     24|| || || (cam|hyp|lidar)_genreadme.cfg || Configuration files generated by generate_readme_config.py ||
     25|| applanix ||         ||  || Raw and processed Applanix navigation (raw files in applanix/raw/DCALM*, final processed data in applanix/*sbet or applanix/Proc/sbet*.out) ||
     26|| || || 2009078a.ppc || POSPac project file ||
     27||             || auto_applanix (optional) || auto.bat ||If navigation was processed using auto_applanix script then this dir will contain additional output from that script||
     28||  || || combine_seperation_bestguess_values || ||
     29||  || || normal_first_convertbase.log || ||
     30||  || || normal_first_convertflight.log || ||
     31|| || || 2009078a.bat || ||
     32||  || || GNSS.cmb || ||
     33||  || || *GNSS.dat || ||
     34||  || || *GNSS_gnuplot  || ||
     35|| || || *GNSS.gnv   || || 
     36|| || || *GNSS.log  || ||
     37||  || || *GNSS.png || ||
     38||  || || *GNSS.sum || ||
     39||             || base (deprecated)   || ||? ||
     40||             || eo (possibly optional and often empty)   ||  || ? ||
     41||             || extract || || Plane gps data after extracted from raw files by applanix ||
     42|| || || dephem_2009078a.dat  || ||
     43||  || || extract_2009078a.log || ||
     44||  || || gps_prim_2009078a.dat || ||
     45||  || || hwconf_2009078a.out || ||
     46||  || ||iinr_2009078a.out || ||
     47||  || || iinz_2009078a.out || ||
     48||  || || imu_2009078a.dat || ||
     49||  || || mgps_2009078a.epp || ||
     50||  || || mgps_2009078a.gpb || POSGNSS friendly file format  ||
     51|| || || mgps_2009078a.nov|| ||
     52|| || || mgps_2009078a.sta || ||
     53|| || || navclk_prim_2009078a.dat || ||
     54|| || || obs_prim_2009078a.dat || ||
     55|| || || rers_2009078a.out || ||
     56|| || || rinv_2009078a.out || ||
     57|| || || rmrs_2009078a.out || ||
     58|| || || rrms_2009078a.out || ||
     59|| || || rtstat_2009078a.txt || ||
     60|| || || tm_2009078a.dat || ||
     61|| || || vnav_2009078a.out || ||
     62|| || || vrms_2009078a.out || ||
     63|| || || *GNSS.dat || ||
     64|| || gps     || || Files associated with POSGNSS ||
     65|| || || 2009078a.cmb  || Combined ||
     66|| || || 2009078a.fbv || ||
     67|| || || 2009078a.fml || ||
     68|| || || 2009078a.fsl || PPP log file ||
     69||  || || 2009078a.fss || ||
     70||  || || 2009078a.fwd || ||
     71|| || || 2009078a.gnv || POSGNSS Project file ||
     72||  || || 2009078a.his || ||
     73|| || || 2009078a.rbv || ||
     74||  || || 2009078a.rev || ||
     75||  || || 2009078a.rml || ||
     76|| || || 2009078a.rsl || PPP log file ||
     77||  || || 2009078a.rss || ||
     78||             || proc    || || Processed applanix data ||
     79|| || || ers_2009078a.out || ||
     80|| || || idx_2009078a.txt || ||
     81|| || || inv_2009078a.out || ||
     82|| || || iin_2009078a.out || ||
     83|| || || mrs_2009078a.out || ||
     84|| || || rms_2009078a.out || ||
     85|| || || rmsg_2009078a.out || ||
     86|| || || sbet_2009078a.out || Main output file ||
     87|| || || sbet_2009078.out|| ||
     88|| || || smhkm_2009078a.dat || ||
     89|| || || smindex_2009078a.dat  || ||
     90|| || ||smout_2009078a.out || ||
     91|| || || smparm_2009078a.txt|| ||
     92|| || || smphi_2009078a.dat|| ||
     93|| || ||smpx_2009078a.dat || ||
     94|| || ||smreset_2009078a.dat || ||
     95|| || ||smrms_2009078a.out || ||
     96|| || ||smrmsg_2009078a.out || ||
     97||             || raw     || DCALM.001 || Raw gps data from plane ||
     98|| atm  (deprecated)       ||         || || Raw ATM data files (az???LLL.JJJ, where ??? = disk id, LLL = 3 digit file number (000 typically contains navigation), JJJ = Julian day of capture), pre-2009 only ||
     99|| calibration ||         || || Typically a symlink to the appropriate calibration directory ||
     100|| casi  (deprecated)      ||         || || Raw CASI data files (ca??LLL.JJJ, where ?? = disk id, LLL = 3 digit file number, JJJ = Julian day of capture), pre-2008 only ||
     101|| delivery    ||         || || Contains copies of data delivered to end users ||
     102||             || ''(see below)'' || || Structure described below ||
     103|| dem         ||         || || Contains azgcorr compatible DEMs ||
     104|| eagle       ||         || || Raw Eagle data files ||
     105|| || || VNIR078a-09-1.hdr || ENVI compatible header ||
     106|| || || VNIR078a-09-1.log || Log files ||
     107|| || || VNIR078a-09-1.nav || Navigation synchronisation data ||
     108|| || || VNIR078a-09-1.raw || Uncalibrated image data in .bil format ||
     109||             || orignal_headers (optional) ||   VNIR078a-GB08-02.hdr-orig    || If the header files have been altered for any reason, the originals should be in here ||
     110|| grid_logs (deprecated)  ||  || || Logs from hyperspectral script runs on gridnode (one log per sct run) ||
     111|| hawk        ||         || || Raw Hawk data files (.raw = image data, .hdr = ENVI compatible header, .nav = navigation synchronisation data) ||
     112|| || || dataCube.raw || ||
     113|| || || SWIR078a-09-1.hdr || ENVI compatible header ||
     114|| || || SWIR078a-09-1.log || Log files ||
     115|| || || SWIR078a-09-1.nav || Navigation synchronisation data ||
     116|| || || SWIR078a-09-1.raw || Uncalibrated image data in .bil format ||
     117||             || original_headers (optional) || SWIR078a-GB08-02.hdr-orig || If the header files have been altered for any reason, the originals should be in here ||
     118|| leica       ||         || || Leica LIDAR data ||
     119||             || ''(see below)'' || || Structure described below ||
     120|| lev1        ||         || ''(see below)'' || Processed hyperspectral level 1 data (should be empty if data has been moved into delivery) ||
     121|| lev3        ||         || || Processed hyperspectral level 3 data ||
     122|| || || e078a013b_n_sct-0.03_p_gpt0.0.tif || ||
     123|| lidar (deprecated)      ||         || || Raw and processed ULM LIDAR (raw LIDAR looks like "surveyXXX.range", processed is typically in a strips/ directory) ||
     124|| logs        ||         || *.logs || Logs from hyperspectral script runs on gridnode (one log per flightline) ||
     125|| photos (deprecated)     ||         || || RC10 scanned aerial photographs ||
     126|| rinex       ||         || || GPS basestation data appropriate to the site ||
     127|| || || oxfr154g30.09n || ||
     128|| || || oxfr154g30.09o || ||
     129|| || || oxfr154g30.epp || ||
     130|| || || oxfr154g30.gpb || POSGNSS friendly file format ||
     131|| || || oxfr154g30.sta || Text file with info about basestation ||
     132|| runa (deprecated)       ||         || || ATM processing scripts ||
     133|| runc (deprecated)       ||         || || CASI processing scripts ||
     134|| rune (deprecated)       ||         || || Eagle processing scripts ||
     135|| runh (deprecated)        ||         || || Hawk processing scripts ||
     136|| tabi        ||         || || Raw TABI data ||
     137|| vectors (deprecated)    ||         || ||(2007/8 only) - symlink to appropriate vectors ||
     138
     139
     140
     141The lev1 and lev3 data filenames are structured as "sdddfnn1b.{hdf|tif}", where:
     142 * s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
     143 * ddd is the Julian day of the data acquisition
     144 * f distinguishes between multiple flights on the same day (“a” being the first flight)
     145 * nn is the flight line number (note this is not necessarily the same as the flight line identifier as specified by the PI - these numbers are the number of the flight in order of when it was taken, as on the logsheet)
     146
     147=== Leica Directory ===
     148
     149|| '''subdir of "leica"''' || '''2nd level dir''' || '''Files''' || '''Comments''' ||
     150|| aux_files  ||           ||   || Reg file used, any logs produced, intensity mosaics, boresight values ||
     151|| || || lidar_2009078a.mos || ENVI mosaic of lidar lines intensity ||
     152|| || || lidar_2009078a.mos.hdr || ENVI mosaic header ||
     153|| || || 2009078a.reg  || Contains ALSPP settings ||
     154|| || || lidar_2009078a.tif || Lidar intensity image ||
     155|| || || line01_2009078a.tif || Individual flightline intensity image ||
     156|| ipas       ||            || || ||
     157|| || || *.gpj || IPAS Pro Project File ||
     158|| || || *.itp || IPAS TC Project File ||
     159|| || base      ||  || Base station data ||
     160|| || || oxfr154g30.09n || ||
     161|| || || oxfr154g30.09o || ||
     162|| || || oxfr154g30.epp || ||
     163|| || || oxfr154g30.gpb || Gravnav friendly file format ||
     164|| || || oxfr154g30.sta || Text file with info about basestation ||
     165|| || eo        ||  || ||
     166|| || extract   ||  || ||
     167|| || || *.epp || ||
     168|| || || *.EVT || ||
     169|| || || *_extractLog.txt || ||
     170|| || || *.fbv || ||
     171|| || || *.rbv || ||
     172|| || || *.gpb || Grafnav friendly file format  ||
     173|| || || *.GPS || ||
     174|| || || *.IMU || ||
     175|| || || *.RNV || ||
     176|| || || *.RTG || ||
     177|| || || *_RTMsg.txt || ||
     178|| || || *.sta || ||
     179|| || || *.SUP || ||
     180|| || || *.tm || ||
     181||            || gps        ||  || ||
     182|| || || *.cfg || Grafnav project file ||
     183|| || || *.fml || ||
     184|| || || *.fss || ||
     185|| || || *.fsl || PPP log file ||
     186|| || || *.fwd || ||
     187|| || || *.his || ||
     188|| || || *.lat || grafnav final output which is used by IPAS (text version) ||
     189|| || || *.lat.bin || grafnav final output which is used by IPAS ||
     190|| || || *.rev || ||
     191|| || || *.rml || ||
     192|| || || *.rsl || PPP log file ||
     193|| || || *.rss || ||
     194||            || proc      || || Processed ipas nav data ||
     195|| || || 20090319_142319.sol || Final output file ||
     196|| || || sbet_20090319_from_sol.out || applanix sbet created from sol file ||
     197|| || || 20090319_142319.ers || ||
     198|| || || 20090319_142319.sup || ||
     199|| || || 2009078a_procLog.txt || ||
     200|| || || 20090319_142319.iqc || IPAS TC quality control file ||
     201||            || raw       || 20090319_111251.001 || Raw GPS/IMU data ||
     202|| logs       ||           || || Contains the flight/ops log files ||
     203|| || || 100411_084840_FlightLineLog.csv || ||
     204|| || || 20091123212501_LogData.mdb || ||
     205|| || || 091005_LogFile.txt || ||
     206|| || || 090812_DLLogFile.txt || ||
     207|| || || 20091123212501_LogDataSysDesc.xml || ||
     208|| proc_laser ||           ||  || Contains the finalised processed LAS files ||
     209|| || || LDR091026_060148_1.LAS || LAS file ||
     210|| || || pplog_20100226_122027.txt || log output from ALSPP ||
     211|| || || stats.txt || info from ALSPP about the LAS files ||
     212|| raw_laser (deprecated) ||            ||  || Contains the raw laser data ||
     213|| || 090601_111411 || || ||
     214|| || || LDR090601_111411_0000000001.scn || ||
     215|| !RawLaser || || || Contains the raw laser data (and works with FW processing) ||
     216|| || 090601_111411 || ||  Contains raw laser for one contiguous recording ||
     217|| || || LDR090601_111411_0000000001.scn || Raw LIDAR scan files ||
     218|| RawWFD  ||            || || Raw waveform data ||
     219|| || 100706_094620 || || ||
     220|| || || WFD100706_094620_0000000001A.LWV || ||
     221|| rcd        ||           ||  || ||
     222||            ||       || 090803-100306ImageEvents1-processed.csv || Processed events file ||
     223||            || logs       ||  || ||
     224|| ||  || camera_delivery.log || ||
     225|| || || 090803-100306ImageEvents(1|2).csv || ||
     226|| || || 090803-145404PhotoId(1|2).csv || ||
     227|| || || 090803-100306SensorLog.csv  || ||
     228|| || || 090803-145404SensorStat(1|2).csv || ||
     229||            || proc_images || 16093417100001G(3|4).tif || ||
     230||            || proc_thumbnails || 16093417100001G(3|4).jpg || ||
     231||            || raw_images || 16093417100001G(3|4).raw|| ||
     232|| webcam     ||           || 100715_124101_000.jpg || Webcam image ||
     233|| || || 20100715_105036_evt.csv || Webcam event file ||
     234
     235
     236=== Delivered data ===
     237
     238As data may be delivered multiple times (in the event of a problem requiring reprocessing) or in stages (if particular sensors are completed first and others look like they'll be delayed), we take copies of any data sent to the end user for future reference (e.g. in support).  The lev1/ and lev3/ directories will typically contain the latest processed version.
     239
     240Delivered data is stored delivery/YYYYMMDD/PROJECTCODE/, where YYYYMMDD is the date the delivery was created.  The directory structure of a delivery looks like:
     241
     242==== Hyperspectral deliveries ====
     243
     244|| '''Dirs''' || '''Example Files''' || '''Comments''' ||
     245|| || COPYRIGHT.txt (deprecated - now in Readme) ||   Copyright notice ||
     246||  ||  Read_Me-20100202.(txt|pdf) ||    Instructions for use and description of dataset, plus any problems encountered. ||
     247|| bin (deprecated - now in readme) || || Instructions for downloading azgcorr ||
     248|| dem (optional) || GB08_19-2010_254b-lidar-ASTER.dem || A dem to help the user with processing to lev3. Can be from their lidar/SRTM/ASTER or a combination ('''not''' from NEXTMAP data - license restrictions). ||
     249|| doc || ||   azgcorr manual and other documentation ||
     250|| || azgcorr_geocorrection.pdf || ||
     251|| || azgcorr_manual.pdf || ||
     252|| || data_quality-20100525.pdf || ||
     253|| || README-manual_outdated.txt || ||
     254|| lev1 || ||   Processed level 1 data ||
     255|| || (e|h)078a031b.bil || Level 1 image data ||
     256|| || (e|h)078a031b.bil.hdr || Header file for image data ||
     257|| || (e|h)078a031b.hdf || Metadata for image ||
     258|| logsheet || ||   Copy of pdf version of logsheet ||
     259|| || FW10_01-2010_187_Dennys_wood.pdf || ||
     260|| misc (optional) || ||   Any additional files needed for level 3 processing (e.g. OS National Grid correction file, geoid-spheroid separation grid file, etc) ||
     261|| || osgb02 || ||
     262|| || sphsep15lx.grd || ||
     263|| screenshots || ||   Images of the level 3 output we managed when testing the data ||
     264|| || (e|h)078a013b.jpg || Individual line screenshot ||
     265|| || (eagle|hawk)_mosaic.jpg || Screenshot of all level 3 data  ||
     266|| || (eagle|hawk_mosaic_with_vectors.jpg  || Screenshot of all level 3 data with overlaid vectors ||
     267
     268The image filenames in the lev1 directory are structured as `sdddfnnLL.hdf`, where:
     269 * `s` is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
     270 * `ddd` is the Julian day of the data acquisition
     271 * `f` distinguishes between multiple flights on the same day ("a" being the first flight)
     272 * `nn` is the flight line number
     273 * `LL` is the processing level
     274   * 1b = ungeocorrected data
     275   * 3a = geocorrected without a DEM
     276   * 3b = geocorrected with a DEM
     277
     278
     279
     280==== Lidar deliveries ====
     281
     282|| '''Dirs''' || '''Sub Dirs''' || '''Example Files''' || '''Comments''' ||
     283|| || || COPYRIGHT.txt (deprecated - now in Readme) ||   Copyright notice ||
     284||  || || Read_Me-20100202.(txt|pdf) ||   Instructions for use and description of dataset, plus any problems encountered ||
     285|| bin ||  ||  ||   Latest versions of the pt_cloud_filter for linux and windows ||
     286|| dem ||  || GB08_19-2010_254b-lidar_ASTER-wgs84_utm.dem || DEM created from the lidar data.  Can be from their lidar/SRTM/ASTER or a combination (not from NEXTMAP data - license restrictions). DEM is in same co-ordinate system as processed LAS files. ||
     287|| ||  || GB08_19-2010_254b-lidar_ASTER-wgs84_latlong.dem|| Above LiDAR dem converted to wgs84_latlong (compatible with apl) ||
     288
     289|| doc || || lidar-quality-report_20110519.pdf  ||  Lidar quality report ||
     290|| flightlines || ||  ||   Classified lidar point clouds ||
     291||  ||  ascii ||  LDR-FW10_02-201010101.txt || ASCII point clouds ||
     292||  ||  las1.0 || LDR-FW10_02-201010101.LAS || LAS1.0 point clouds ||
     293||  ||  las1.3 (FW only) ||  LDR-FW-FW10_02-201009801.LAS || LAS1.3 point clouds ||
     294|| fw_extractions (FW only) ||  ||  || Full waveform area extractions ||
     295|| ||  || FW10_02-2010_101_extractions.jpg || ||
     296|| ||  || FW10_02-2010_101_extractions.txt || ||
     297|| || area1 || || ||
     298|| || || waveform_42437_649348_only.txt ? || ||
     299|| logsheet ||  || ||   Logsheet as stored in admin/ above ||
     300|| screenshots ||  || || Images of the dem and intensity mosaic ||
     301|| ||  || GB08_19-2010_254b_intensity.jpg || ||
     302|| ||  || GB08_19-2010_254b_intensity_vectors.jpg || ||
     303|| ||  || GB08_19-2010_254b_dem.jpg || ||
     304
     305==== Photographic deliveries ====
     306
     307|| '''Dirs''' || '''Example Files''' || '''Comments''' ||
     308||  || COPYRIGHT.txt (deprecated - now in Readme) || Copyright document  ||
     309||  || GB08_02-2009_078a.kml ||  Google earth file containing eagle/hawk/photo positions ||
     310||   || Read_Me-20100202.(txt|pdf) ||  Instructions for use and description of dataset, plus any problems encountered   ||
     311|| doc  || data_quality-20091202.pdf || Camera data quality report ||
     312|| || RCD105_Calibration_Certificate.pdf (2010 only) || Camera Calibration Certificate ||
     313|| eventfile || RCD105-GB08_02-2009078a-!ImageEvents.csv || CSV file containing pos/att info per photograph event (may be missing if camera crashed) ||
     314|| photographs || RCD105-GB08_02-2009078a-00001.tif || Tagged tif files of each photograph in area of interest ||
     315|| thumbnails || RCD105-GB08_02-2009078a-00001.jpg || JPEG thumbnails of each photograph in area of interest ||
     316
     317'''Rogue File Types'''
     318
     319Anyone know what these are?
     320
     321GB08_06-2009_182_Bleaklow/leica/logs/RTC090701_141724.las [[BR]]