Changes between Initial Version and Version 1 of Processing/FilenameConventions2011

Timestamp:

May 9, 2012, 3:37:54 PM (13 years ago)

Author:

knpa

Comment:

--

Processing/FilenameConventions2011

@@ +1 @@
+= File structure and naming =
+[[PageOutline]]
+
+This page denotes the 2011 project structure. The current structure can be found [wiki:Processing/FilenameConventions here].
+
+A single site/logical group of flight lines/sortie should be contained in a single directory.  If there are multiple sites from different projects, significantly different times of overflight or large time gaps, each should have its own directory.
+
+The directory name should look like: PROJECTCODE-YYYY_JJJxx_SITENAME, e.g. GB07_07-2007_102a_Inverclyde, boresight-2007_198, etc.
+
+There should be:
+ * no spaces, brackets or other Unix-upsetting characters in filenames
+ * execute permission removed from anything non-executable
+ * no empty directories? (need to think about this, maybe only do after checking it's not a stub for later data) 
+
+The structure under the project directory should look like:
+||'''Top Level Dirs '''|| '''2nd Level Dirs''' || '''3rd Level Dirs''' || '''4th Level Dirs''' || '''5th Level Dirs''' || '''6th Level Dirs''' || '''Example File Formats''' || '''Comments''' ||
+|| admin || || || || || || || Contains the flight logsheet and any miscellaneous information (readme's, copies of trac tickets, etc) ||
+|| || || || || || || GB08_02-2009_078a_Delamere.(doc|txt|pdf) || Logsheet ||
+|| || || || || || || GB08_02-2009_078a_Delamere.fpd || Flight planning document ||
+|| || delivery_readmes || ||  ||  || || Read_Me-20100809.(aux|log|tex) || Intermediate files used to create the !ReadMe(s) || 
+|| || || || || || || (cam|hyp|lidar)_genreadme.cfg || Configuration files generated by generate_readme_config.py ||
+|| calibration || || || || || || || Typically a symlink to the appropriate calibration directory ||
+|| camera || || || || || || || Camera-type sensors ||  
+|| || rcd || || || ||  ||  || RCD digital camera ||
+|| || || || || ||  || 090803-100306ImageEvents1-processed.csv || Processed events file ||
+|| || || logs || || || || || ||
+|| || || || || || || camera_delivery.log || ||
+|| || || || || || ||  090803-100306ImageEvents(1|2).csv || ||
+|| || || || || || ||  090803-145404PhotoId(1|2).csv || ||
+|| || || || || || ||  090803-100306SensorLog.csv  || ||
+|| || || || || || ||  090803-145404SensorStat(1|2).csv || ||
+|| || || raw_images || || || || 16093417100001G(3|4).raw|| ||
+|| || || webcam     || || || || 100715_124101_000.jpg || Webcam image ||
+|| || || || || || || 20100715_105036_evt.csv || Webcam event file ||
+|| hyperspectral || || || || || || || ||
+|| || eagle || || || || || || Raw Eagle data files ||
+|| || || || || || || dataCube.raw || ||
+|| || || || || || || VNIR078a-09-1.hdr || ENVI compatible header ||
+|| || || || || || || VNIR078a-09-1.log || Log files ||
+|| || || || || || || VNIR078a-09-1.nav || Navigation synchronisation data ||
+|| || || || || || || VNIR078a-09-1.raw || Uncalibrated image data in .bil format || 
+|| || hawk || || || || || || Raw Hawk data files ||
+|| || || || || || || dataCube.raw || ||
+|| || || || || || || SWIR078a-09-1.hdr || ENVI compatible header ||
+|| || || || || || || SWIR078a-09-1.log || Log files ||
+|| || || || || || || SWIR078a-09-1.nav || Navigation synchronisation data ||
+|| || || || || || || SWIR078a-09-1.raw || Uncalibrated image data in .bil format ||
+|| lidar || als50 || || || || || || || 
+|| || || aux_files || || || || || Reg file used, any logs produced, intensity mosaics, boresight values ||
+|| || || || || || || lidar_2009078a.mos || ENVI mosaic of lidar lines intensity ||
+|| || || || || || || lidar_2009078a.mos.hdr || ENVI mosaic header ||
+|| || || || || || || 2009078a.reg  || Contains ALSPP settings ||
+|| || || || || || || lidar_2009078a.tif || Lidar intensity image ||
+|| || || || || || || line01_2009078a.tif || Individual flightline intensity image ||
+|| || || logs || || ||  || || Contains the flight/ops log files || 
+|| || || || || || || 100411_084840_FlightLineLog.csv || ||
+|| || || || || || || 20091123212501_LogData.mdb || ||
+|| || || || || || ||  091005_LogFile.txt || ||
+|| || || || || || ||  090812_DLLogFile.txt || ||
+|| || || || || || ||  20091123212501_LogDataSysDesc.xml || || 
+|| || || !RawLaser  || || || || || Contains the raw laser data ||
+|| || || || 090601_111411 || || || || Contains raw laser for one contiguous recording ||
+|| || || || || || ||  LDR090601_111411_0000000001.scn || Raw LIDAR scan files ||
+|| || || RawWFD  || || || || || Raw waveform data ||
+|| || || || 100706_094620 || || || || || 
+|| || || || || || || WFD100706_094620_0000000001A.LWV || ||
+|| logfiles  || || || ||  || ||  EU11_06_2011-098_eagle_-3.o2810538 || Logs from hyperspectral script runs on gridnode (one log per flightline) || 
+|| posatt || || || || || || || Position/attitude data || 
+|| || applanix_litton || || || || || || Data from applanix using the litton imu ||
+|| || || || || || || 2009078a.ppc || POSPac project file ||
+|| || || extract  || || || || || Plane gps data after extracted from raw files by applanix ||
+|| || || || || || || dephem_2009078a.dat  || ||
+|| || || || || || || extract_2009078a.log || ||
+|| || || || || || || gps_prim_2009078a.dat || ||
+|| || || || || || || hwconf_2009078a.out || ||
+|| || || || || || || iinr_2009078a.out || ||
+|| || || || || || || iinz_2009078a.out || ||
+|| || || || || || || imu_2009078a.dat || ||
+|| || || || || || || mgps_2009078a.epp || ||
+|| || || || || || || mgps_2009078a.gpb || POSGNSS friendly file format  ||
+|| || || || || || || mgps_2009078a.nov|| ||
+|| || || || || || || mgps_2009078a.sta || ||
+|| || || || || || || navclk_prim_2009078a.dat || ||
+|| || || || || || || obs_prim_2009078a.dat || ||
+|| || || || || || || rers_2009078a.out || ||
+|| || || || || || || rinv_2009078a.out || ||
+|| || || || || || || rmrs_2009078a.out || ||
+|| || || || || || || rrms_2009078a.out || ||
+|| || || || || || || rtstat_2009078a.txt || ||
+|| || || || || || || tm_2009078a.dat || ||
+|| || || || || || || vnav_2009078a.out || ||
+|| || || || || || || vrms_2009078a.out || ||
+|| || || || || || || *GNSS.dat || ||
+|| || || gps || || || || || Files associated with POSGNSS ||
+|| || || || || || || 2009078a.cmb  || Combined ||
+|| || || || || || || 2009078a.fbv || ||
+|| || || || || || || 2009078a.fml || ||
+|| || || || || || || 2009078a.fsl || PPP log file || 
+|| || || || || || || 2009078a.fss || ||
+|| || || || || || || 2009078a.fwd || ||
+|| || || || || || || 2009078a.gnv || POSGNSS Project file ||
+|| || || || || || || 2009078a.his || ||
+|| || || || || || || 2009078a.rbv || ||
+|| || || || || || || 2009078a.rev || ||
+|| || || || || || || 2009078a.rml || ||
+|| || || || || || || 2009078a.rsl || PPP log file || 
+|| || || || || || || 2009078a.rss || ||
+|| || || proc || || || || ||  Processed applanix data ||
+|| || || || || || || ers_2009078a.out || ||
+|| || || || || || || idx_2009078a.txt || ||
+|| || || || || || || inv_2009078a.out || ||
+|| || || || || || || iin_2009078a.out || ||
+|| || || || || || || mrs_2009078a.out || ||
+|| || || || || || || rms_2009078a.out || ||
+|| || || || || || || rmsg_2009078a.out || ||
+|| || || || || || || sbet_2009078a.out || Main output file ||
+|| || || || || || || sbet_2009078.out|| ||
+|| || || || || || || smhkm_2009078a.dat || ||
+|| || || || || || || smindex_2009078a.dat  || ||
+|| || || || || || || smout_2009078a.out || ||
+|| || || || || || || smparm_2009078a.txt|| ||
+|| || || || || || || smphi_2009078a.dat|| ||
+|| || || || || || || smpx_2009078a.dat || ||
+|| || || || || || || smreset_2009078a.dat || ||
+|| || || || || || || smrms_2009078a.out || ||
+|| || || || || || || smrmsg_2009078a.out || ||
+|| || || raw || || || ||  DCALM.001 || Raw gps data from plane ||
+|| || basestation || || || || || || Basestation data ||
+|| || || || || || || oxfr154g30.09n || ||
+|| || || || || || || oxfr154g30.09o  || || 
+|| || || || || || || oxfr154g30.epp || || 
+|| || || || || || || oxfr154g30.gpb || POSGNSS friendly file format || 
+|| || || || || || ||  oxfr154g30.sta || Text file with info about basestation ||
+|| || ipas_honeywell || || || || || || Data from ipas using the honeywell imu ||
+|| || || || || || || *.gpj || IPAS Pro Project File ||
+|| || || || || || || *.itp || IPAS TC Project File ||
+|| || || extract || || || || || ||
+|| || || || || || || *.epp || ||
+|| || || || || ||  || *.EVT || ||
+|| || || || || || || *_extractLog.txt || ||
+|| || || || || || || *.fbv || ||
+|| || || || || || || *.rbv || ||
+|| || || || || || || *.gpb || Grafnav friendly file format  ||
+|| || || || || || || *.GPS || ||
+|| || || || || || || *.IMU || ||
+|| || || || || || || *.RNV || ||
+|| || || || || || || *.RTG || ||
+|| || || || || || || *_RTMsg.txt || ||
+|| || || || || || || *.sta || ||
+|| || || || || || || *.SUP || ||
+|| || || || || || || *.tm || || 
+|| || || gps || || || ||  || ||
+|| || || || || || || *.cfg || Grafnav project file ||
+|| || || || || || || *.fml || ||
+|| || || || || || || *.fss || ||
+|| || || || || || || *.fsl || PPP log file || 
+|| || || || || || || *.fwd || ||
+|| || || || || || || *.his || ||
+|| || || || || || || *.lat || Grafnav final output which is used by IPAS (text version) || 
+|| || || || || || || *.lat.bin || Grafnav final output which is used by IPAS ||
+|| || || || || || || *.rev || ||
+|| || || || || || || *.rml || ||
+|| || || || || || || *.rsl || PPP log file || 
+|| || || || || || || *.rss || ||
+|| || || proc || || || || || Processed ipas nav data ||
+|| || || || || || || 20090319_142319.sol || Final output file ||
+|| || || || || || || sbet_20090319_from_sol.out || Applanix sbet created from sol file ||
+|| || || || || || || 20090319_142319.ers || ||
+|| || || || || || || 20090319_142319.sup || ||
+|| || || || || || || 2009078a_procLog.txt || ||
+|| || || || || || || 20090319_142319.iqc || IPAS TC quality control file ||
+|| || || raw || || || || 20090319_111251.001 || Raw GPS/IMU data ||
+|| processing || || || || || || || ||
+|| || als50  || || || || || || ||
+|| || || las || || || ||  || Contains the finalised processed LAS files ||
+|| || || || || || || LDR091026_060148_1.LAS || LAS file ||
+|| || || || || || || pplog_20100226_122027.txt || Log output from ALSPP ||
+|| || || || || || || stats.txt || Info from ALSPP about the LAS files ||
+|| || || las-classified || || || ||  || Contains the classified LAS files for delivery||
+|| || || || || || || LDR091026_060148_1.LAS || LAS file ||
+|| || delivery   || || || || || || Contains copies of data delivered to end users ||
+|| || || EUFAR11_06-098-hyperspectral-20110621 || ''Delivery structures described below'' || || || || Delivery folder ||
+|| || hyperspectral || || || || || 0532010.cfg  || Config file ||
+|| || ||  dem  || || || || || ||
+|| || || flightlines || || || || || ||
+|| || || || georeferencing || || || || ||
+|| || || || || igm || || || ||
+|| || || || || mapped || || e078a013b_n_sct-0.03_p_gpt0.0.tif || Geo-rectified images ||
+|| || || || level 1b || || || || ||
+|| || || || line information || || || || ||
+|| || || || navigation || || || || ||
+|| || || || || interpolated || || || ||
+|| || || || || || post_processed || || ||
+|| || || || || || real_time || || ||
+|| || rcd || || || || || || ||
+|| || || proc_images || || ||  || 16093417100001G(3|4).tif || ||
+|| || || thumbnails || || ||  || 16093417100001G(3|4).jpg || ||
+|| tabi || || || || || || || Raw TABI data ||
+
+The lev1 and lev3 data filenames are structured as "sdddfnn1b.{hdf|tif}", where:
+ * s is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
+ * ddd is the Julian day of the data acquisition
+ * f distinguishes between multiple flights on the same day (“a” being the first flight)
+ * nn is the flight line number (note this is not necessarily the same as the flight line identifier as specified by the PI - these numbers are the number of the flight in order of when it was taken, as on the logsheet)
+
+
+== Delivered data ==
+
+As data may be delivered multiple times (in the event of a problem requiring reprocessing) or in stages (if particular sensors are completed first and others look like they'll be delayed), we take copies of any data sent to the end user for future reference (e.g. in support).  The lev1/ and lev3/ directories will typically contain the latest processed version.
+
+Delivered data is stored delivery/YYYYMMDD/PROJECTCODE/, where YYYYMMDD is the date the delivery was created.  The directory structure of a delivery looks like:
+
+==== Hyperspectral deliveries ==== #HyperspecDel
+
+|| '''Top level Dir''' || '''2nd level Dir''' || '''Example Files''' || '''Comments''' ||
+|| || ||  Read_Me-20100202.pdf ||    Instructions for use and description of dataset, plus any problems encountered. || 
+|| dem || ||  || A dem to help the user with processing to lev3. Should be ASTER. || 
+|| || || GB08_19-2010-254b-ASTER.dem || ASTER binary dem ||
+|| || || GB08_19-2010-254b-ASTER.hdr || Header file for DEM ||
+|| doc || || || data quality report and other documentation ||
+|| || || data_quality-20100525.pdf || data quality report || 
+|| flightlines || || || || 
+|| || fodis || || Data for the FODIS instrument ||
+|| || || e220041b_FODIS.bil || ||
+|| || || e220041b_FODIS.bil.hdr || ||
+|| || level1b || || Level 1 image data ||
+|| || || (e|h)078a031b.bil || ENVI bil file of level 1 data ||
+|| || || (e|h)078a031b.bil.hdr || Header file for image data ||
+|| || || e223041b_mask.bil || An image showing masked pixels ||
+|| || || e223041b_mask.bil.hdr || Header file for mask image ||
+|| || || h223041b_badpixels.txt || List of bad pixels (Hawk only) ||
+|| || line_information || (e|h)221021b.xml || metadata for each line ||
+|| || navigation || || Navigation data for each line ||
+|| || || e098101b_nav_post_processed.bil || Envi bil file with navigation data ||
+|| || || e098101b_nav_post_processed.bil.hdr || Header file for navigation ||
+|| || || e098101b_nav_post_processed_quality.bil || Bil file indicating quality of navigation file || 
+|| || || e098101b_nav_post_processed_quality.bil.hdr || Header file for quality file ||
+|| logsheet || || FW10_01-2010_187_Dennys_wood.pdf || Copy of pdf version of logsheet || 
+|| misc (optional) || ||  ||  Any additional files needed for level 3 processing (e.g. OS National Grid correction file, geoid-spheroid separation grid file, etc) || 
+|| || || osgb02 || ||
+|| || || sphsep15lx.grd || ||
+|| project_information ||  || BB11_01-2011_244a-project.xml || Meta data for the project as a whole  ||
+|| screenshots || || || Images of the level 3 output we created when testing the data || 
+|| || || (e|h)078a013b.jpg || Individual line screenshot ||
+|| || || (eagle|hawk)_mosaic.jpg || Screenshot of all level 3 data  || 
+|| || || (eagle|hawk_mosaic_with_vectors.jpg  || Screenshot of all level 3 data with overlaid vectors (UK only) || 
+|| sensor_FOV_vectors || || || ||
+|| || || (eagle|hawk)_fov_fullccd_vectors.bil || ||
+|| || || (eagle|hawk)_fov_fullccd_vectors.hdr || ||
+
+The image filenames in the lev1 directory are structured as `sdddfnnLL.hdf`, where:
+ * `s` is the sensor id (a=ATM, c=CASI, e=Eagle, h=Hawk)
+ * `ddd` is the Julian day of the data acquisition
+ * `f` distinguishes between multiple flights on the same day ("a" being the first flight)
+ * `nn` is the flight line number
+ * `LL` is the processing level
+   * 1b = ungeocorrected data
+   * 3a = geocorrected without a DEM
+   * 3b = geocorrected with a DEM
+
+==== Lidar deliveries ====
+
+|| '''Dirs''' || '''Sub Dirs''' || '''Example Files''' || '''Comments''' ||
+||  || || Read_Me-20100202.pdf ||   Instructions for use and description of dataset, plus any problems encountered || 
+|| bin ||  ||  ||   Latest versions of the pt_cloud_filter for linux and windows ||
+|| dem ||  || || DEMs created from the LiDAR ||
+|| || || GB08_19-2010-254b-lidar_ASTER-wgs84_utm.dem || LiDAR dem patched with ASTER data (do not use NEXTMAP data - license restrictions). DEM is in same co-ordinate system as processed LAS files. ||
+|| || || GB08_19-2010-254b-lidar_ASTER-wgs84_utm.hdr || ||
+|| || || GB08_19-2010-254b-lidar_ASTER-wgs84_latlong.dem || Above LiDAR dem converted to wgs84_latlong (compatible with apl) ||
+|| || || GB08_19-2010-254b-lidar_ASTER-wgs84_latlong.hdr || ||
+|| || || README_DEM_MAY_NOT_BE_ACCURATE.txt || Warning to end user that there are potential flaws with the provided DEMs ||
+|| doc || || lidar-quality-report_20110519.pdf  ||  Lidar quality report ||
+|| flightlines || ||  ||   Classified lidar point clouds ||
+||  ||  ascii ||  LDR-EU11_03-2011-102-10.txt || ASCII point clouds ||
+||  ||  las1.0 || LDR-EU11_03-2011-102-10.LAS || LAS1.0 point clouds ||
+||  ||  las1.3 (FW only) || LDR-FF-EU11_03-201110201.LAS || LAS1.3 point clouds ||
+|| fw_extractions (FW only) ||  ||  || Full waveform area extractions || 
+|| ||  || FW10_02-2010_101_extractions.jpg || ||
+|| ||  || FW10_02-2010_101_extractions.txt || ||
+|| || area1 || || ||
+|| || || waveform_42437_649348_only.txt ? || ||
+|| logsheet ||  || GB08_02-2009_078a_Delamere.pdf || Flight logsheet, typed up or created by generate_logsheet.py ||
+|| screenshots ||  || || Images of the dem and intensity mosaic ||
+|| ||  || GB08_19-2010-254b_intensity.jpg || ||
+|| ||  || GB08_19-2010-254b_intensity_vectors.jpg || UK only ||
+|| ||  || GB08_19-2010-254b_dem.jpg || ||
+
+==== Photographic deliveries ====
+
+|| '''Dirs''' || '''Example Files''' || '''Comments''' ||
+||  || GB08_02-2009_078a.kml ||  Google earth file containing eagle/hawk/photo positions ||
+||   || Read_Me-20100202.pdf ||  Instructions for use and description of dataset, plus any problems encountered ||
+|| doc  || || Useful documents ||
+|| || data_quality-20091202.pdf || Camera data quality report ||
+|| || RCD105_Calibration_Certificate.pdf || Camera Calibration Certificate ||
+|| eventfile || RCD105-GB08_02-2009078a-!ImageEvents.csv || CSV file containing pos/att info per photograph event (may be missing if camera crashed) ||
+|| photographs || RCD105-GB08_02-2009078a-00001.tif || Tagged tif files of each photograph in area of interest ||
+|| thumbnails || RCD105-GB08_02-2009078a-00001.jpg || JPEG thumbnails of each photograph in area of interest ||