ARSF Scripts and Binaries

Please keep this page updated. Also please keep it sorted in a sensible order or we're never going to be able to find anything...

LiDAR

Script	Language	Use	Author
alspp.exe	bash	Runs wine version of ALSPP
arsf_winesetup.sh	bash	Configure wine environment for ARSF users
attune	bash	Runs wine version of Attune (boresight calibration)
check_ascii_lidar.sh	bash	Checks the ASCII LiDAR files in the delivery for erroneous data
check_ascii_lidar	Binary (C)	Checks the ASCII LiDAR files in the delivery for erroneous data, prints the bounds of the ascii file
classify_las	Binary (C)	LiDAR classification
classify_las_files.sh	bash	Runs classify_las on a set of LAS files using a specified classification algorithm
create_latex_lidar_readme.py	Python	Creates a LaTeX file for generating a PDF read-me for delivery
fugroviewer.exe	bash	Runs wine version of Fugroviewer
fwf_extract.py	python	Extracts waveform data for specified area
getDEMbufferpy	Python	Get the buffer size around LiDAR data for use when creating a DEM for use with apl
get_first_return.sh	bash	Creates new ASCII lidar files with 1st returns only
import_ldr_txt_to_grass.sh	bash	Imports lidar ascii file into grass (run from within grass)	mark1
lag	Binary (C++)	LiDAR file viewer	harg + chrfi
las2txt.sh	bash	Convert a directory of LAS files or a single LAS file into ascii format
las_hdr_pop.py	Python	Populates LAS file headers with information that may have been lost: min and max of x,y,z and number of points by return	knpa
las_merge.sh	bash	Merges sets of LAS files of the form *_000.LAS, *_001.LAS... etc. together
las_sampler.py	Python	Samples a LAS file at a specified interval and creates an output LAS file of these points	knpa
las_single_class.py	Python	Classifies all points as a user defined number (quite slow)	knpa
LASsorter	Python	sorts Las files by time, x, y, z , ascending and descending
leica_photo_processing	bash	Runs wine version of Leica's photo processor
lidardem.sh	Bash	Create a DEM from LiDAR data either las of ascii format
lidar_aster_dem.sh	Bash	Create a DEM from LiDAR and aster data, wrapper from lidardem.sh and asterdem.sh
lidar_stats.py	Python	Extracts statistics from a set of LAS files	emca
make_lidar_delivery.py	bash	Create a LiDAR delivery folder
make_lidardem_or_intensity.sh make_lidardem_or_intensity.grass?	bash	Create a lidar dem or intensity image from ascii lidar files
pt_cloud_filter	Binary (C)	Removes points of a specified classification from ascii lidar point cloud	knpa

Hyperspectral

Script	Language	Use	Author
adjustBPF.py	Python	Generate a bad pixel file in CSV for a delivery, uses the CSV bad pixel file from calibration
bil_binning.py	Python	Spectrally bins (x2) a hyperspectral raw file	knpa
bounding_box.py	Python	Get the North/South/East/West extent of a number of raw hdr files in bng coords	knpa
cal_radiometric.py	Python	Generates a calibration curve in each band given input data in BIL or BSQ format and a known lamp file	benj
correlate.py	Python	Wrapper for phasecorrelator.py
check_apl_cmd	bash	Runs the apl commands in the config file for all level1 flightlines, output to tmp_cmd_check dir	adbe
check_bil.py	Python	Checks quality of bil files in hyperspectral delivery	adbe
check_fsm.py	Python	Works out the corrected value for frame shift smear at a given band assuming a constant DN value through the spectrum	benj
check_qpftiming.py	Python	Test specified Eagle/Hawk framerates against actual framerates calculated from qpfTiming	benj
checkpeaks.py	Python	Matches measured spectral lines to fityk-detected peaks and eagle/hawk pixel numbers for use in spectral calibration	benj
create_latex_hyperspectral_readme.py	Python	Create a LaTeX file for a PDF Readme for Hyperspectral azgcorr deliveries
create_latex_hyperspectral_apl_readme.py	Python	Create a LaTeX file for a PDF Readme for Hyperspectral APL deliveries
dimfixer.py	Python	Truncates BIL files so that the file size matches what the header file says it should	benj
dummy_linecount.py	Python	Replaces a broken line counter on the first band of eagle and hawk (values 0 to number of lines)	knpa
fastcheckoverflows.py	Python	Checks a BIL or BSQ file to see if it contains a specified overflow value	benj
fastQC	Binary (C)	Program for viewing BIL files (raw data or level 1)	harg
gen_bandsets.py	Python	Generates bandset (.prn, .bnd, .wls) files from a wavelength scale file	benj
gen_synth_radcal.py	Python	Generates a synthetic Eagle or Hawk calibration file	benj
gen_synth_raw.py	Python	Generates a synthetic Eagle or Hawk dataset	benj
gen_synth_sbet.py	Python	Generates a synthetic SBET file	benj
generate_prn.py	Python	Generates a .prn file from some subset of bands in a given .wls file	benj
generate_apl_runscripts.py	Python	Generates config file for APL hyperspectral processing
get_boresight_vals.py	Python	Extracts boresight values for given day from csv file
get_calib_vals.py	Python	Extracts calibration file names for given day from data file
gen_eg_cmds.py	Python	Generates example commands for APL for the read me file
get_sbet_bounds.py	Python	Gets North/South/East/West extent of (part of) an SBET file in lat/lon	benj
getposition	Python	Gets position information at from an SBET file for a specified time (or file limits)	benj
getsync.py	Python	Finds sync records on one or more AISA nav files
gettiles.sh	bash	Determines BNG grid squares for flight from Eagle/Hawk header files	benj
hdr_fps_check.py	Python	Checks if a Specim file has a mismatch between recorded and actual frame rate (Hawk problem)
hyper_latex_table_tools.py	Python	Python tools for hyperspectral readme table creation
igm2map.sh igm2map.pro	bash	Script to use IGM files to generate a GLT file and map level 1 data
interleave_convert.py	Python	Convert interleave of a 3D file between BIL/BSQ
logwriter.py	Python	Outputs mean altitude, heading and speed from an SBET file for a number of Eagle or Hawk lines	benj
make_mosaic.sh make_mosaic.grass	bash	Script to convert GeoTIFFs into JPGs and create a mosaic image (with vectors)
make_hyper_delivery.py	Python	Make a hyperspectral delivery fro APL data
match_dark_frames.py	Python	Checks which of a selection of Eagle/Hawk header files have compatible binning, framerate and integration time settings	benj
mergeCFSync.py	Python	Merges a .cfg and a .sync (eagle/hawk line sct offsets)
nav2txt	Binary (C)	Output contents of an AISA nav file to an ASCII text file
new_nav_file_fixer.py	Python	Script to remove once-per-second sync messages from a eagle/hawk nav file that were introduced with the firmware upgrade in October 2011 and sometimes break aplcorr	knpa
phasecorrelator.py	Python	Perform phase correlation on a lidar intensity image and a hyperspectral image
process_specim_apl_line.py	Python	Process the specim data using APL
qsub_correlate.py	Python	Submit a correlation job to the grid
process_specim_line.py	Python	Runs az suite on specified Eagle/Hawk flightline for processing	benj
run_calibration.py	Python	Master script for running calibration scripts	benj
split_autodark.py	Python	Script to split Eagle or Hawk raw files containing autodark frames into a light file and a dark file.	benj
splitline.py	Python	Script/Library for splitting Eagle/Hawk flightlines, using headers
slice_nav.py	Python	Cuts a chunk of an AISA nav file out into a smaller nav file
specim_gap_check.py	Python	Searches Eagle/Hawk raw files for dropped frames by looking at the band 1 frame counter
specim_qsub.py	Python	Submits an ARSF project for processing through the grid engine	benj
wavelength_scale.py	Python	Generates a new wavelength scale from matching spectral peaks	benj

Camera

Script	Language	Use	Author
create_latex_camera_readme.py	Python	Creates a LaTeX file for generating a PDF read_me for delivery
image_events_file_fixer.py	Python	Corrects image event files, mainly by using a sol/sbet file to correct position accuracy
make_camera_delivery.py	Python	Create a camera delivery
photo2thumb.sh	bash	Converts RCD TIF files to JPG thumbnails
rename_photos_update_events_file.py	Python	generates new file names for tiffs and their entries in the image events file
tag_tiffs.py	Python	Adds tags to tiff files (using image event files) which define position, attitude and other metadata
pitch_roll_correction.py	Python	Swap pitch and roll in tiff if different from that in csv file

General

Script	Language	Use	Author(s)
add_to_dem.sh	bash/grass	Adds an ascii or binary grid to a DEM - useful when doing geoid-spheroid seperation	knpa
archiving_tarballer.sh	bash	Tarballs projects for NEODC	mggr/knpa
asterdem.sh	Bash	Creates a ASTER DEM for use with aplcorr	knpa/adbe
atcor	bash	Runs ATCOR-4
auto_grass_vector_tiff_conversion_driver.sh auto_grass_vector_tiff_conversion_control_script.grass	Python	uses grass to produce tiff images of vectors
binary_converter.py	Python	Converts binary files from one data type to another (eg. Float32 -> Int16)
bil2tiff	Bash	Convert bil files to geotiffs using gdal
bnglookup.py	Python	Converts BNG numeric grid references to OS grid square co-ordinates
checkoverflows.py	Python	Checks binary data files for presence of a specified overflow value
convert_delivery.sh	Bash	Will convert a delivery to a new project code and PI name. Can also be used on single deliveries to correct PI/proj-code inconsistencies
convert_nonuk_dem.sh	Bash / Grass	Convert a non-UK DEM into APL format (may have to be in WGS84 heights?)
convert_uk_dem.sh	Bash / Grass	Convert a UK DEM into APL format
data_arithmetic.py	Python	Performs an arithmetic operation on all of the values from two data files and writes the results to an output data file
demgen.py	Python	Reads eagle or hawk header files and SBET file to generate a list of the DEM tiles that the SBET covers
demhdr_az2grass.sh	bash	Converts dems from az format header to grass header
demheaderconvert.sh	bash	Converts dems from grass format header to az header
doc2pdf.py	Python	Converts .doc or .odt files to PDF format
DocumentConverter.py	Python	Converts between various document formats using Open Office
findbaseline.py	Python	Script to find the baseline value of a list of numbers containing value spikes. Intended for part of Specim cal procedure.
fix_naughty_chars.py	Python	Finds linux-unfriendly characters in filenames
fixnames_sct.py	Python	Auto-renames (line numbers and removes sct section) level 1, level 3, screenshots or rcd images/thumbnails/raws for a delivery.
freelook	bash	Runs ENVI Freelook
gen_fityk_script.py	Python	Generates a fityk script file that can be used to fit peaks to particular datasets
generate_readme_config.py	Python	Creates a config file for hyperspectral and camera read_me generators
generateKMLforPI.py	Python	Generate a KML file containing Eagle, Hawk or RCD information
GML2Shape.sh	bash	Convert GML Vector files to Shapefiles
gpstime	Python	Converts GPS time-of-week to human-readable format and vice-versa
hdf_output.py	Python	Lists contents of an HDF file to the terminal
hdf_to_bil.py	Python	Extracts navigation data from hdf files to a bil file	chrfi/knpa
julian	CShell	Converts between julian day+year and day/month/year
kmloverview_all_projects_this_year.py	Python	Generate KML Overview for all projects in repository for a given year
ll2bng.sh	bash	Converts specified lat/lon into BNG co-ordinates (wrapper around appropriate proj command)
lsolder	bash	Like ls, but can filter list to get files/directories older than a certain time
mask_bad_pixels.py	Python	Sets defined bad pixels in a BIL or BSQ file to be a mask value
make_delivery_letter_email.py	Python	Automatically produce delivery letters and emails for a PI
make_kmloverview.py	Python	Make the KML file and overview support files - part of unpacking procedure
nextmapdem.sh	bash	Generates a DEM from NextMap data	benj/mggr/knpa
projdir	bash	Automatically CDs to a project directory	anch
proj_tidy.sh	bash	Checks that a specified project conforms to the current defined structure and file name formats	knpa
raw_eagle_hawk_bil2jpg.sh raw_eagle_hawk_bil2jpg.grass	Create JPGs of raw eagle/hawk data for KML overview
spatial_average.py	Python	Averages a BIL or BSQ file in the spatial direction
symlink_src_dir.sh	bash	Symlinks all files in a source directory into a target directory
teqc	Binary (third-party)	Program for slicing/splicing/fiddling with RINEX or other GPS base station data (like raw data)
times4grafnav.py	Python	Searches E/H headers and gets start/stop times for importing into Grafnav to show on trajectory
trmdatefinder.py	Python	Extracts date from Trimble GPS format file names

Libraries

Script	Language	Use	Author
alphaconv.py	Python	Functions to convert numbers to letters and back (A=1, B=2 etc)
arsf_database.py	Python	Functions to access key/value ARSF database (please use arsf_db_reader.py)
arsf_db_reader.py	Python	Provides classes and functions to read from the ARSF database
band_reader.py	Python	Reads AISA band (.prn) files
configSorter.py	Python	Sorts config files into sensible order (can also be executed directly)
convert_interpolate.py	Python	Interpolates an array and/or many arrays wrt. one array. Finds peaks/Derivates/Integrates for 1D arrays.
common_readme_functions.py	Python	Functions for latex readme generator scripts
correlator_reader.py	Python	Read data in that is required for the correlator
csv_handler.py	Python	Reads CSV columns as lists and writes lists to a CSV file
data_handler.py	Python	Reads/writes BIL and BSQ files
flightlineclass.py	Python	Class to hold info on Eagle/Hawk flight lines (used in generateKMLforPI.py)
folder_structure.py	Python	Holds the current arsf folder structure
hyperconfigutils.py	Python	Set of functions for reading and checking hyperspectral config files
kml_writer.py	Python	Creates and writes KML files.
las1_3_handler.py	Python	Reads LAS 1.3 files
libgpstime.py	Python	Converts GPS time-of-week to human-readable format and vice-versa
liblogwriter.py	Python	Gets mean altitude, heading and speed from an SBET file for a number of Eagle or Hawk lines
rcdclasses.py	Python	Class for holding RCD image information (used in generateKMLforPI.py)
sbet_handler.py	Python	Functions to manipulate SBET files, also limited AISA nav file support
sol_reader.py	Python	Sol reading library	chrfi

Obsolete

These are not in everyday use but are still useful on occasion

Script	Language	Use	Author
arsf_processing_monitor.py	Python	Part of old processing job submission system
az_log_parser.py	Python	Parse output from az* programs to remove cruft - not kept up to date with newer versions and possibly too ambitious
create_lidar_readme.py	Python	Creates a .txt ReadMe for lidar deliveries
fastcopy_arsf_proj_dir.sh	bash	Script to make a workspace copy of a project directory (ie. avoid copying raw files, etc)
generate_mask.py generate_mask.sh	Python	Script for generating mask file for use in GRASS for old Optech LiDAR
generate_runscripts.py	Python	Generates AZ runscripts or config files for hyperspectral data processing	mark1/benj
gtiff2jpg.py	Python	Convert GeoTIFFs to JPEGs
kqsub	bash	Handy for quick submission of individual flightlines and sct values. Designed for use with old rune/runh scripts
kmlise_project.py	Python
lidar2dem.sh	bash	Generates an azgcorr-format ASCII DEM from old Optech lidar point cloud files
lidar_histogram.py lidar_histogram.sh	Python	Reads old Optech LiDAR files and outputs a numeric histogram for use in filtering
make_delivery_folder.sh	bash	Old script for making deliveries
ppv_to_envi_igm.py	Python	Converts azgcorr's ppv (per-pixel view vector) output to an ENVI IGM	chrfi
trim_lidar.sh	bash	Cuts UTM zone number off old Optech LiDAR point cloud files
unpack_file_check.py	Python	Unpacking check files script
unpack_folder_structure.py	Python	Unpacking check the folder structure

Symlinks

Explanation of symlinks in ~arsf/usr/bin

Link name	Link target	Use
configSorter	/users/rsg/arsf/usr/lib/python/site-packages/configSorter.py	Link to configSorter library (see Libraries section) - library is also executable

Unknown

Note: See also Python resources/libraries on main RSG wiki at ​http://rsg.pml.ac.uk/intranet/trac/wiki/Python