Version 79 (modified by benj, 8 years ago) (diff)

--

ARSF Scripts and Binaries

Please keep this page updated. Also please keep it sorted in a sensible order or we're never going to be able to find anything...

LiDAR

Script Language Use Author
alspp.exe bash Runs wine version of ALSPP
arsf_winesetup.sh bash Configure wine environment for ARSF users
attune bash Runs wine version of Attune (boresight calibration)
check_ascii_lidar.sh bash Checks the ASCII LiDAR files in the delivery for erroneous data
classify_las Binary (C) LiDAR classification
classify_las_files.sh bash Runs classify_las on a set of LAS files using a specified classification algorithm
create_latex_lidar_readme.py Python Creates a LaTeX file for generating a PDF read-me for delivery
create_lidar_readme.py Python Creates a Readme for lidar deliveries
fugroviewer.exe bash Runs wine version of Fugroviewer
generate_mask.py
generate_mask.sh
Python Script for generating mask file for use in GRASS for old Optech LiDAR
get_first_return.sh bash Creates new ASCII lidar files with 1st returns only
import_ldr_txt_to_grass.sh ASCII English text Imports lidar .txt file into grass (run from within grass)
lag Binary (C++) LiDAR file viewer harg + chrfi
las2txt.sh bash Convert a directory of LAS files or a single LAS file into ascii format
las_hdr_pop.py Python Populates las files with header with information that may have been lost: min and max of x,y,z and number of points by return knpa
las_merge.sh bash Merges sets of LAS files of the form *_000.LAS, *_001.LAS... etc. together
las_sampler.py Python Samples a LAS file at a specified interval and creates an output LAS file of these points knpa
las_single_class.py Python Classifies all points as a user defined number (quite slow) knpa
LASsorter Python sorts Las files by time, x, y, z , ascending and descending
leica_photo_processing bash Runs wine version of Leica's photo processor
lidar2dem.sh bash Generates an azgcorr-format ASCII DEM from old Optech lidar point cloud files
lidar_histogram.py
lidar_histogram.sh
Python Reads old Optech LiDAR files and outputs a numeric histogram for use in filtering
lidar_stats.py Python Extracts statistics from a set of LAS files
make_lidar_delivery.sh bash Create a LiDAR delivery folder
make_lidardem_or_intensity.sh
make_lidardem_or_intensity.grass
bash Create a lidar dem or intensity image from ascii lidar files
pt_cloud_filter Binary (C) Removes points of a specified classification from ascii lidar point cloud knpa
sol_reader.py Python Sol reading library chrfi
trim_lidar.sh bash Cuts UTM zone number off old Optech LiDAR point cloud files

Hyperspectral

Script Language Use Author
adjustBPF.py Python Generate a bad pixel file in CSV for a delivery, uses the CSV bad pixel file from calibration
bil_binning.py Python Spectrally bins (x2) a hyperspectral raw file knpa
bounding_box.py Python Get the North/South/East/West extent of a number of hdr files in bng coords
cal_radiometric.py Python Generates a calibration curve in each band given input data in BIL or BSQ format and a known lamp file benj
check_fsm.py Python Works out the corrected value for frame shift smear at a given band assuming a constant DN value through the spectrum benj
check_qpftiming.py Python Test specified Eagle/Hawk framerates against actual framerates calculated from qpfTiming benj
checkpeaks.py Python Matches measured spectral lines to fityk-detected peaks and eagle/hawk pixel numbers for use in spectral calibration benj
create_latex_hyperspectral_readme.py Python Create a LaTeX file for a PDF Readme for Hyperspectral deliveries
dimfixer.py Python Truncates BIL files so that the file size matches what the header file says it should benj
dummy_linecount.py Python Replaces a broken line counter on the first band of eagle and hawk (values 0 to number of lines) knpa
fastcheckoverflows.py Python Checks a BIL or BSQ file to see if it contains a specified overflow value benj
fastQC Binary (C) Program for viewing BIL files (raw data or level 1) harg
gen_bandsets.py Python Generates bandset (.prn, .bnd, .wls) files from a wavelength scale file benj
gen_synth_radcal.py Python Generates a synthetic Eagle or Hawk calibration file benj
gen_synth_raw.py Python Generates a synthetic Eagle or Hawk dataset benj
gen_synth_sbet.py Python Generates a synthetic SBET file benj
generate_prn.py Python Generates a .prn file from some subset of bands in a given .wls file benj
generate_runscripts.py Python Generates config files for hyperspectral data processing mark1/benj
get_boresight_vals.py Python Extracts boresight values for given day from data file
get_calib_vals.py Python Extracts calibration file names for given day from data file
get_sbet_bounds.py Python Gets North/South/East/West extent of (part of) an SBET file in lat/lon benj
getlatlong.sh bash Determines lat/lon for flight from Eagle/Hawk header files benj
getposition Python Gets position information at from an SBET file for a specified time (or file limits) benj
getsync.py Python Finds sync records on one or more AISA nav files
gettiles.sh bash Determines BNG grid squares for flight from Eagle/Hawk header files benj
hdr_fps_check.py Python Checks if a Specim file has a mismatch between recorded and actual frame rate (Hawk problem)
igm2map.sh
igm2map.pro
bash Script to use IGM files to generate a GLT file and map level 1 data
interleave_convert.py Python Convert interleave of a 3D file between BIL/BSQ
logwriter.py Python Outputs mean altitude, heading and speed from an SBET file for a number of Eagle or Hawk lines benj
make_delivery_folder.sh bash Creates a hyperspectral delivery folder
make_mosaic.sh
make_mosaic.grass
bash Script to convert GeoTIFFs into JPGs and create a mosaic image (with vectors)
match_dark_frames.py Python Checks which of a selection of Eagle/Hawk header files have compatible binning, framerate and integration time settings benj
mergeCFSync.py Python Merges a .cfg and a .sync (eagle/hawk line sct offsets)
nav2txt Binary (C) Output contents of an AISA nav file to an ASCII text file
ppv_to_envi_igm.py Python Converts azgcorr's ppv (per-pixel view vector) output to an ENVI IGM
process_specim_line.py Python Runs az suite on specified Eagle/Hawk flightline for processing benj
run_calibration.py Python Master script for running calibration scripts benj
splitline.py Python Script/Library for splitting flightlines Eagle/Hawk
slice_nav.py Python Cuts a chunk of an AISA nav file out into a smaller nav file
specim_gap_check.py Python Searches Eagle/Hawk raw files for dropped frames by looking at the band 1 frame counter
specim_qsub.py Python Submits an ARSF project for processing through the grid engine benj
wavelength_scale.py Python Generates a new wavelength scale from matching spectral peaks benj

Camera

Script Language Use
create_latex_camera_readme.py Python Creates a LaTeX file for generating a PDF read_me for delivery
image_events_file_fixer.py Python Corrects image event files, mainly by using a sol/sbet file to correct position accuracy
make_delivery_folder.sh bash Creates a Photograph delivery folder
photo2thumb.sh bash Converts RCD TIF files to JPG thumbnails
rename_photos_update_events_file.py Python generates new file names for tiffs and their entries in the image events file
tag_tiffs.py Python Adds tags to tiff files (using image event files) which define position, attitude and other metadata

General

Script Language Use
atcor bash Runs ATCOR-4
auto_grass_vector_tiff_conversion_driver.sh
auto_grass_vector_tiff_conversion_control_script.grass
Python uses grass to produce tiff images of vectors
binary_converter.py Python Converts binary files from one data type to another (eg. Float32 -> Int16)
bnglookup.py Python Converts BNG numeric grid references to OS grid square co-ordinates
checkoverflows.py Python Checks binary data files for presence of a specified overflow value
data_arithmetic.py Python Performs an arithmetic operation on all of the values from two data files and writes the results to an output data file
deliverProject.sh Bash Will convert a delivery as to its project code and PI name, all necessary files will be changed. Also it could create tex/pdf for any delivery that might be single PI.
demgen.py Python Reads eagle or hawk header files and SBET file to generate a list of the DEM tiles that the SBET covers
demhdr_az2grass.sh bash Converts dems from az format header to grass header
demheaderconvert.sh bash Converts dems from grass format header to az header
doc2pdf.py Python Converts .doc or .odt files to PDF format
fastcopy_arsf_proj_dir.sh bash Script to make a workspace copy of a project directory (ie. avoid copying raw files, etc)
findbaseline.py Python Script to find the baseline value of a list of numbers containing value spikes. Intended for part of Specim cal procedure.
fix_naughty_chars.py Python Finds linux-unfriendly characters in filenames
fixnamesl13.py Python Auto-renames (line numbers and removes sct section) level 1, level 3, screenshots or rcd images/thumbnails/raws for a delivery.
freelook bash Runs ENVI Freelook
gen_fityk_script.py Python Generates a fityk script file that can be used to fit peaks to particular datasets
generate_readme_config.py Python Creates a config file for hyperspectral and camera read_me generators
generateKMLforPI.py Python Generate a KML file containing Eagle, Hawk or RCD information
GML2Shape.sh bash Convert GML Vector files to Shapefiles
gpstime Python Converts GPS time-of-week to human-readable format and vice-versa
hdf_output.py Python Lists contents of an HDF file to the terminal
hdf_to_bil Python converts hdf files to bil files (navigation data)
julian CShell Converts between julian day+year and day/month/year
ll2bng.sh bash Converts specified lat/lon into BNG co-ordinates (wrapper around appropriate proj command)
mask_bad_pixels.py Python Sets defined bad pixels in a BIL or BSQ file to be a mask value
nextmapdem.sh bash Generates an azgcorr-style DEM from NextMap data
ppv_to_envi_igm Python converts a ppv file (whatever those are) to an igm file and creates a envi header file for it
proj_tidy.sh bash Checks that a specified project conforms to the current defined structure and file name formats
spatial_average.py Python Averages a BIL or BSQ file in the spatial direction
symlink_src_dir.sh bash Symlinks all files in a source directory into a target directory
teqc Binary (third-party) Program for slicing/splicing/fiddling with RINEX or other GPS base station data (like raw data)
times4grafnav.py Python Searches E/H headers and gets start/stop times for importing into Grafnav to show on trajectory
trmdatefinder.py Python Extracts date from Trimble GPS format file names

Libraries

Script Language Use
alphaconv.py Python Functions to convert numbers to letters and back (A=1, B=2 etc)
band_reader.py Python Reads AISA band (.prn) files
configSorter.py Python Sorts config files into sensible order (can also be executed directly)
convert_interpolate.py Python Interpolates an array and/or many arrays wrt. one array. Finds peaks/Derivates/Integrates for 1D arrays.
common_readme_functions.py Python Functions for latex readme generator scripts
csv_handler.py Python Reads CSV columns as lists and writes lists to a CSV file
data_handler.py Python Reads/writes BIL and BSQ files
flightlineclass.py Python Class to hold info on Eagle/Hawk flight lines (used in generateKMLforPI.py)
libgpstime.py Python Converts GPS time-of-week to human-readable format and vice-versa
liblogwriter.py Python Gets mean altitude, heading and speed from an SBET file for a number of Eagle or Hawk lines
rcdclasses.py Python Class for holding RCD image information (used in generateKMLforPI.py)
sbet_handler.py Python Functions to manipulate SBET files, also limited AISA nav file support

Obsolete

Script Language Use
arsf_processing_monitor.py Python Part of old processing job submission system
az_log_parser.py Python Parse output from az* programs to remove cruft - not kept up to date with newer versions and possibly too ambitious
gtiff2jpg.py Python Convert GeoTIFFs to JPEGs
lagf12 bash fedora 12 version of lag, when the main version was designed for f10
kqsub bash Handy for quick submission of individual flightlines and sct values. Designed for use with old rune/runh scripts
kmlise_project.py Python
unpack_file_check.py Python Unpacking check files script
unpack_folder_structure.py Python Unpacking check the folder structure

If it's obsolete and been replaced by something newer / better, delete the obsoleted one. If we don't have a replacement, it may be worth keeping around but remember the cost in confusion.

Explanation of symlinks in ~arsf/usr/bin

Link name Link target Use
configSorter /users/rsg/arsf/usr/lib/python/site-packages/configSorter.py Link to configSorter library (see Libraries section) - library is also executable

Unknown

Script Language Use
atm_bad_scans.py Python
casefixer.py Python
combined_band_mask.py symbolic link to `/users/rsg/arsf/code/mask_creation/combined_band_mask.py'
delivery_prep_2006.sh bash
hdf_to_bil.py Python
kml_line_photosorter.py Python
LASsorter.py Python
overlay_combined_band_mask.py symbolic link to `/users/rsg/arsf/code/mask_creation/overlay_combined_band_mask.py'
photo_renamer.py Python
read_arsf_logsheets.py Python
remove_bands_from_header.py Python
seperate_band_mask.py symbolic link to `/users/rsg/arsf/code/mask_creation/seperate_band_mask.py'
tifftagger2_exiftool.py Python

Note: See also Python resources/libraries on main RSG wiki at http://rsg.pml.ac.uk/intranet/trac/wiki/Python