= ARSF Scripts and Binaries =

Please keep this page updated. Also please keep it sorted in a sensible order or we're never going to be able to find anything...

== LiDAR ==

|| '''Script''' || '''Language''' || '''Use''' || '''Author''' || 
|| alspp.exe || bash || Runs wine version of ALSPP || ||
|| arsf_winesetup.sh || bash || Configure wine environment for ARSF users || ||
|| attune || bash || Runs wine version of Attune (boresight calibration) || ||
|| check_ascii_lidar.sh || bash || Checks the ASCII LiDAR files in the delivery for erroneous data || ||
|| classify_las || Binary (C) || LiDAR classification ||  ||
|| classify_las_files.sh || bash || Runs classify_las on a set of LAS files using a specified classification algorithm || ||
|| create_latex_lidar_readme.py || Python || Creates a LaTeX file for generating a PDF read-me for delivery || ||
|| create_lidar_readme.py || Python || Creates a Readme for lidar deliveries || ||
|| fugroviewer.exe || bash || Runs wine version of Fugroviewer || ||
|| generate_mask.py [[BR]] generate_mask.sh || Python || Script for generating mask file for use in GRASS for old Optech LiDAR || ||
|| get_first_return.sh || bash || Creates new ASCII lidar files with 1st returns only || ||
|| import_ldr_txt_to_grass.sh ||  ASCII English text || Imports lidar .txt file into grass (run from within grass)|| ||
|| lag || Binary (C++) || LiDAR file viewer || harg + chrfi ||
|| las2txt.sh || bash || Convert a directory of LAS files or a single LAS file into ascii format || ||
|| las_hdr_pop.py || Python || Populates las files with header with information that may have been lost: min and max of x,y,z and number of points by return || knpa ||
|| las_merge.sh|| bash || Merges sets of LAS files of the form  ***_000.LAS, ***_001.LAS... etc. together ||  ||
|| las_sampler.py || Python || Samples a LAS file at a specified interval and creates an output LAS file of these points || knpa ||
|| las_single_class.py || Python || Classifies all points as a user defined number (quite slow) || knpa || 
|| LASsorter || Python || sorts Las files by time, x, y, z , ascending and descending || ||
|| leica_photo_processing || bash || Runs wine version of Leica's photo processor || ||
|| lidar2dem.sh || bash || Generates an azgcorr-format ASCII DEM from old Optech lidar point cloud files || ||
|| lidar_histogram.py [[BR]] lidar_histogram.sh || Python || Reads old Optech LiDAR files and outputs a numeric histogram for use in filtering || ||
|| lidar_stats.py || Python || Extracts statistics from a set of LAS files || ||
|| make_lidar_delivery.sh || bash || Create a LiDAR delivery folder || ||
|| make_lidardem_or_intensity.sh [[br]] make_lidardem_or_intensity.grass|| bash || Create a lidar dem or intensity image from ascii lidar files || ||
|| pt_cloud_filter || Binary (C) || Removes points of a specified classification from ascii lidar point cloud || knpa ||
|| sol_reader.py || Python || Sol reading library || chrfi ||
|| trim_lidar.sh || bash || Cuts UTM zone number off old Optech LiDAR point cloud files || ||

== Hyperspectral ==

|| '''Script''' || '''Language''' || '''Use''' || '''Author''' ||
|| adjustBPF.py || Python || Generate a bad pixel file in CSV for a delivery, uses the CSV bad pixel file from calibration || ||
|| bil_binning.py || Python || Spectrally bins (x2) a hyperspectral raw file || knpa ||
|| bounding_box.py || Python || Get the North/South/East/West extent of a number of hdr files in bng coords || ||
|| cal_radiometric.py || Python || Generates a calibration curve in each band given input data in BIL or BSQ format and a known lamp file || benj ||
|| check_apl_cmd || bash || Runs the apl commands in the config file for all level1 flightlines, output to tmp_cmd_check dir || adbe ||
|| check_bil.py || Python || Checks quality of bil files in hyperspectral delivery, to be run in flightlines/level1b/ || adbe ||
|| check_fsm.py || Python || Works out the corrected value for frame shift smear at a given band assuming a constant DN value through the spectrum || benj ||
|| check_qpftiming.py || Python || Test specified Eagle/Hawk framerates against actual framerates calculated from qpfTiming || benj ||
|| checkpeaks.py || Python || Matches measured spectral lines to fityk-detected peaks and eagle/hawk pixel numbers for use in spectral calibration || benj ||
|| create_latex_hyperspectral_readme.py || Python || Create a LaTeX file for a PDF Readme for Hyperspectral deliveries || ||
|| dimfixer.py || Python || Truncates BIL files so that the file size matches what the header file says it should || benj ||
|| dummy_linecount.py || Python || Replaces a broken line counter on the first band of eagle and hawk (values 0 to number of lines) || knpa ||
|| fastcheckoverflows.py || Python || Checks a BIL or BSQ file to see if it contains a specified overflow value || benj ||
|| fastQC || Binary (C) || Program for viewing BIL files (raw data or level 1) || harg ||
|| gen_bandsets.py || Python || Generates bandset (.prn, .bnd, .wls) files from a wavelength scale file || benj ||
|| gen_synth_radcal.py || Python || Generates a synthetic Eagle or Hawk calibration file || benj ||
|| gen_synth_raw.py || Python || Generates a synthetic Eagle or Hawk dataset || benj ||
|| gen_synth_sbet.py || Python || Generates a synthetic SBET file || benj ||
|| generate_prn.py || Python || Generates a .prn file from some subset of bands in a given .wls file || benj ||
|| generate_runscripts.py || Python || Generates config files for hyperspectral data processing || mark1/benj ||
|| get_boresight_vals.py || Python || Extracts boresight values for given day from data file || ||
|| get_calib_vals.py || Python || Extracts calibration file names for given day from data file || ||
|| get_sbet_bounds.py || Python || Gets North/South/East/West extent of (part of) an SBET file in lat/lon || benj ||
|| getlatlong.sh || bash || Determines lat/lon for flight from Eagle/Hawk header files || benj ||
|| getposition || Python || Gets position information at from an SBET file for a specified time (or file limits) || benj ||
|| getsync.py || Python || Finds sync records on one or more AISA nav files || ||
|| gettiles.sh || bash || Determines BNG grid squares for flight from Eagle/Hawk header files || benj ||
|| hdr_fps_check.py || Python || Checks if a Specim file has a mismatch between recorded and actual frame rate (Hawk problem) || ||
|| igm2map.sh [[br]] igm2map.pro|| bash || Script to use IGM files to generate a GLT file and map level 1 data || ||
|| interleave_convert.py || Python || Convert interleave of a 3D file between BIL/BSQ || ||
|| logwriter.py || Python || Outputs mean altitude, heading and speed from an SBET file for a number of Eagle or Hawk lines || benj ||
|| make_delivery_folder.sh || bash || Creates a hyperspectral delivery folder || ||
|| make_mosaic.sh [[br]] make_mosaic.grass|| bash || Script to convert GeoTIFFs into JPGs and create a mosaic image (with vectors) || ||
|| match_dark_frames.py || Python || Checks which of a selection of Eagle/Hawk header files have compatible binning, framerate and integration time settings || benj ||
|| mergeCFSync.py || Python || Merges a .cfg and a .sync (eagle/hawk line sct offsets) || ||
|| nav2txt || Binary (C) || Output contents of an AISA nav file to an ASCII text file || ||
|| new_nav_file_fixer.py || Python || Script to remove once-per-second sync messages from a eagle/hawk nav file that were introduced with the firmware upgrade in October 2011 and sometimes break aplcorr || knpa ||
|| ppv_to_envi_igm.py || Python || Converts azgcorr's ppv (per-pixel view vector) output to an ENVI IGM || ||
|| process_specim_line.py || Python || Runs az suite on specified Eagle/Hawk flightline for processing || benj ||
|| run_calibration.py || Python || Master script for running calibration scripts || benj ||
|| split_autodark.py || Python || Script to split Eagle or Hawk raw files containing autodark frames into a light file and a dark file. || benj ||
|| splitline.py || Python || Script/Library for splitting flightlines Eagle/Hawk with headers || ||
|| slice_nav.py || Python || Cuts a chunk of an AISA nav file out into a smaller nav file || ||
|| specim_gap_check.py || Python || Searches Eagle/Hawk raw files for dropped frames by looking at the band 1 frame counter || ||
|| specim_qsub.py || Python || Submits an ARSF project for processing through the grid engine || benj ||
|| wavelength_scale.py || Python || Generates a new wavelength scale from matching spectral peaks || benj ||

== Camera ==

|| '''Script''' || Language || Use ||
|| create_latex_camera_readme.py ||  Python  || Creates a LaTeX file for generating a PDF read_me for delivery  ||
|| image_events_file_fixer.py || Python || Corrects image event files, mainly by using a sol/sbet file to correct position accuracy ||
|| make_delivery_folder.sh || bash || Creates a Photograph delivery folder ||
|| photo2thumb.sh || bash || Converts RCD TIF files to JPG thumbnails ||
|| rename_photos_update_events_file.py || Python || generates new file names for tiffs and their entries in the image events file ||
|| tag_tiffs.py || Python || Adds tags to tiff files (using image event files) which define position, attitude and other metadata ||
|| pitch_roll_correction.py || Python || Swap pitch and roll in tiff if different from that in csv file ||

== General ==

|| '''Script''' || Language || Use || Author(s) ||
|| asterdem.sh || Bash || Creates a ASTER DEM for use with aplcorr || knpa ||
|| atcor || bash || Runs ATCOR-4 || ||
|| auto_grass_vector_tiff_conversion_driver.sh [[br]] auto_grass_vector_tiff_conversion_control_script.grass|| Python || uses grass to produce tiff images of vectors ||  ||
|| binary_converter.py || Python || Converts binary files from one data type to another (eg. Float32 -> Int16) ||   ||
|| bnglookup.py || Python || Converts BNG numeric grid references to OS grid square co-ordinates ||  ||
|| checkoverflows.py || Python || Checks binary data files for presence of a specified overflow value || ||
|| data_arithmetic.py || Python || Performs an arithmetic operation on all of the values from two data files and writes the results to an output data file || ||
|| deliverProject.sh || Bash || Will convert a delivery as to its project code and PI name, all necessary files will be changed. Also it could create tex/pdf for any delivery that might be single PI. || ||
|| demgen.py || Python || Reads eagle or hawk header files and SBET file to generate a list of the DEM tiles that the SBET covers ||  ||
|| demhdr_az2grass.sh ||bash || Converts dems from az format header to grass header ||  ||
|| demheaderconvert.sh || bash || Converts dems from grass format header to az header ||  ||
|| doc2pdf.py || Python || Converts .doc or .odt files to PDF format ||  ||
|| !DocumentConverter.py || Python || Converts between various document formats using Open Office || ||
|| fastcopy_arsf_proj_dir.sh || bash || Script to make a workspace copy of a project directory (ie. avoid copying raw files, etc) || ||
|| findbaseline.py || Python || Script to find the baseline value of a list of numbers containing value spikes. Intended for part of Specim cal procedure. || ||
|| fix_naughty_chars.py || Python || Finds linux-unfriendly characters in filenames ||  ||
|| fixnamesl13.py || Python || Auto-renames (line numbers and removes sct section) level 1, level 3, screenshots or rcd images/thumbnails/raws for a delivery. || ||
|| freelook || bash || Runs ENVI Freelook || ||
|| gen_fityk_script.py || Python || Generates a fityk script file that can be used to fit peaks to particular datasets || ||
|| generate_readme_config.py || Python || Creates a config file for hyperspectral and camera read_me generators ||  ||
|| generateKMLforPI.py || Python || Generate a KML file containing Eagle, Hawk or RCD information || ||
|| GML2Shape.sh || bash || Convert GML Vector files to Shapefiles ||  ||
|| gpstime || Python || Converts GPS time-of-week to human-readable format and vice-versa ||  ||
|| hdf_output.py || Python || Lists contents of an HDF file to the terminal ||  ||
|| hdf_to_bil || Python || converts hdf files to bil files (navigation data) ||   ||
|| julian || CShell || Converts between julian day+year and day/month/year ||   ||
|| ll2bng.sh || bash || Converts specified lat/lon into BNG co-ordinates (wrapper around appropriate proj command) ||  ||
|| lsolder || bash || Like ls, but can filter list to get files/directories older than a certain time || ||
|| mask_bad_pixels.py || Python || Sets defined bad pixels in a BIL or BSQ file to be a mask value ||  ||
|| make_delivery_letter_email.py || Python || Automatically produce delivery letters and emails for a PI || ||
|| nextmapdem.sh || bash || Generates a DEM from !NextMap data || benj/mggr/knpa ||
|| ppv_to_envi_igm || Python || converts a ppv file (whatever those are) to an igm file and creates a envi header file for it ||   ||
|| projdir || bash || Automatically CDs to a project directory || ||
|| proj_tidy.sh || bash || Checks that a specified project conforms to the current defined structure and file name formats || ||
|| spatial_average.py || Python || Averages a BIL or BSQ file in the spatial direction || ||
|| symlink_src_dir.sh || bash ||Symlinks all files in a source directory into a target directory ||  ||
|| teqc || Binary (third-party) || Program for slicing/splicing/fiddling with RINEX or other GPS base station data (like raw data) ||  ||
|| times4grafnav.py || Python || Searches E/H headers and gets start/stop times for importing into Grafnav to show on trajectory ||  ||
|| trmdatefinder.py || Python || Extracts date from Trimble GPS format file names || ||

== Libraries ==
|| '''Script''' || Language || Use ||
|| alphaconv.py || Python || Functions to convert numbers to letters and back (A=1, B=2 etc) ||
|| arsf_database.py || Python || Functions to access key/value ARSF database (please use arsf_db_reader.py) ||
|| arsf_db_reader.py || Python || Provides classes and functions to read from the ARSF database ||
|| band_reader.py || Python || Reads AISA band (.prn) files ||
|| configSorter.py || Python || Sorts config files into sensible order (can also be executed directly) ||
|| convert_interpolate.py || Python || Interpolates an array and/or many arrays wrt. one array. Finds peaks/Derivates/Integrates for 1D arrays. ||
|| common_readme_functions.py || Python || Functions for latex readme generator scripts ||
|| csv_handler.py || Python || Reads CSV columns as lists and writes lists to a CSV file ||
|| data_handler.py || Python || Reads/writes BIL and BSQ files ||
|| flightlineclass.py || Python || Class to hold info on Eagle/Hawk flight lines (used in generateKMLforPI.py) ||
|| kml_writer.py || Python || Creates and writes KML files. ||
|| libgpstime.py || Python || Converts GPS time-of-week to human-readable format and vice-versa ||
|| liblogwriter.py || Python || Gets mean altitude, heading and speed from an SBET file for a number of Eagle or Hawk lines ||
|| rcdclasses.py || Python || Class for holding RCD image information (used in generateKMLforPI.py) ||
|| sbet_handler.py || Python || Functions to manipulate SBET files, also limited AISA nav file support ||

== Obsolete ==

|| '''Script''' || Language || Use ||
|| arsf_processing_monitor.py || Python || Part of old processing job submission system ||
|| az_log_parser.py || Python || Parse output from az* programs to remove cruft - not kept up to date with newer versions and possibly too ambitious ||
|| gtiff2jpg.py || Python || Convert GeoTIFFs to JPEGs ||
|| lagf12 || bash || fedora 12 version of lag, when the main version was designed for f10 ||
|| kqsub || bash || Handy for quick submission of individual flightlines and sct values. Designed for use with old rune/runh scripts ||
|| kmlise_project.py || Python || ||
|| unpack_file_check.py || Python || Unpacking check files script ||
|| unpack_folder_structure.py || Python || Unpacking check the folder structure ||

If it's obsolete and been replaced by something newer / better, delete the obsoleted one.  If we don't have a replacement, it may be worth keeping around but remember the cost in confusion.

== Symlinks ==

Explanation of symlinks in ~arsf/usr/bin

|| Link name || Link target || Use ||
|| configSorter || /users/rsg/arsf/usr/lib/python/site-packages/configSorter.py || Link to configSorter library (see Libraries section) - library is also executable ||


== Unknown ==

|| '''Script''' || Language || Use ||
|| atm_bad_scans.py || Python || ||
|| casefixer.py || Python || ||
|| combined_band_mask.py ||  symbolic link to `/users/rsg/arsf/code/mask_creation/combined_band_mask.py' || ||
|| delivery_prep_2006.sh || bash || ||
|| hdf_to_bil.py || Python || ||
|| kml_line_photosorter.py || Python || ||
|| LASsorter.py || Python || ||
|| overlay_combined_band_mask.py ||  symbolic link to `/users/rsg/arsf/code/mask_creation/overlay_combined_band_mask.py' || ||
|| photo_renamer.py || Python || ||
|| read_arsf_logsheets.py || Python || ||
|| remove_bands_from_header.py || Python || ||
|| seperate_band_mask.py ||  symbolic link to `/users/rsg/arsf/code/mask_creation/seperate_band_mask.py' || ||
|| tifftagger2_exiftool.py || Python || ||

Note: See also Python resources/libraries on main RSG wiki at http://rsg.pml.ac.uk/intranet/trac/wiki/Python