= ARSF Scripts and Binaries = Please keep this page updated. Also please keep it sorted in a sensible order or we're never going to be able to find anything... == LiDAR == || '''Script''' || '''Language''' || '''Use''' || '''Author''' || || alspp.exe || bash || Runs wine version of ALSPP || || || arsf_winesetup.sh || bash || Configure wine environment for ARSF users || || || attune || bash || Runs wine version of Attune (boresight calibration) || || || check_ascii_lidar.sh || bash || Checks the ASCII LiDAR files in the delivery for erroneous data || || || classify_las || Binary (C) || LiDAR classification || || || classify_las_files.sh || bash || Runs classify_las on a set of LAS files using a specified classification algorithm || || || create_latex_lidar_readme.py || Python || Creates a LaTeX file for generating a PDF read-me for delivery || || || create_lidar_readme.py || Python || Creates a Readme for lidar deliveries || || || fugroviewer.exe || bash || Runs wine version of Fugroviewer || || || generate_mask.py [[BR]] generate_mask.sh || Python || Script for generating mask file for use in GRASS for old Optech LiDAR || || || get_first_return.sh || bash || Creates new ASCII lidar files with 1st returns only || || || import_ldr_txt_to_grass.sh || ASCII English text || Imports lidar .txt file into grass (run from within grass)|| || || lag || Binary (C++) || LiDAR file viewer || harg + chrfi || || las2txt.sh || bash || Convert a directory of LAS files or a single LAS file into ascii format || || || las_hdr_pop.py || Python || Populates las files with header with information that may have been lost: min and max of x,y,z and number of points by return || knpa || || las_merge.sh|| bash || Merges sets of LAS files of the form ***_000.LAS, ***_001.LAS... etc. together || || || las_sampler.py || Python || Samples a LAS file at a specified interval and creates an output LAS file of these points || knpa || || las_single_class.py || Python || Classifies all points as a user defined number (quite slow) || knpa || || LASsorter || Python || sorts Las files by time, x, y, z , ascending and descending || || || leica_photo_processing || bash || Runs wine version of Leica's photo processor || || || lidar2dem.sh || bash || Generates an azgcorr-format ASCII DEM from old Optech lidar point cloud files || || || lidar_histogram.py [[BR]] lidar_histogram.sh || Python || Reads old Optech LiDAR files and outputs a numeric histogram for use in filtering || || || lidar_stats.py || Python || Extracts statistics from a set of LAS files || || || make_lidar_delivery.sh || bash || Create a LiDAR delivery folder || || || make_lidardem_or_intensity.sh [[br]] make_lidardem_or_intensity.grass|| bash || Create a lidar dem or intensity image from ascii lidar files || || || pt_cloud_filter || Binary (C) || Removes points of a specified classification from ascii lidar point cloud || knpa || || sol_reader.py || Python || Sol reading library || chrfi || || trim_lidar.sh || bash || Cuts UTM zone number off old Optech LiDAR point cloud files || || == Hyperspectral == || '''Script''' || '''Language''' || '''Use''' || '''Author''' || || adjustBPF.py || Python || Generate a bad pixel file in CSV for a delivery, uses the CSV bad pixel file from calibration || || || bil_binning.py || Python || Spectrally bins (x2) a hyperspectral raw file || knpa || || bounding_box.py || Python || Get the North/South/East/West extent of a number of hdr files in bng coords || || || cal_radiometric.py || Python || Generates a calibration curve in each band given input data in BIL or BSQ format and a known lamp file || benj || || check_apl_cmd || bash || Runs the apl commands in the config file for all level1 flightlines, output to tmp_cmd_check dir || adbe || || check_bil.py || Python || Checks quality of bil files in hyperspectral delivery, to be run in flightlines/level1b/ || adbe || || check_fsm.py || Python || Works out the corrected value for frame shift smear at a given band assuming a constant DN value through the spectrum || benj || || check_qpftiming.py || Python || Test specified Eagle/Hawk framerates against actual framerates calculated from qpfTiming || benj || || checkpeaks.py || Python || Matches measured spectral lines to fityk-detected peaks and eagle/hawk pixel numbers for use in spectral calibration || benj || || create_latex_hyperspectral_readme.py || Python || Create a LaTeX file for a PDF Readme for Hyperspectral deliveries || || || dimfixer.py || Python || Truncates BIL files so that the file size matches what the header file says it should || benj || || dummy_linecount.py || Python || Replaces a broken line counter on the first band of eagle and hawk (values 0 to number of lines) || knpa || || fastcheckoverflows.py || Python || Checks a BIL or BSQ file to see if it contains a specified overflow value || benj || || fastQC || Binary (C) || Program for viewing BIL files (raw data or level 1) || harg || || gen_bandsets.py || Python || Generates bandset (.prn, .bnd, .wls) files from a wavelength scale file || benj || || gen_synth_radcal.py || Python || Generates a synthetic Eagle or Hawk calibration file || benj || || gen_synth_raw.py || Python || Generates a synthetic Eagle or Hawk dataset || benj || || gen_synth_sbet.py || Python || Generates a synthetic SBET file || benj || || generate_prn.py || Python || Generates a .prn file from some subset of bands in a given .wls file || benj || || generate_runscripts.py || Python || Generates config files for hyperspectral data processing || mark1/benj || || get_boresight_vals.py || Python || Extracts boresight values for given day from data file || || || get_calib_vals.py || Python || Extracts calibration file names for given day from data file || || || get_sbet_bounds.py || Python || Gets North/South/East/West extent of (part of) an SBET file in lat/lon || benj || || getlatlong.sh || bash || Determines lat/lon for flight from Eagle/Hawk header files || benj || || getposition || Python || Gets position information at from an SBET file for a specified time (or file limits) || benj || || getsync.py || Python || Finds sync records on one or more AISA nav files || || || gettiles.sh || bash || Determines BNG grid squares for flight from Eagle/Hawk header files || benj || || hdr_fps_check.py || Python || Checks if a Specim file has a mismatch between recorded and actual frame rate (Hawk problem) || || || igm2map.sh [[br]] igm2map.pro|| bash || Script to use IGM files to generate a GLT file and map level 1 data || || || interleave_convert.py || Python || Convert interleave of a 3D file between BIL/BSQ || || || logwriter.py || Python || Outputs mean altitude, heading and speed from an SBET file for a number of Eagle or Hawk lines || benj || || make_delivery_folder.sh || bash || Creates a hyperspectral delivery folder || || || make_mosaic.sh [[br]] make_mosaic.grass|| bash || Script to convert GeoTIFFs into JPGs and create a mosaic image (with vectors) || || || match_dark_frames.py || Python || Checks which of a selection of Eagle/Hawk header files have compatible binning, framerate and integration time settings || benj || || mergeCFSync.py || Python || Merges a .cfg and a .sync (eagle/hawk line sct offsets) || || || nav2txt || Binary (C) || Output contents of an AISA nav file to an ASCII text file || || || new_nav_file_fixer.py || Python || Script to remove once-per-second sync messages from a eagle/hawk nav file that were introduced with the firmware upgrade in October 2011 and sometimes break aplcorr || knpa || || ppv_to_envi_igm.py || Python || Converts azgcorr's ppv (per-pixel view vector) output to an ENVI IGM || || || process_specim_line.py || Python || Runs az suite on specified Eagle/Hawk flightline for processing || benj || || run_calibration.py || Python || Master script for running calibration scripts || benj || || split_autodark.py || Python || Script to split Eagle or Hawk raw files containing autodark frames into a light file and a dark file. || benj || || splitline.py || Python || Script/Library for splitting flightlines Eagle/Hawk with headers || || || slice_nav.py || Python || Cuts a chunk of an AISA nav file out into a smaller nav file || || || specim_gap_check.py || Python || Searches Eagle/Hawk raw files for dropped frames by looking at the band 1 frame counter || || || specim_qsub.py || Python || Submits an ARSF project for processing through the grid engine || benj || || wavelength_scale.py || Python || Generates a new wavelength scale from matching spectral peaks || benj || == Camera == || '''Script''' || Language || Use || || create_latex_camera_readme.py || Python || Creates a LaTeX file for generating a PDF read_me for delivery || || image_events_file_fixer.py || Python || Corrects image event files, mainly by using a sol/sbet file to correct position accuracy || || make_delivery_folder.sh || bash || Creates a Photograph delivery folder || || photo2thumb.sh || bash || Converts RCD TIF files to JPG thumbnails || || rename_photos_update_events_file.py || Python || generates new file names for tiffs and their entries in the image events file || || tag_tiffs.py || Python || Adds tags to tiff files (using image event files) which define position, attitude and other metadata || || pitch_roll_correction.py || Python || Swap pitch and roll in tiff if different from that in csv file || == General == || '''Script''' || Language || Use || Author(s) || || asterdem.sh || Bash || Creates a ASTER DEM for use with aplcorr || knpa || || atcor || bash || Runs ATCOR-4 || || || auto_grass_vector_tiff_conversion_driver.sh [[br]] auto_grass_vector_tiff_conversion_control_script.grass|| Python || uses grass to produce tiff images of vectors || || || binary_converter.py || Python || Converts binary files from one data type to another (eg. Float32 -> Int16) || || || bnglookup.py || Python || Converts BNG numeric grid references to OS grid square co-ordinates || || || checkoverflows.py || Python || Checks binary data files for presence of a specified overflow value || || || data_arithmetic.py || Python || Performs an arithmetic operation on all of the values from two data files and writes the results to an output data file || || || deliverProject.sh || Bash || Will convert a delivery as to its project code and PI name, all necessary files will be changed. Also it could create tex/pdf for any delivery that might be single PI. || || || demgen.py || Python || Reads eagle or hawk header files and SBET file to generate a list of the DEM tiles that the SBET covers || || || demhdr_az2grass.sh ||bash || Converts dems from az format header to grass header || || || demheaderconvert.sh || bash || Converts dems from grass format header to az header || || || doc2pdf.py || Python || Converts .doc or .odt files to PDF format || || || !DocumentConverter.py || Python || Converts between various document formats using Open Office || || || fastcopy_arsf_proj_dir.sh || bash || Script to make a workspace copy of a project directory (ie. avoid copying raw files, etc) || || || findbaseline.py || Python || Script to find the baseline value of a list of numbers containing value spikes. Intended for part of Specim cal procedure. || || || fix_naughty_chars.py || Python || Finds linux-unfriendly characters in filenames || || || fixnamesl13.py || Python || Auto-renames (line numbers and removes sct section) level 1, level 3, screenshots or rcd images/thumbnails/raws for a delivery. || || || freelook || bash || Runs ENVI Freelook || || || gen_fityk_script.py || Python || Generates a fityk script file that can be used to fit peaks to particular datasets || || || generate_readme_config.py || Python || Creates a config file for hyperspectral and camera read_me generators || || || generateKMLforPI.py || Python || Generate a KML file containing Eagle, Hawk or RCD information || || || GML2Shape.sh || bash || Convert GML Vector files to Shapefiles || || || gpstime || Python || Converts GPS time-of-week to human-readable format and vice-versa || || || hdf_output.py || Python || Lists contents of an HDF file to the terminal || || || hdf_to_bil || Python || converts hdf files to bil files (navigation data) || || || julian || CShell || Converts between julian day+year and day/month/year || || || ll2bng.sh || bash || Converts specified lat/lon into BNG co-ordinates (wrapper around appropriate proj command) || || || lsolder || bash || Like ls, but can filter list to get files/directories older than a certain time || || || mask_bad_pixels.py || Python || Sets defined bad pixels in a BIL or BSQ file to be a mask value || || || make_delivery_letter_email.py || Python || Automatically produce delivery letters and emails for a PI || || || nextmapdem.sh || bash || Generates a DEM from !NextMap data || benj/mggr/knpa || || ppv_to_envi_igm || Python || converts a ppv file (whatever those are) to an igm file and creates a envi header file for it || || || projdir || bash || Automatically CDs to a project directory || || || proj_tidy.sh || bash || Checks that a specified project conforms to the current defined structure and file name formats || || || spatial_average.py || Python || Averages a BIL or BSQ file in the spatial direction || || || symlink_src_dir.sh || bash ||Symlinks all files in a source directory into a target directory || || || teqc || Binary (third-party) || Program for slicing/splicing/fiddling with RINEX or other GPS base station data (like raw data) || || || times4grafnav.py || Python || Searches E/H headers and gets start/stop times for importing into Grafnav to show on trajectory || || || trmdatefinder.py || Python || Extracts date from Trimble GPS format file names || || == Libraries == || '''Script''' || Language || Use || || alphaconv.py || Python || Functions to convert numbers to letters and back (A=1, B=2 etc) || || arsf_database.py || Python || Functions to access key/value ARSF database (please use arsf_db_reader.py) || || arsf_db_reader.py || Python || Provides classes and functions to read from the ARSF database || || band_reader.py || Python || Reads AISA band (.prn) files || || configSorter.py || Python || Sorts config files into sensible order (can also be executed directly) || || convert_interpolate.py || Python || Interpolates an array and/or many arrays wrt. one array. Finds peaks/Derivates/Integrates for 1D arrays. || || common_readme_functions.py || Python || Functions for latex readme generator scripts || || csv_handler.py || Python || Reads CSV columns as lists and writes lists to a CSV file || || data_handler.py || Python || Reads/writes BIL and BSQ files || || flightlineclass.py || Python || Class to hold info on Eagle/Hawk flight lines (used in generateKMLforPI.py) || || kml_writer.py || Python || Creates and writes KML files. || || libgpstime.py || Python || Converts GPS time-of-week to human-readable format and vice-versa || || liblogwriter.py || Python || Gets mean altitude, heading and speed from an SBET file for a number of Eagle or Hawk lines || || rcdclasses.py || Python || Class for holding RCD image information (used in generateKMLforPI.py) || || sbet_handler.py || Python || Functions to manipulate SBET files, also limited AISA nav file support || == Obsolete == || '''Script''' || Language || Use || || arsf_processing_monitor.py || Python || Part of old processing job submission system || || az_log_parser.py || Python || Parse output from az* programs to remove cruft - not kept up to date with newer versions and possibly too ambitious || || gtiff2jpg.py || Python || Convert GeoTIFFs to JPEGs || || lagf12 || bash || fedora 12 version of lag, when the main version was designed for f10 || || kqsub || bash || Handy for quick submission of individual flightlines and sct values. Designed for use with old rune/runh scripts || || kmlise_project.py || Python || || || unpack_file_check.py || Python || Unpacking check files script || || unpack_folder_structure.py || Python || Unpacking check the folder structure || If it's obsolete and been replaced by something newer / better, delete the obsoleted one. If we don't have a replacement, it may be worth keeping around but remember the cost in confusion. == Symlinks == Explanation of symlinks in ~arsf/usr/bin || Link name || Link target || Use || || configSorter || /users/rsg/arsf/usr/lib/python/site-packages/configSorter.py || Link to configSorter library (see Libraries section) - library is also executable || == Unknown == || '''Script''' || Language || Use || || atm_bad_scans.py || Python || || || casefixer.py || Python || || || combined_band_mask.py || symbolic link to `/users/rsg/arsf/code/mask_creation/combined_band_mask.py' || || || delivery_prep_2006.sh || bash || || || hdf_to_bil.py || Python || || || kml_line_photosorter.py || Python || || || LASsorter.py || Python || || || overlay_combined_band_mask.py || symbolic link to `/users/rsg/arsf/code/mask_creation/overlay_combined_band_mask.py' || || || photo_renamer.py || Python || || || read_arsf_logsheets.py || Python || || || remove_bands_from_header.py || Python || || || seperate_band_mask.py || symbolic link to `/users/rsg/arsf/code/mask_creation/seperate_band_mask.py' || || || tifftagger2_exiftool.py || Python || || Note: See also Python resources/libraries on main RSG wiki at http://rsg.pml.ac.uk/intranet/trac/wiki/Python