= ARSF Scripts and Binaries = Please keep this page updated. Also please keep it sorted in a sensible order or we're never going to be able to find anything... == LiDAR == || '''Script''' || '''Language''' || '''Use''' || '''Author''' || || alspp.exe || bash || Runs wine version of ALSPP || || || arsf_winesetup.sh || bash || Configure wine environment for ARSF users || || || attune || bash || Runs wine version of Attune (boresight calibration) || || || batch_fw_class.sh || bash || Runs get_class_las on full directories, so it doesn't have to be done one by one. Does NOT check if las files are sorted || dap || check_ascii_lidar.sh || bash || Checks the ASCII LiDAR files in the delivery for erroneous data || || || check_ascii_lidar || Binary (C) || Checks the ASCII LiDAR files in the delivery for erroneous data, prints the bounds of the ascii file || || || classify_las || Binary (C) || LiDAR classification || || || classify_las_files.sh || bash || Runs classify_las on a set of LAS files using a specified classification algorithm || || || create_latex_lidar_readme.py || Python || Creates a LaTeX file for generating a PDF read-me for delivery || || || [wiki:Processing/demcompare demcompare.py] || Used to compare lidar patched aster dems and raw aster dems || stgo || || fugroviewer.exe || bash || Runs wine version of Fugroviewer || || || fwf_extract.py || python || Extracts waveform data for specified area || || || getDEMbufferpy || Python || Get the buffer size around LiDAR data for use when creating a DEM for use with apl || || || get_class_las || c++ || Copy the classification from one las file to another || unknown - adbe? || || get_first_return.sh || bash || Creates new ASCII lidar files with 1st returns only || || || import_ldr_txt_to_grass.sh || bash || Imports lidar ascii file into grass (run from within grass) || mark1 || || lag || Binary (C++) || LiDAR file viewer || harg + chrfi || || las2txt.sh || bash || Convert a directory of LAS files or a single LAS file into ascii format || || || las_hdr_pop.py || Python || Populates LAS file headers with information that may have been lost: min and max of x,y,z and number of points by return || knpa || || las_merge.sh || bash || Merges sets of LAS files of the form ***_000.LAS, ***_001.LAS... etc. together. Can also split a LAS file into several. || || || las_sampler.py || Python || Samples a LAS file at a specified interval and creates an output LAS file of these points || knpa || || las_single_class.py || Python || Classifies all points as a user defined number (quite slow) || knpa || || LASsorter || Python || sorts Las files by time, x, y, z , ascending and descending || || || leica_photo_processing || bash || Runs wine version of Leica's photo processor || || || lidardem.sh || Bash || Create a DEM from LiDAR data either las of ascii format || || || lidar_aster_dem.sh || Bash || Create a DEM from LiDAR and aster data, wrapper from lidardem.sh and asterdem.sh || || || lidar_stats.py || Python || Extracts statistics from a set of LAS files || emca || || make_lidar_delivery.py || bash || Create a LiDAR delivery folder || || || [wiki:Processing/CreateTifs#lidar_intensity.sh lidar_intensity.sh] || bash || Create a lidar intensity image from ascii / las + output jpgs + use vectors || || || pt_cloud_filter || Binary (C) || Removes points of a specified classification from ascii lidar point cloud || knpa || || split_las.py || Python || Splits las files using lasmerge. || tipo || == Hyperspectral == || '''Script''' || '''Language''' || '''Use''' || '''Author''' || || adjustBPF.py || Python || Generate a bad pixel file in CSV for a delivery, uses the CSV bad pixel file from calibration || || || aplxml_delivery_wrapper || Python || Generate an aplxml command for a flight in delivery format || || || bil_binning.py || Python || Spectrally bins (x2) a hyperspectral raw file || knpa || || bounding_box.py || Python || Get the North/South/East/West extent of a number of raw hdr files in bng coords || knpa || || cal_radiometric.py || Python || Generates a calibration curve in each band given input data in BIL or BSQ format and a known lamp file || benj || || correlate.py || Python || Wrapper for phasecorrelator.py || || || check_apl_cmd || bash || Runs the apl commands in the config file for all level1 flightlines, output to tmp_cmd_check dir || adbe || || check_bil.py || Python || Checks quality of bil files in hyperspectral delivery || adbe || || check_fsm.py || Python || Works out the corrected value for frame shift smear at a given band assuming a constant DN value through the spectrum || benj || || check_qpftiming.py || Python || Test specified Eagle/Hawk framerates against actual framerates calculated from qpfTiming || benj || || checkpeaks.py || Python || Matches measured spectral lines to fityk-detected peaks and eagle/hawk pixel numbers for use in spectral calibration || benj || || create_latex_hyperspectral_readme.py || Python || Create a LaTeX file for a PDF Readme for Hyperspectral azgcorr deliveries || || || create_latex_hyperspectral_apl_readme.py || Python || Create a LaTeX file for a PDF Readme for Hyperspectral APL deliveries || || || dimfixer.py || Python || Truncates BIL files so that the file size matches what the header file says it should || benj || || dummy_linecount.py || Python || Replaces a broken line counter on the first band of eagle and hawk (values 0 to number of lines) || knpa || || fastcheckoverflows.py || Python || Checks a BIL or BSQ file to see if it contains a specified overflow value || benj || || fastQC || Binary (C) || Program for viewing BIL files (raw data or level 1) || harg || || gen_bandsets.py || Python || Generates bandset (.prn, .bnd, .wls) files from a wavelength scale file || benj || || gen_synth_radcal.py || Python || Generates a synthetic Eagle or Hawk calibration file || benj || || [wiki:Processing/SyntheticDataset gen_synth_raw.py] || Python || Generates a synthetic Eagle or Hawk dataset || benj || || [wiki:Processing/SyntheticDataset gen_synth_sbet.py] || Python || Generates a synthetic SBET file || benj || || generate_prn.py || Python || Generates a .prn file from some subset of bands in a given .wls file || benj || || generate_apl_runscripts.py || Python || Generates config file for APL hyperspectral processing || || || get_ascii_fov_for_level1.py || Python || Generates an ascii file containing binned and trimmed sensor fov view vectors for level 1 files || mark1 || || get_boresight_vals.py || Python || Extracts boresight values for given day from csv file || || || get_calib_vals.py || Python || Extracts calibration file names for given day from data file || || || gen_eg_cmds.py || Python || Generates example commands for APL for the read me file || || || get_sbet_bounds.py || Python || Gets North/South/East/West extent of (part of) an SBET file in lat/lon || benj || || getposition || Python || Gets position information at from an SBET file for a specified time (or file limits) || benj || || getsync.py || Python || Finds sync records on one or more AISA nav files || || || gettiles.sh || bash || Determines BNG grid squares for flight from Eagle/Hawk header files || benj || || hdr_fps_check.py || Python || Checks if a Specim file has a mismatch between recorded and actual frame rate (Hawk problem) || || || hyper_latex_table_tools.py || Python || Python tools for hyperspectral readme table creation || || || igm2map.sh [[br]] igm2map.pro || bash || Script to use IGM files to generate a GLT file and map level 1 data || || || interleave_convert.py || Python || Convert interleave of a 3D file between BIL/BSQ || || || logwriter.py || Python || Outputs mean altitude, heading and speed from an SBET file for a number of Eagle or Hawk lines || benj || || make_mosaic.sh [[br]] make_mosaic.grass || bash || Script to convert GeoTIFFs into JPGs and create a mosaic image (with vectors) || || || make_hyper_delivery.py || Python || Make a hyperspectral delivery fro APL data || || || match_dark_frames.py || Python || Checks which of a selection of Eagle/Hawk header files have compatible binning, framerate and integration time settings || benj || || mergeCFSync.py || Python || Merges a .cfg and a .sync (eagle/hawk line sct offsets) || || || nav2txt || Binary (C) || Output contents of an AISA nav file to an ASCII text file || || || new_nav_file_fixer.py || Python || Script to remove once-per-second sync messages from a eagle/hawk nav file that were introduced with the firmware upgrade in October 2011 and sometimes break aplcorr || knpa || || phasecorrelator.py || Python || Perform phase correlation on a lidar intensity image and a hyperspectral image || || || process_specim_apl_line.py || Python || Process the specim data using APL || || || qsub_correlate.py || Python || Submit a correlation job to the grid || || || process_specim_line.py || Python || Runs az suite on specified Eagle/Hawk flightline for processing || benj || || run_calibration.py || Python || Master script for running calibration scripts || benj || || split_autodark.py || Python || Script to split Eagle or Hawk raw files containing autodark frames into a light file and a dark file. || benj || || splitline.py || Python || Script/Library for splitting Eagle/Hawk flightlines, using headers || || || slice_nav.py || Python || Cuts a chunk of an AISA nav file out into a smaller nav file || || || specim_gap_check.py || Python || Searches Eagle/Hawk raw files for dropped frames by looking at the band 1 frame counter || || || specim_qsub.py || Python || Submits an ARSF project for processing through the grid engine || benj || || split_bils.py || Python || Splits hyperspectral lines using apl processing || tipo || || update_delivery_mask_headers.py || Python || Re-writes the comment in the APL level 1 mask header files so that it refers to re-named level-1 file. Run after delivery has been created || || || wavelength_scale.py || Python || Generates a new wavelength scale from matching spectral peaks || benj || == Camera == || '''Script''' || Language || Use || Author || || create_latex_camera_readme.py || Python || Creates a LaTeX file for generating a PDF read_me for delivery || || || image_events_file_fixer.py || Python || Corrects image event files, mainly by using a sol/sbet file to correct position accuracy || || || make_camera_delivery.py || Python || Create a camera delivery || || || photo2thumb.sh || bash || Converts RCD TIF files to JPG thumbnails || || || rcdimages_in_pdf.py || Python || Converts RCD TIF files into JPG thumbnails using lidar log files to filter out of bounds images || || || rename_photos_update_events_file.py || Python || generates new file names for tiffs and their entries in the image events file || || || tag_tiffs.py || Python || Adds tags to tiff files (using image event files) which define position, attitude and other metadata || || || pitch_roll_correction.py || Python || Swap pitch and roll in tiff if different from that in csv file || || == General == || '''Script''' || Language || Use || Author(s) || || add_to_dem.sh || bash/grass || Adds an ascii or binary grid to a DEM - useful when doing geoid-spheroid seperation || knpa || || archiving_tarballer.sh || bash || Tarballs projects for NEODC || mggr/knpa || || asterdem.sh || Bash || Creates a ASTER DEM for use with aplcorr || knpa/adbe || || atcor || bash || Runs ATCOR-4 || || || [wiki:Processing/VecToRast auto_grass_vector_tiff_conversion_driver.sh auto_grass_vector_tiff_conversion_control_script.grass] || Python || uses grass to produce tiff images of vectors || || || binary_converter.py || Python || Converts binary files from one data type to another (eg. Float32 -> Int16) || || || bil2tiff || Bash || Convert bil files to geotiffs using gdal || || || [wiki:Processing/LLToBNG bnglookup.py] || Python || Converts BNG numeric grid references to OS grid square co-ordinates || || || build_structure.py || Python || Allows easy building of project directory structure from the command line. || || || checkoverflows.py || Python || Checks binary data files for presence of a specified overflow value || || || convert_delivery.sh || Bash || Will convert a delivery to a new project code and PI name. Can also be used on single deliveries to correct PI/proj-code inconsistencies || || || convert_nonuk_dem.sh || Bash / Grass || Convert a non-UK DEM into APL format (may have to be in WGS84 heights?) || || || convert_uk_dem.sh || Bash / Grass || Convert a UK DEM into APL format || || || [wiki:Processing/CreateImagePy createimage.py ] || Python || Convert file formats and create images || || || data_arithmetic.py || Python || Performs an arithmetic operation on all of the values from two data files and writes the results to an output data file || || || demgen.py || Python || Reads eagle or hawk header files and SBET file to generate a list of the DEM tiles that the SBET covers || || || demhdr_az2grass.sh || bash || Converts dems from az format header to grass header || || || demheaderconvert.sh || bash || Converts dems from grass format header to az header || || || doc2pdf.py || Python || Converts .doc or .odt files to PDF format || || || embargocheck.py || Python || Tests if a project folder is still in embargo (1 year from last delivered data) || stgo || || !DocumentConverter.py || Python || Converts between various document formats using Open Office || || || findbaseline.py || Python || Script to find the baseline value of a list of numbers containing value spikes. Intended for part of Specim cal procedure. || || || fix_naughty_chars.py || Python || Finds linux-unfriendly characters in filenames || || || fixnames_sct.py || Python || Auto-renames (line numbers and removes sct section) level 1, level 3, screenshots or rcd images/thumbnails/raws for a delivery. || || || freelook || bash || Runs ENVI Freelook || || || gen_fityk_script.py || Python || Generates a fityk script file that can be used to fit peaks to particular datasets || || || generate_readme_config.py || Python || Creates a config file for hyperspectral and camera read_me generators || || || generateKMLforPI.py || Python || Generate a KML file containing Eagle, Hawk or RCD information || || || [wiki:Processing/OsVecImport GML2Shape.sh] || bash || Convert GML Vector files to Shapefiles || || || gpstime || Python || Converts GPS time-of-week to human-readable format and vice-versa || || || hdf_output.py || Python || Lists contents of an HDF file to the terminal || || || hdf_to_bil.py || Python || Extracts navigation data from hdf files to a bil file || chrfi/knpa || || julian || CShell || Converts between julian day+year and day/month/year || || || kmloverview_all_projects_this_year.py || Python || Generate KML Overview for all projects in repository for a given year || || || [wiki:Processing/LLToBNG ll2bng.sh] || bash || Converts specified lat/lon into BNG co-ordinates (wrapper around appropriate proj command) || || || lsolder || bash || Like ls, but can filter list to get files/directories older than a certain time || || || mask_bad_pixels.py || Python || Sets defined bad pixels in a BIL or BSQ file to be a mask value || || || make_delivery_letter_email.py || Python || Automatically produce delivery letters and emails for a PI || || || [wiki:Processing/makekmloverview make_kmloverview.py] || Python || Make the KML file and overview support files - part of unpacking procedure || || || [wiki:Processing/NextMapDEMs nextmapdem.sh] || bash || Generates a DEM from !NextMap data || benj/mggr/knpa || || projdir || bash || Automatically CDs to a project directory || anch || || proj_tidy.sh || bash || Checks that a specified project conforms to the current defined structure and file name formats || knpa || || raw_eagle_hawk_bil2jpg.sh [[br]] raw_eagle_hawk_bil2jpg.grass || Create JPGs of raw eagle/hawk data for KML overview || || || spatial_average.py || Python || Averages a BIL or BSQ file in the spatial direction || || || symlink_src_dir.sh || bash || Symlinks all files in a source directory into a target directory || || || teqc || Binary (third-party) || Program for slicing/splicing/fiddling with RINEX or other GPS base station data (like raw data) || || || times4grafnav.py || Python || Searches E/H headers and gets start/stop times for importing into Grafnav to show on trajectory || || || trmdatefinder.py || Python || Extracts date from Trimble GPS format file names || || == Libraries == || '''Script''' || Language || Use || Author || || alphaconv.py || Python || Functions to convert numbers to letters and back (A=1, B=2 etc) || || || arsf_database.py || Python || Functions to access key/value ARSF database (please use arsf_db_reader.py) || || || arsf_db_reader.py || Python || Provides classes and functions to read from the ARSF database || || || band_reader.py || Python || Reads AISA band (.prn) files || || || configSorter.py || Python || Sorts config files into sensible order (can also be executed directly) || || || convert_interpolate.py || Python || Interpolates an array and/or many arrays wrt. one array. Finds peaks/Derivates/Integrates for 1D arrays. || || || common_readme_functions.py || Python || Functions for latex readme generator scripts || || || correlator_reader.py || Python || Read data in that is required for the correlator || || || csv_handler.py || Python || Reads CSV columns as lists and writes lists to a CSV file || || || data_handler.py || Python || Reads/writes BIL and BSQ files || || || flightlineclass.py || Python || Class to hold info on Eagle/Hawk flight lines (used in generateKMLforPI.py) || || || folder_structure.py || Python || Holds the current arsf folder structure || || || geographic.py || Python || For converting between different geographic coordinate systems || knpa || || GRASS_library.py || Python || For working with Envi DEM files in grass || stgo || || hyperconfigutils.py || Python || Set of functions for reading and checking hyperspectral config files || || || kml_writer.py || Python || Creates and writes KML files. || || || las1_3_handler.py || Python || Reads LAS 1.3 files || || || libgpstime.py || Python || Converts GPS time-of-week to human-readable format and vice-versa || || || liblogwriter.py || Python || Gets mean altitude, heading and speed from an SBET file for a number of Eagle or Hawk lines || || || navparser.py || Python || Basic Specim nav file parser for ASCII nav files || || || project_info.py || Python || Returns basic project info (project code, jday etc) from an available source || || || rcdclasses.py || Python || Class for holding RCD image information (used in generateKMLforPI.py) || || || sbet_handler.py || Python || Functions to manipulate SBET files, also limited AISA nav file support || || || sol_handler.py || Python || Sol handling library || chrfi and adth || == Obsolete == ''These are not in everyday use but are still useful on occasion'' || '''Script''' || Language || Use || Author || || arsf_processing_monitor.py || Python || Part of old processing job submission system || || || az_log_parser.py || Python || Parse output from az* programs to remove cruft - not kept up to date with newer versions and possibly too ambitious || || || create_lidar_readme.py || Python || Creates a .txt ReadMe for lidar deliveries || || || fastcopy_arsf_proj_dir.sh || bash || Script to make a workspace copy of a project directory (ie. avoid copying raw files, etc) || || || [wiki:Processing/LIDARDEMs generate_mask.py & generate_mask.sh] || Python || Script for generating mask file for use in GRASS for old Optech LiDAR || || || generate_runscripts.py || Python || Generates AZ runscripts or config files for hyperspectral data processing || mark1/benj || || gtiff2jpg.py || Python || Convert GeoTIFFs to JPEGs || || || kqsub || bash || Handy for quick submission of individual flightlines and sct values. Designed for use with old rune/runh scripts || || || kmlise_project.py || Python || || || || [wiki:Processing/LIDARDEMs lidar2dem.sh] || bash || Generates an azgcorr-format ASCII DEM from old Optech lidar point cloud files || || || [wiki:Processing/LIDARDEMs lidar_histogram.py & lidar_histogram.sh] || Python || Reads old Optech LiDAR files and outputs a numeric histogram for use in filtering || || || make_delivery_folder.sh || bash || Old script for making deliveries || || || ppv_to_envi_igm.py || Python || Converts azgcorr's ppv (per-pixel view vector) output to an ENVI IGM || chrfi || || trim_lidar.sh || bash || Cuts UTM zone number off old Optech LiDAR point cloud files || || || unpack_file_check.py || Python || Unpacking check files script || || || unpack_folder_structure.py || Python || Unpacking check the folder structure || || || [wiki:Processing/CreateTifs make_lidardem_or_intensity.sh make_lidardem_or_intensity.grass] || bash || Create a lidar dem or intensity image from ascii lidar files || || == Symlinks == Explanation of symlinks in ~arsf/usr/bin || Link name || Link target || Use || || configSorter || /users/rsg/arsf/usr/lib/python/site-packages/configSorter.py || Link to configSorter library (see Libraries section) - library is also executable || == Unknown == Note: See also Python resources/libraries on main RSG wiki at http://rsg.pml.ac.uk/intranet/trac/wiki/Python