ARSF Scripts and Binaries
Please keep this page updated. Also please keep it sorted in a sensible order or we're never going to be able to find anything...
LiDAR
| Script | Language | Use | Author |
| alspp.exe | bash | Runs wine version of ALSPP | |
| arsf_winesetup.sh | bash | Configure wine environment for ARSF users | |
| attune | bash | Runs wine version of Attune (boresight calibration) | |
| batch_fw_class.sh | bash | Runs get_class_las on full directories, so it doesn't have to be done one by one. Does NOT check if las files are sorted | dap |
| check_ascii_lidar.sh | bash | Checks the ASCII LiDAR files in the delivery for erroneous data | |
| check_ascii_lidar | Binary (C) | Checks the ASCII LiDAR files in the delivery for erroneous data, prints the bounds of the ascii file | |
| classify_las | Binary (C) | LiDAR classification | |
| classify_las_files.sh | bash | Runs classify_las on a set of LAS files using a specified classification algorithm | |
| create_latex_lidar_readme.py | Python | Creates a LaTeX file for generating a PDF read-me for delivery | |
| demcompare.py | Python | Used to compare lidar patched aster dems and raw aster dems | stgo |
| fugroviewer.exe | bash | Runs wine version of Fugroviewer | |
| fwf_extract.py | python | Extracts waveform data for specified area | |
| getDEMbufferpy | Python | Get the buffer size around LiDAR data for use when creating a DEM for use with apl | |
| get_class_las | c++ | Copy the classification from one las file to another | unknown - adbe? |
| get_first_return.sh | bash | Creates new ASCII lidar files with 1st returns only | |
| import_ldr_txt_to_grass.sh | bash | Imports lidar ascii file into grass (run from within grass) | mark1 |
| lag | Binary (C++) | LiDAR file viewer | harg + chrfi |
| las2txt.sh | bash | Convert a directory of LAS files or a single LAS file into ascii format | |
| las_hdr_pop.py | Python | Populates LAS file headers with information that may have been lost: min and max of x,y,z and number of points by return | knpa |
| las_merge.sh | bash | Merges sets of LAS files of the form *_000.LAS, *_001.LAS... etc. together. Can also split a LAS file into several. | |
| las_sampler.py | Python | Samples a LAS file at a specified interval and creates an output LAS file of these points | knpa |
| las_single_class.py | Python | Classifies all points as a user defined number (quite slow) | knpa |
| LASsorter | Python | sorts Las files by time, x, y, z , ascending and descending | |
| leica_photo_processing | bash | Runs wine version of Leica's photo processor | |
| lidardem.sh | Bash | Create a DEM from LiDAR data either las of ascii format | |
| lidar_aster_dem.sh | Bash | Create a DEM from LiDAR and aster data, wrapper from lidardem.sh and asterdem.sh | |
| lidar_stats.py | Python | Extracts statistics from a set of LAS files | emca |
| make_lidar_delivery.py | bash | Create a LiDAR delivery folder | |
| lidar_intensity.sh? | bash | Create a lidar intensity image from ascii / las + output jpgs + use vectors | |
| pt_cloud_filter | Binary (C) | Removes points of a specified classification from ascii lidar point cloud | knpa |
| split_las.py | Python | Splits las files using lasmerge. | tipo |
Hyperspectral
| Script | Language | Use | Author |
| adjustBPF.py | Python | Generate a bad pixel file in CSV for a delivery, uses the CSV bad pixel file from calibration | |
| aplxml_delivery_wrapper | Python | Generate an aplxml command for a flight in delivery format | |
| bil_binning.py | Python | Spectrally bins (x2) a hyperspectral raw file | knpa |
| bounding_box.py | Python | Get the North/South/East/West extent of a number of raw hdr files in bng coords | knpa |
| cal_radiometric.py | Python | Generates a calibration curve in each band given input data in BIL or BSQ format and a known lamp file | benj |
| correlate.py | Python | Wrapper for phasecorrelator.py | |
| check_apl_cmd | bash | Runs the apl commands in the config file for all level1 flightlines, output to tmp_cmd_check dir | adbe |
| check_bil.py | Python | Checks quality of bil files in hyperspectral delivery | adbe |
| check_fenix_spectra_py6s.py | Python | Plots the at-sensor radiance spectra recorded by hyperspectral sensor and compare it to the at-sensor Py6S model | asm/dac |
| check_fsm.py | Python | Works out the corrected value for frame shift smear at a given band assuming a constant DN value through the spectrum | benj |
| check_qpftiming.py | Python | Test specified Eagle/Hawk framerates against actual framerates calculated from qpfTiming | benj |
| checkpeaks.py | Python | Matches measured spectral lines to fityk-detected peaks and eagle/hawk pixel numbers for use in spectral calibration | benj |
| create_latex_hyperspectral_readme.py | Python | Create a LaTeX file for a PDF Readme for Hyperspectral azgcorr deliveries | |
| create_latex_hyperspectral_apl_readme.py | Python | Create a LaTeX file for a PDF Readme for Hyperspectral APL deliveries | |
| dimfixer.py | Python | Truncates BIL files so that the file size matches what the header file says it should | benj |
| dummy_linecount.py | Python | Replaces a broken line counter on the first band of eagle and hawk (values 0 to number of lines) | knpa |
| fastcheckoverflows.py | Python | Checks a BIL or BSQ file to see if it contains a specified overflow value | benj |
| fastQC | Binary (C) | Program for viewing BIL files (raw data or level 1) | harg |
| gen_bandsets.py | Python | Generates bandset (.prn, .bnd, .wls) files from a wavelength scale file | benj |
| gen_synth_radcal.py | Python | Generates a synthetic Eagle or Hawk calibration file | benj |
| gen_synth_raw.py? | Python | Generates a synthetic Eagle or Hawk dataset | benj |
| gen_synth_sbet.py? | Python | Generates a synthetic SBET file | benj |
| generate_prn.py | Python | Generates a .prn file from some subset of bands in a given .wls file | benj |
| generate_apl_runscripts.py | Python | Generates config file for APL hyperspectral processing | |
| get_ascii_fov_for_level1.py | Python | Generates an ascii file containing binned and trimmed sensor fov view vectors for level 1 files | mark1 |
| get_boresight_vals.py | Python | Extracts boresight values for given day from csv file | |
| get_calib_vals.py | Python | Extracts calibration file names for given day from data file | |
| gen_eg_cmds.py | Python | Generates example commands for APL for the read me file | |
| get_sbet_bounds.py | Python | Gets North/South/East/West extent of (part of) an SBET file in lat/lon | benj |
| getposition | Python | Gets position information at from an SBET file for a specified time (or file limits) | benj |
| getsync.py | Python | Finds sync records on one or more AISA nav files | |
| gettiles.sh | bash | Determines BNG grid squares for flight from Eagle/Hawk header files | benj |
| hdr_fps_check.py | Python | Checks if a Specim file has a mismatch between recorded and actual frame rate (Hawk problem) | |
| hyper_latex_table_tools.py | Python | Python tools for hyperspectral readme table creation | |
| igm2map.sh igm2map.pro | bash | Script to use IGM files to generate a GLT file and map level 1 data | |
| interleave_convert.py | Python | Convert interleave of a 3D file between BIL/BSQ | |
| logwriter.py | Python | Outputs mean altitude, heading and speed from an SBET file for a number of Eagle or Hawk lines | benj |
| make_mosaic.sh make_mosaic.grass | bash | Script to convert GeoTIFFs into JPGs and create a mosaic image (with vectors) | |
| make_hyper_delivery.py | Python | Make a hyperspectral delivery fro APL data | |
| match_dark_frames.py | Python | Checks which of a selection of Eagle/Hawk header files have compatible binning, framerate and integration time settings | benj |
| mergeCFSync.py | Python | Merges a .cfg and a .sync (eagle/hawk line sct offsets) | |
| nav2txt | Binary (C) | Output contents of an AISA nav file to an ASCII text file | |
| NCom2SBET | Binary (C) | Convert NCom .nav files to sbet format | dac |
| new_nav_file_fixer.py | Python | Script to remove once-per-second sync messages from a eagle/hawk nav file that were introduced with the firmware upgrade in October 2011 and sometimes break aplcorr | knpa |
| phasecorrelator.py | Python | Perform phase correlation on a lidar intensity image and a hyperspectral image | |
| plot_1b_spectra.py | Python | Plot spectral profile from BIL file for a given pixel and line | dac |
| process_specim_apl_line.py | Python | Process the specim data using APL | |
| qsub_correlate.py | Python | Submit a correlation job to the grid | |
| process_specim_line.py | Python | Runs az suite on specified Eagle/Hawk flightline for processing | benj |
| run_calibration.py | Python | Master script for running calibration scripts | benj |
| sbet_datum_offset.py | Python | Change altitudes in sbet file from EGM96 datum to WGS-84 elipsoid | dac |
| split_autodark.py | Python | Script to split Eagle or Hawk raw files containing autodark frames into a light file and a dark file. | benj |
| splitline.py | Python | Script/Library for splitting Eagle/Hawk flightlines, using headers | |
| slice_nav.py | Python | Cuts a chunk of an AISA nav file out into a smaller nav file | |
| specim_gap_check.py | Python | Searches Eagle/Hawk raw files for dropped frames by looking at the band 1 frame counter | |
| specim_qsub.py | Python | Submits an ARSF project for processing through the grid engine | benj |
| split_bils.py | Python | Splits hyperspectral lines using apl processing | tipo |
| update_delivery_mask_headers.py | Python | Re-writes the comment in the APL level 1 mask header files so that it refers to re-named level-1 file. Run after delivery has been created | |
| wavelength_scale.py | Python | Generates a new wavelength scale from matching spectral peaks | benj |
Camera
| Script | Language | Use | Author |
| create_latex_camera_readme.py | Python | Creates a LaTeX file for generating a PDF read_me for delivery | |
| image_events_file_fixer.py | Python | Corrects image event files, mainly by using a sol/sbet file to correct position accuracy | |
| make_camera_delivery.py | Python | Create a camera delivery | |
| photo2thumb.sh | bash | Converts RCD TIF files to JPG thumbnails | |
| rcdimages_in_pdf.py | Python | Converts RCD TIF files into JPG thumbnails using lidar log files to filter out of bounds images | |
| rename_photos_update_events_file.py | Python | generates new file names for tiffs and their entries in the image events file | |
| tag_tiffs.py | Python | Adds tags to tiff files (using image event files) which define position, attitude and other metadata | |
| pitch_roll_correction.py | Python | Swap pitch and roll in tiff if different from that in csv file |
General
| Script | Language | Use | Author(s) |
| add_to_dem.sh | bash/grass | Adds an ascii or binary grid to a DEM - useful when doing geoid-spheroid seperation | knpa |
| archiving_tarballer.sh | bash | Tarballs projects for NEODC | mggr/knpa |
| asterdem.sh | Bash | Creates a ASTER DEM for use with aplcorr | knpa/adbe |
| atcor | bash | Runs ATCOR-4 | |
| auto_grass_vector_tiff_conversion_driver.sh auto_grass_vector_tiff_conversion_control_script.grass | Python | uses grass to produce tiff images of vectors | |
| binary_converter.py | Python | Converts binary files from one data type to another (eg. Float32 -> Int16) | |
| bil2tiff | Bash | Convert bil files to geotiffs using gdal | |
| bnglookup.py | Python | Converts BNG numeric grid references to OS grid square co-ordinates | |
| build_structure.py | Python | Allows easy building of project directory structure from the command line. | |
| checkoverflows.py | Python | Checks binary data files for presence of a specified overflow value | |
| convert_delivery.sh | Bash | Will convert a delivery to a new project code and PI name. Can also be used on single deliveries to correct PI/proj-code inconsistencies | |
| convert_nonuk_dem.sh | Bash / Grass | Convert a non-UK DEM into APL format (may have to be in WGS84 heights?) | |
| convert_uk_dem.sh | Bash / Grass | Convert a UK DEM into APL format | |
| createimage.py | Python | Convert file formats and create images | |
| data_arithmetic.py | Python | Performs an arithmetic operation on all of the values from two data files and writes the results to an output data file | |
| demgen.py | Python | Reads eagle or hawk header files and SBET file to generate a list of the DEM tiles that the SBET covers | |
| demhdr_az2grass.sh | bash | Converts dems from az format header to grass header | |
| demheaderconvert.sh | bash | Converts dems from grass format header to az header | |
| doc2pdf.py | Python | Converts .doc or .odt files to PDF format | |
| embargocheck.py | Python | Tests if a project folder is still in embargo (1 year from last delivered data) | stgo |
| DocumentConverter.py | Python | Converts between various document formats using Open Office | |
| findbaseline.py | Python | Script to find the baseline value of a list of numbers containing value spikes. Intended for part of Specim cal procedure. | |
| fix_naughty_chars.py | Python | Finds linux-unfriendly characters in filenames | |
| fix_undessirable_chars.py | Python | Finds linux-unfriendly characters in filenames, solves some problems found on fix_naughty_chars.py and do not need to be run as part of a pipe | asm |
| fixnames_sct.py | Python | Auto-renames (line numbers and removes sct section) level 1, level 3, screenshots or rcd images/thumbnails/raws for a delivery. | |
| freelook | bash | Runs ENVI Freelook | |
| gen_fityk_script.py | Python | Generates a fityk script file that can be used to fit peaks to particular datasets | |
| generate_readme_config.py | Python | Creates a config file for hyperspectral and camera read_me generators | |
| generateKMLforPI.py | Python | Generate a KML file containing Eagle, Hawk or RCD information | |
| GML2Shape.sh | bash | Convert GML Vector files to Shapefiles | |
| gpstime | Python | Converts GPS time-of-week to human-readable format and vice-versa | |
| hdf_output.py | Python | Lists contents of an HDF file to the terminal | |
| hdf_to_bil.py | Python | Extracts navigation data from hdf files to a bil file | chrfi/knpa |
| julian | CShell | Converts between julian day+year and day/month/year | |
| kmloverview_all_projects_this_year.py | Python | Generate KML Overview for all projects in repository for a given year | |
| ll2bng.sh | bash | Converts specified lat/lon into BNG co-ordinates (wrapper around appropriate proj command) | |
| lsolder | bash | Like ls, but can filter list to get files/directories older than a certain time | |
| mask_bad_pixels.py | Python | Sets defined bad pixels in a BIL or BSQ file to be a mask value | |
| make_delivery_letter_email.py | Python | Automatically produce delivery letters and emails for a PI | |
| make_kmloverview.py | Python | Make the KML file and overview support files - part of unpacking procedure | |
| nextmapdem.sh | bash | Generates a DEM from NextMap data | benj/mggr/knpa |
| projdir | bash | Automatically CDs to a project directory | anch |
| proj_tidy.sh | bash | Checks that a specified project conforms to the current defined structure and file name formats | knpa |
| raw_eagle_hawk_bil2jpg.sh raw_eagle_hawk_bil2jpg.grass | Create JPGs of raw eagle/hawk data for KML overview | ||
| spatial_average.py | Python | Averages a BIL or BSQ file in the spatial direction | |
| symlink_src_dir.sh | bash | Symlinks all files in a source directory into a target directory | |
| teqc | Binary (third-party) | Program for slicing/splicing/fiddling with RINEX or other GPS base station data (like raw data) | |
| times4grafnav.py | Python | Searches E/H headers and gets start/stop times for importing into Grafnav to show on trajectory | |
| trmdatefinder.py | Python | Extracts date from Trimble GPS format file names | |
| unpack_project.py | Python | Moves/copies files and folders from given structure to our structure using a config file approach (unpacking_template.cfg) | asm |
Libraries
| Script | Language | Use | Author |
| alphaconv.py | Python | Functions to convert numbers to letters and back (A=1, B=2 etc) | |
| arsf_database.py | Python | Functions to access key/value ARSF database (please use arsf_db_reader.py) | |
| arsf_db_reader.py | Python | Provides classes and functions to read from the ARSF database | |
| band_reader.py | Python | Reads AISA band (.prn) files | |
| configSorter.py | Python | Sorts config files into sensible order (can also be executed directly) | |
| convert_interpolate.py | Python | Interpolates an array and/or many arrays wrt. one array. Finds peaks/Derivates/Integrates for 1D arrays. | |
| common_readme_functions.py | Python | Functions for latex readme generator scripts | |
| correlator_reader.py | Python | Read data in that is required for the correlator | |
| csv_handler.py | Python | Reads CSV columns as lists and writes lists to a CSV file | |
| data_handler.py | Python | Reads/writes BIL and BSQ files | |
| egm96 | Python / Fortran | Calculate offset between EGM96 and WGS-84. Python wrapper for Fortran functions | dac |
| flightlineclass.py | Python | Class to hold info on Eagle/Hawk flight lines (used in generateKMLforPI.py) | |
| folder_structure.py | Python | Holds the current arsf folder structure | |
| geographic.py | Python | For converting between different geographic coordinate systems | knpa |
| GRASS_library.py | Python | For working with Envi DEM files in grass | stgo |
| hyperconfigutils.py | Python | Set of functions for reading and checking hyperspectral config files | |
| kml_writer.py | Python | Creates and writes KML files. | |
| las1_3_handler.py | Python | Reads LAS 1.3 files | |
| libgpstime.py | Python | Converts GPS time-of-week to human-readable format and vice-versa | |
| liblogwriter.py | Python | Gets mean altitude, heading and speed from an SBET file for a number of Eagle or Hawk lines | |
| navparser.py | Python | Basic Specim nav file parser for ASCII nav files | |
| project_info.py | Python | Returns basic project info (project code, jday etc) from an available source | |
| rcdclasses.py | Python | Class for holding RCD image information (used in generateKMLforPI.py) | |
| sbet_handler.py | Python | Functions to manipulate SBET files, also limited AISA nav file support | |
| sol_handler.py | Python | Sol handling library | chrfi and adth |
Obsolete
These are not in everyday use but are still useful on occasion
| Script | Language | Use | Author |
| arsf_processing_monitor.py | Python | Part of old processing job submission system | |
| az_log_parser.py | Python | Parse output from az* programs to remove cruft - not kept up to date with newer versions and possibly too ambitious | |
| create_lidar_readme.py | Python | Creates a .txt ReadMe for lidar deliveries | |
| fastcopy_arsf_proj_dir.sh | bash | Script to make a workspace copy of a project directory (ie. avoid copying raw files, etc) | |
| generate_mask.py & generate_mask.sh | Python | Script for generating mask file for use in GRASS for old Optech LiDAR | |
| generate_runscripts.py | Python | Generates AZ runscripts or config files for hyperspectral data processing | mark1/benj |
| gtiff2jpg.py | Python | Convert GeoTIFFs to JPEGs | |
| kqsub | bash | Handy for quick submission of individual flightlines and sct values. Designed for use with old rune/runh scripts | |
| kmlise_project.py | Python | ||
| lidar2dem.sh | bash | Generates an azgcorr-format ASCII DEM from old Optech lidar point cloud files | |
| lidar_histogram.py & lidar_histogram.sh | Python | Reads old Optech LiDAR files and outputs a numeric histogram for use in filtering | |
| make_delivery_folder.sh | bash | Old script for making deliveries | |
| ppv_to_envi_igm.py | Python | Converts azgcorr's ppv (per-pixel view vector) output to an ENVI IGM | chrfi |
| trim_lidar.sh | bash | Cuts UTM zone number off old Optech LiDAR point cloud files | |
| unpack_file_check.py | Python | Unpacking check files script | |
| unpack_folder_structure.py | Python | Unpacking check the folder structure | |
| make_lidardem_or_intensity.sh make_lidardem_or_intensity.grass? | bash | Create a lidar dem or intensity image from ascii lidar files |
Symlinks
Explanation of symlinks in ~arsf/usr/bin
| Link name | Link target | Use |
| configSorter | /users/rsg/arsf/usr/lib/python/site-packages/configSorter.py | Link to configSorter library (see Libraries section) - library is also executable |
Unknown
Note: See also Python resources/libraries on main RSG wiki at http://rsg.pml.ac.uk/intranet/trac/wiki/Python
Last modified 9 years ago
Last modified on Mar 1, 2016, 12:27:31 PM