Version 124 (modified by knpa, 13 years ago) (diff) |
---|
ARSF Scripts and Binaries
Please keep this page updated. Also please keep it sorted in a sensible order or we're never going to be able to find anything...
LiDAR
Script | Language | Use | Author |
alspp.exe | bash | Runs wine version of ALSPP | |
arsf_winesetup.sh | bash | Configure wine environment for ARSF users | |
attune | bash | Runs wine version of Attune (boresight calibration) | |
check_ascii_lidar.sh | bash | Checks the ASCII LiDAR files in the delivery for erroneous data | |
check_ascii_lidar | Binary (C) | Checks the ASCII LiDAR files in the delivery for erroneous data, prints the bounds of the ascii file | |
classify_las | Binary (C) | LiDAR classification | |
classify_las_files.sh | bash | Runs classify_las on a set of LAS files using a specified classification algorithm | |
create_latex_lidar_readme.py | Python | Creates a LaTeX file for generating a PDF read-me for delivery | |
fugroviewer.exe | bash | Runs wine version of Fugroviewer | |
getDEMbufferpy | Python | Get the buffer size around LiDAR data for use when creating a DEM for use with apl | |
get_first_return.sh | bash | Creates new ASCII lidar files with 1st returns only | |
import_ldr_txt_to_grass.sh | bash | Imports lidar ascii file into grass (run from within grass) | mark1 |
lag | Binary (C++) | LiDAR file viewer | harg + chrfi |
las2txt.sh | bash | Convert a directory of LAS files or a single LAS file into ascii format | |
las_hdr_pop.py | Python | Populates LAS file headers with information that may have been lost: min and max of x,y,z and number of points by return | knpa |
las_merge.sh | bash | Merges sets of LAS files of the form *_000.LAS, *_001.LAS... etc. together | |
las_sampler.py | Python | Samples a LAS file at a specified interval and creates an output LAS file of these points | knpa |
las_single_class.py | Python | Classifies all points as a user defined number (quite slow) | knpa |
LASsorter | Python | sorts Las files by time, x, y, z , ascending and descending | |
leica_photo_processing | bash | Runs wine version of Leica's photo processor | |
lidardem.sh | Bash | Create a DEM from LiDAR data either las of ascii format | |
lidar_aster_dem.sh | Bash | Create a DEM from LiDAR and aster data, wrapper from lidardem.sh and asterdem.sh | |
lidar_stats.py | Python | Extracts statistics from a set of LAS files | emca |
make_lidar_delivery.sh | bash | Create a LiDAR delivery folder | |
make_lidardem_or_intensity.sh make_lidardem_or_intensity.grass | bash | Create a lidar dem or intensity image from ascii lidar files | |
pt_cloud_filter | Binary (C) | Removes points of a specified classification from ascii lidar point cloud | knpa |
Hyperspectral
Script | Language | Use | Author |
adjustBPF.py | Python | Generate a bad pixel file in CSV for a delivery, uses the CSV bad pixel file from calibration | |
bil_binning.py | Python | Spectrally bins (x2) a hyperspectral raw file | knpa |
bounding_box.py | Python | Get the North/South/East/West extent of a number of raw hdr files in bng coords | knpa |
cal_radiometric.py | Python | Generates a calibration curve in each band given input data in BIL or BSQ format and a known lamp file | benj |
correlate.py | Python | Wrapper for phasecorrelator.py | |
check_apl_cmd | bash | Runs the apl commands in the config file for all level1 flightlines, output to tmp_cmd_check dir | adbe |
check_bil.py | Python | Checks quality of bil files in hyperspectral delivery | adbe |
check_fsm.py | Python | Works out the corrected value for frame shift smear at a given band assuming a constant DN value through the spectrum | benj |
check_qpftiming.py | Python | Test specified Eagle/Hawk framerates against actual framerates calculated from qpfTiming | benj |
checkpeaks.py | Python | Matches measured spectral lines to fityk-detected peaks and eagle/hawk pixel numbers for use in spectral calibration | benj |
create_latex_hyperspectral_readme.py | Python | Create a LaTeX file for a PDF Readme for Hyperspectral deliveries | |
dimfixer.py | Python | Truncates BIL files so that the file size matches what the header file says it should | benj |
dummy_linecount.py | Python | Replaces a broken line counter on the first band of eagle and hawk (values 0 to number of lines) | knpa |
fastcheckoverflows.py | Python | Checks a BIL or BSQ file to see if it contains a specified overflow value | benj |
fastQC | Binary (C) | Program for viewing BIL files (raw data or level 1) | harg |
gen_bandsets.py | Python | Generates bandset (.prn, .bnd, .wls) files from a wavelength scale file | benj |
gen_synth_radcal.py | Python | Generates a synthetic Eagle or Hawk calibration file | benj |
gen_synth_raw.py | Python | Generates a synthetic Eagle or Hawk dataset | benj |
gen_synth_sbet.py | Python | Generates a synthetic SBET file | benj |
generate_prn.py | Python | Generates a .prn file from some subset of bands in a given .wls file | benj |
get_boresight_vals.py | Python | Extracts boresight values for given day from csv file | |
get_calib_vals.py | Python | Extracts calibration file names for given day from data file | |
get_sbet_bounds.py | Python | Gets North/South/East/West extent of (part of) an SBET file in lat/lon | benj |
getposition | Python | Gets position information at from an SBET file for a specified time (or file limits) | benj |
getsync.py | Python | Finds sync records on one or more AISA nav files | |
gettiles.sh | bash | Determines BNG grid squares for flight from Eagle/Hawk header files | benj |
hdr_fps_check.py | Python | Checks if a Specim file has a mismatch between recorded and actual frame rate (Hawk problem) | |
igm2map.sh igm2map.pro | bash | Script to use IGM files to generate a GLT file and map level 1 data | |
interleave_convert.py | Python | Convert interleave of a 3D file between BIL/BSQ | |
logwriter.py | Python | Outputs mean altitude, heading and speed from an SBET file for a number of Eagle or Hawk lines | benj |
make_mosaic.sh make_mosaic.grass | bash | Script to convert GeoTIFFs into JPGs and create a mosaic image (with vectors) | |
match_dark_frames.py | Python | Checks which of a selection of Eagle/Hawk header files have compatible binning, framerate and integration time settings | benj |
mergeCFSync.py | Python | Merges a .cfg and a .sync (eagle/hawk line sct offsets) | |
nav2txt | Binary (C) | Output contents of an AISA nav file to an ASCII text file | |
new_nav_file_fixer.py | Python | Script to remove once-per-second sync messages from a eagle/hawk nav file that were introduced with the firmware upgrade in October 2011 and sometimes break aplcorr | knpa |
phasecorrelator.py | Python | Perform phase correlation on a lidar intensity image and a hyperspectral image | |
qsub_correlate.py | Python | Submit a correlation job to the grid | |
process_specim_line.py | Python | Runs az suite on specified Eagle/Hawk flightline for processing | benj |
run_calibration.py | Python | Master script for running calibration scripts | benj |
split_autodark.py | Python | Script to split Eagle or Hawk raw files containing autodark frames into a light file and a dark file. | benj |
splitline.py | Python | Script/Library for splitting Eagle/Hawk flightlines, using headers | |
slice_nav.py | Python | Cuts a chunk of an AISA nav file out into a smaller nav file | |
specim_gap_check.py | Python | Searches Eagle/Hawk raw files for dropped frames by looking at the band 1 frame counter | |
specim_qsub.py | Python | Submits an ARSF project for processing through the grid engine | benj |
wavelength_scale.py | Python | Generates a new wavelength scale from matching spectral peaks | benj |
Camera
Script | Language | Use | author |
create_latex_camera_readme.py | Python | Creates a LaTeX file for generating a PDF read_me for delivery | |
image_events_file_fixer.py | Python | Corrects image event files, mainly by using a sol/sbet file to correct position accuracy | |
make_camera_delivery.py | Python | Create a camera delivery | |
photo2thumb.sh | bash | Converts RCD TIF files to JPG thumbnails | |
rename_photos_update_events_file.py | Python | generates new file names for tiffs and their entries in the image events file | |
tag_tiffs.py | Python | Adds tags to tiff files (using image event files) which define position, attitude and other metadata | |
pitch_roll_correction.py | Python | Swap pitch and roll in tiff if different from that in csv file |
General
Script | Language | Use | Author(s) |
add_to_dem.sh | bash/grass | Adds an ascii or binary grid to a DEM - useful when doing geoid-spheroid seperation | knpa |
archiving_tarballer.sh | bash | Tarballs projects for NEODC | mggr/knpa |
asterdem.sh | Bash | Creates a ASTER DEM for use with aplcorr | knpa/adbe |
atcor | bash | Runs ATCOR-4 | |
auto_grass_vector_tiff_conversion_driver.sh auto_grass_vector_tiff_conversion_control_script.grass | Python | uses grass to produce tiff images of vectors | |
binary_converter.py | Python | Converts binary files from one data type to another (eg. Float32 -> Int16) | |
bnglookup.py | Python | Converts BNG numeric grid references to OS grid square co-ordinates | |
checkoverflows.py | Python | Checks binary data files for presence of a specified overflow value | |
convert_delivery.sh | Bash | Will convert a delivery to a new project code and PI name. Can also be used on single deliveries to correct PI/proj-code inconsistencies | |
data_arithmetic.py | Python | Performs an arithmetic operation on all of the values from two data files and writes the results to an output data file | |
demgen.py | Python | Reads eagle or hawk header files and SBET file to generate a list of the DEM tiles that the SBET covers | |
demhdr_az2grass.sh | bash | Converts dems from az format header to grass header | |
demheaderconvert.sh | bash | Converts dems from grass format header to az header | |
doc2pdf.py | Python | Converts .doc or .odt files to PDF format | |
DocumentConverter.py | Python | Converts between various document formats using Open Office | |
findbaseline.py | Python | Script to find the baseline value of a list of numbers containing value spikes. Intended for part of Specim cal procedure. | |
fix_naughty_chars.py | Python | Finds linux-unfriendly characters in filenames | |
fixnamesl13.py | Python | Auto-renames (line numbers and removes sct section) level 1, level 3, screenshots or rcd images/thumbnails/raws for a delivery. | |
freelook | bash | Runs ENVI Freelook | |
gen_fityk_script.py | Python | Generates a fityk script file that can be used to fit peaks to particular datasets | |
generate_readme_config.py | Python | Creates a config file for hyperspectral and camera read_me generators | |
generateKMLforPI.py | Python | Generate a KML file containing Eagle, Hawk or RCD information | |
GML2Shape.sh | bash | Convert GML Vector files to Shapefiles | |
gpstime | Python | Converts GPS time-of-week to human-readable format and vice-versa | |
hdf_output.py | Python | Lists contents of an HDF file to the terminal | |
hdf_to_bil | Python | Extracts navigation data from hdf files to a bil file | chrfi/knpa |
julian | CShell | Converts between julian day+year and day/month/year | |
ll2bng.sh | bash | Converts specified lat/lon into BNG co-ordinates (wrapper around appropriate proj command) | |
lsolder | bash | Like ls, but can filter list to get files/directories older than a certain time | |
mask_bad_pixels.py | Python | Sets defined bad pixels in a BIL or BSQ file to be a mask value | |
make_delivery_letter_email.py | Python | Automatically produce delivery letters and emails for a PI | |
nextmapdem.sh | bash | Generates a DEM from NextMap data | benj/mggr/knpa |
projdir | bash | Automatically CDs to a project directory | anch |
proj_tidy.sh | bash | Checks that a specified project conforms to the current defined structure and file name formats | knpa |
spatial_average.py | Python | Averages a BIL or BSQ file in the spatial direction | |
symlink_src_dir.sh | bash | Symlinks all files in a source directory into a target directory | |
teqc | Binary (third-party) | Program for slicing/splicing/fiddling with RINEX or other GPS base station data (like raw data) | |
times4grafnav.py | Python | Searches E/H headers and gets start/stop times for importing into Grafnav to show on trajectory | |
trmdatefinder.py | Python | Extracts date from Trimble GPS format file names |
Libraries
Script | Language | Use | |
alphaconv.py | Python | Functions to convert numbers to letters and back (A=1, B=2 etc) | |
arsf_database.py | Python | Functions to access key/value ARSF database (please use arsf_db_reader.py) | |
arsf_db_reader.py | Python | Provides classes and functions to read from the ARSF database | |
band_reader.py | Python | Reads AISA band (.prn) files | |
configSorter.py | Python | Sorts config files into sensible order (can also be executed directly) | |
convert_interpolate.py | Python | Interpolates an array and/or many arrays wrt. one array. Finds peaks/Derivates/Integrates for 1D arrays. | |
common_readme_functions.py | Python | Functions for latex readme generator scripts | |
correlator_reader.py | Python | Read data in that is required for the correlator | |
csv_handler.py | Python | Reads CSV columns as lists and writes lists to a CSV file | |
data_handler.py | Python | Reads/writes BIL and BSQ files | |
flightlineclass.py | Python | Class to hold info on Eagle/Hawk flight lines (used in generateKMLforPI.py) | |
kml_writer.py | Python | Creates and writes KML files. | |
libgpstime.py | Python | Converts GPS time-of-week to human-readable format and vice-versa | |
liblogwriter.py | Python | Gets mean altitude, heading and speed from an SBET file for a number of Eagle or Hawk lines | |
ppv_to_envi_igm.py | Python | Converts azgcorr's ppv (per-pixel view vector) output to an ENVI IGM | |
rcdclasses.py | Python | Class for holding RCD image information (used in generateKMLforPI.py) | |
sbet_handler.py | Python | Functions to manipulate SBET files, also limited AISA nav file support | |
sol_reader.py | Python | Sol reading library | chrfi |
Obsolete
These are not in everyday use but are still useful on occasion
Script | Language | Use | |
arsf_processing_monitor.py | Python | Part of old processing job submission system | |
az_log_parser.py | Python | Parse output from az* programs to remove cruft - not kept up to date with newer versions and possibly too ambitious | |
create_lidar_readme.py | Python | Creates a .txt ReadMe for lidar deliveries | |
fastcopy_arsf_proj_dir.sh | bash | Script to make a workspace copy of a project directory (ie. avoid copying raw files, etc) | |
generate_mask.py generate_mask.sh | Python | Script for generating mask file for use in GRASS for old Optech LiDAR | |
generate_runscripts.py | Python | Generates AZ runscripts or config files for hyperspectral data processing | mark1/benj |
gtiff2jpg.py | Python | Convert GeoTIFFs to JPEGs | |
kqsub | bash | Handy for quick submission of individual flightlines and sct values. Designed for use with old rune/runh scripts | |
kmlise_project.py | Python | ||
lidar2dem.sh | bash | Generates an azgcorr-format ASCII DEM from old Optech lidar point cloud files | |
lidar_histogram.py lidar_histogram.sh | Python | Reads old Optech LiDAR files and outputs a numeric histogram for use in filtering | |
make_delivery_folder.sh | bash | Old script for making deliveries | |
trim_lidar.sh | bash | Cuts UTM zone number off old Optech LiDAR point cloud files | |
unpack_file_check.py | Python | Unpacking check files script | |
unpack_folder_structure.py | Python | Unpacking check the folder structure |
If it's obsolete and been replaced by something newer / better, delete the obsoleted one. If we don't have a replacement, it may be worth keeping around but remember the cost in confusion.
Symlinks
Explanation of symlinks in ~arsf/usr/bin
Link name | Link target | Use |
configSorter | /users/rsg/arsf/usr/lib/python/site-packages/configSorter.py | Link to configSorter library (see Libraries section) - library is also executable |
Unknown
Script | Language | Use |
atm_bad_scans.py | Python | |
casefixer.py | Python | |
combined_band_mask.py | symbolic link to `/users/rsg/arsf/code/mask_creation/combined_band_mask.py' | |
delivery_prep_2006.sh | bash | |
hdf_to_bil.py | Python | |
kml_line_photosorter.py | Python | |
LASsorter.py | Python | |
overlay_combined_band_mask.py | symbolic link to `/users/rsg/arsf/code/mask_creation/overlay_combined_band_mask.py' | |
photo_renamer.py | Python | |
read_arsf_logsheets.py | Python | |
remove_bands_from_header.py | Python | |
seperate_band_mask.py | symbolic link to `/users/rsg/arsf/code/mask_creation/seperate_band_mask.py' | |
tifftagger2_exiftool.py | Python |
Note: See also Python resources/libraries on main RSG wiki at http://rsg.pml.ac.uk/intranet/trac/wiki/Python