Alsproc Development

Architecture Overview

It is worth noting the overall architecture of alsproc, which has been structured in a way similar to APL where users generate configuration files from a template which they can then edit, and each configuration file should have a deterministic outcome when processed using the main program.

Structure

Alsproc is built around the proprietary alspro.dll, forcing it to take the form of a winelib program, something which isn't quite native to either Windows or Linux. Some complications arise out of this, such as barring the use of gdb. A python program wraps around alsproc to handle broader more context-specific logic, doing things like reading in .cfg files and computing the correct arguments to pass.

Generally, the structure of the program is arranged by branches where alsproc.c is the root of the tree and branches are things like scn parsing, sup reading, navigation reading, etcetera. Output formats are modular, making them very easy to add and remove. Parameters passed to the program externally should be passed around internally using pointers to a config_struct structure, to help avoid dramatic changes in the API on a regular basis among other things, any developer should be sure to use it. The broadness of its use could be expanded at the time of writing.

Arguments are passed into alsproc.c and processing into a config_struct structure, which is passed to processor.c which commissions each branch and passes the data into alspro.dll and gives the result to output.c.

SCN reading and interpreting is the most complex part of the problem, split into 3 abstraction layers. At the first level, scn_stream.c, raw data is read in from the relevant files and serialised. At the next layer, scn_record.c, the program makes sense of the raw data casting it into records and validating them, dropping any garbage. At the third layer, scn_interpreter.c, records are combined with meta-data and a-priori knowledge to produce meaningful interpretations of each record.

Output modules can be added by defining 3 functions and a structure instance: functions for each of opening, writing, and closing; and a structure to hold pointers to those functions. Access to these should be exposed in a header file, while actual code should be contained in their own source file. out_txt is a good example of an output module, which outputs point data to a .txt file simply translating points to human readable text using fprintf, it can be copied and used as a template for any future modules. Once you have a module, edit output.c and add a clause for your module in open_output_stream which translates config->output_format to your structure holding pointers to your functions, and stores the result in out->fmt.

else if (strcmp(config->output_format, "netcdf")==0)
  out->fmt = &NETCDF_FORMAT;

SCN reading is arguably the most complex part of the program. It's best not to debug it directly in alsproc if at all possible, as this bars the use of gdb. Instead, consider using the scn_info program, which is compiled normally but still uses the same code for reading SCN records.

GTX Files

As ARSF often delivers data projected in the British National Grid, and Alsproc performs all projection operations using proj.4, performing the OSGM02 transformation specified by Ordnance Survey is essential to producing vertically accurate data. proj.4 can only perform grid-shift transformations vertically using .gtx files. For this purpose, a .gtx file had to be produced from the Ordnance Survey data from their website. You may need to produce or fine tune a .gtx file, so the tools used to generate them have been retained and are available in the svn repository under tools/gtx/.

You also require proj of at least version 4.8.0, the source code is available online, but be warned that as of the time of writing this doesn't compile out-of-the-box. It will compile, but requires minor adjustments to one source file and possibly the Makefile.

LAS Files

LAS files can be written with the las12_las format specifier, which makes use of the las12 module. This is a minimal module made to provide access to the basic data types used in the LAS 1.2 format. When the las12_las format is specified the wrapper script will detect this and generate a header using the lashwrite program. Be warned that this is not strictly a good practice, since it relies on the close co-operation of two pieces of software which ought to be interchangeable. With the header written, the main program will simply tack the data onto the end of the given LAS file, then seek backwards and update any relevant metadata.

Ideally in future this would be implemented entirely in the C program. The reason this isn't the case now is because of the effort involved in passing meta data into and around the program. This kind of problem weighs in favour of implementing a reader for .cfg files in alsproc and allowing components in the program to query the reader as they require. Otherwise, the maintainer will face the problem of keeping within the character limit for each line of Bash in passing arguments into alsproc directly via command line. This kind of solution is part of a bigger discussion which has not yet been resolved, justifying the semi-temporary solution currently in-place.

Subversion

Alsproc can be found in subversion under tools/alsproc. Obtain a copy with svn checkout.

svn checkout https://rsg.pml.ac.uk/intranet/svnroots/arsf-internal/trunk/tools/alsproc

You can make and manage your changes in this checkout. When you've tested it and you're happy, you may need to advance the version number. Edit the Makefile in src/, and find the line which says vers=x.x.x and advance it as you need.

# VERSION NUMBER
vers=0.0.9

When you're satisfied, commit the changes.

svn commit

And finally, tag the branch version if you advanced the version number.

svn cp https://rsg.pml.ac.uk/intranet/svnroots/arsf-internal/trunk/tools/alsproc https://rsg.pml.ac.uk/intranet/svnroots/arsf-internal/tags/alsproc-0.0.9 -m "tagging new version of alsproc"

Testing

Testing alsproc is easily possible without first installing it system-wide. All scripts should be able to run from the scripts/ directory. To test the processor itself you can build it as normal, and configure your .cfg file to point to this new executable file, rather than the official installed one, but note you will need to copy the .dll file into the src/ directory for any processing to work.

Installation in ARSF

Building alsproc requires some extra dependencies that are not required for runtime:

glibc-devel.i686
wine-devel.i686
proj-devel.i686

Other known dependencies required for runtime are:

proj.i686

Current policy is to install in the home directory of the arsf user, as this asserts that all users across the department are using the same and current version of alsproc. Enter usr/opt/alsproc and check out the version you wish to build, check out the repository, compile it, and install it locally using the purpose-built Makefile rule: arsf_install.

cd ~arsf/usr/opt/alsproc
svn checkout https://rsg.pml.ac.uk/intranet/svnroots/arsf-internal/tags/alsproc-0.0.9
cd alsproc-0.0.9/src
make
make arsf_install

To Do

• Analysis and refinement of accuracy of individual points (ongoing)
• Geotagging las files (partially done)
• adding log and config files as VLRs to las files
• Better verification of proj4 strings
• Better verbose mode, current verbose mode is more like debug mode
• Better mechanism for passing values into alsproc, with checks for required arguments
• More reliable detection of test flightlines or verification of current methods
• scn_info is very slow for what it does, it should be able to skip unneeded data
• las12_las output module is very inter-dependant on the calling script, it should ideally be fixed or replaced

Common Problems

proj.i686

Q: What does this error mean.

wine: Bad EXE format for Z:\users\rsg\arsf\usr\opt\alsproc\alsproc.exe.so.

A: I don't know why it says that when you're missing dependencies, but on ARSF workstations this is probably going to be resolved by installing the proj.i686 package.

GDB

Q: Why doesn't gdb work on alsproc.exe?

A: alsproc.exe is a winelib program, which means gdb annoyingly wont work with it. This is why alsproc needs better verbose and debugging modes. If you're really lucky you might be able to get winedbg to work.