Changes between Version 8 and Version 9 of Processing/lagdevelopersfaq


Ignore:
Timestamp:
Jul 6, 2012 3:56:50 PM (7 years ago)
Author:
jaho
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Processing/lagdevelopersfaq

    v8 v9  
    22[#q1 1. General] \\
    33[#q1.1 1.1 What is LAG?] \\
    4 [#q1.2 1.2 What are major components of LAG?] \\
     4[#q1.2 1.2 What are the main components of LAG?] \\
    55[#q1.3 1.3 Why is LAG's code so messy?] \\
    66[#q1.4 1.4 How can I make LAG's code better?] \\
    7 [#q1.5 1.5 Where can I learn good C++?] \\
     7[#q1.5 1.5 Where can I learn some good C++?] \\
    88[#q1.6 1.6 Who can I ask for help?] \\
    99
     
    4040[#q5.7 5.7 How do I use profiling tools to optimise my code?] \\
    4141
     42----
     43
    4244== 1. General [=#q1] ==
    4345
    4446=== 1.1 What is LAG? [=#q1.1] ===
    4547
    46 LAG stands for [http://en.wikipedia.org/wiki/LIDAR LiDAR] Analysis GUI. It is a software which provides graphical user interface for processing and visualisation of LiDAR [http://en.wikipedia.org/wiki/Point_cloud point cloud] data. LAG is written in C++ and uses Gtk ([http://www.gtkmm.org/en/ gtkmm]) for its GUI, and [http://www.opengl.org/ OpenGL] for rendering.
    47 
    48 === 1.2 What are major components of LAG [=#q1.2] ===
    49 
    50 LAG consist of the main GUI application and the Lidar Quadtree library used for storing and querying the data. It also uses laslib library for handling LAS files.
     48LAG stands for [http://en.wikipedia.org/wiki/LIDAR LiDAR] Analysis GUI. It is a software which provides graphical user interface for processing and visualisation of LiDAR [http://en.wikipedia.org/wiki/Point_cloud point clouds]. LAG is written in C++ and uses Gtk ([http://www.gtkmm.org/en/ gtkmm]) for its GUI, and [http://www.opengl.org/ OpenGL] for rendering.
     49
     50=== 1.2 What are the main components of LAG [=#q1.2] ===
     51
     52LAG consist of the main GUI application and the Lidar Quadtree library used for storing and manipulating the data. It also uses laslib library for handling LAS files.
    5153
    5254=== 1.3 Why is LAG's code so messy? [=#q1.3] ===
    5355
    54 The lag has been in development since 2009 and there were at least four different people working on it, each with different coding style, ideas and barely any supervision. It is now much better then it used to be, but it's easy to see that different parts have been written by different people. There also still seems to be a lot of temporary solutions which were never implemented properly, since they worked at the time. //
     56The lag has been in development since 2009 and there were at least four different people working on it, each with different coding style, ideas and barely any supervision. It is now much better then it used to be, but it's easy to see that different parts have been written by different people. Some parts of the code contain temporary solutions which were never implemented properly, since they worked at the time. Just because they work though, doesn't mean they are good solutions in terms of efficiency, maintainability etc. There is still a lot of room for improvement. //
    5557
    5658=== 1.4 How can I make LAG's code better? [=#q1.4] ===
    5759
    58 Think before you code. Take users of your classes into consideration and try to design easy to use interfaces. Do some refactoring if you think it's going to make things easier to maintain, and to understand the code better. Use consistent coding style and comments. Try to learn good coding practices before you start writing the code (eg. implementing const correctness from the start is much easier then adding it once everything has been written). Profile your code instead of relying on what seems faster.
    59 
    60 === 1.5 Where can I learn good C++? [=#q1.5] ===
    61 
    62 Whether you already know C++, come from C or are just starting to learn, it is a good idea to familiarise yourself with good programming practice before writing production code. Assuming you already know the basics of the language I highly recommend these two sources:
    63 
    64 [http://www.parashift.com/c++-faq-lite/ C++ FAQ] - every C++ programmer should read this at least once. It's well written, to the point and covers a lot of things from OO design to freestore management.
     60Think first, code later. Take users of your classes into consideration and try to design easy to use and logical interfaces. Do some refactoring if you think it's going to make things easier to maintain, and to understand the code better. Use consistent coding style and comments. \\ \\
     61Try to learn good coding practices before you start writing the code. It is better to spend some time planning and then writing an implementation, then writing a code that just works and then trying to fix it. For example implementing const correctness from the start is much easier then adding it once everything has been written. \\ \\
     62Profile your code instead of relying on your intuition of what "should be" faster. Use OO principles for the benefit of more robust code (eg. use polymorphism instead of huge blocks of if statements). Use common sense.
     63
     64=== 1.5 Where can I learn some good C++? [=#q1.5] ===
     65
     66Whether you already know C++, come from C or are just starting to learn, it is a good idea to familiarise yourself with good programming practices before writing production code. Assuming you already know the basics of the language I highly recommend these two sources:
     67
     68[http://www.parashift.com/c++-faq-lite/ C++ FAQ] - every C++ programmer should read. It's well written, to the point and covers a lot of things from OO design to freestore management.
    6569
    6670[http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml Google C++ Style Guide] - you don't have to blindly follow it, but it should give you an idea of what a good coding style looks like.
     
    6872=== 1.6 Who can I ask for help? [=#q1.6] ===
    6973
    70 One problem with LAG is that it has been entirely developed by students, so the only people that know how it really works are most likely gone (thus this FAQ). For general help with programming you should probably ask Mike Grant (he knows a bit about LAG) or Mark Warren as they seem to be the best programmers in ARSF. They also tend to be quite busy though so if you don't feel like bugging them [http://stackoverflow.com/ Stack Overflow] is always a good place to look for help.
     74One problem with LAG is that it has been entirely developed by students, so the only people that know how it really works are most likely gone (thus this FAQ). For general help with programming you should probably ask Mike Grant or Mark Warren as they seem to know a thing or two about it. They also tend to be quite busy though so if you don't feel like bugging them [http://stackoverflow.com/ Stack Overflow] is always a good place to look for help. \\
     75
     76----
    7177
    7278== 2. Laslib [=#q2] ==
     
    7884=== 2.2 What are LAS files? [=#q2.2] ===
    7985
    80 LAS is a binary, public file format designed to hold LiDAR points data. An alternative to LAS are ASCII files, which are however less efficient in terms of both processing time and file size. \\
    81 
    82 LAS files consist of several parts: Public Header Block, Variable Length Records, Point Records and in format 1.3 and later Waveform Records. \\
    83 
    84 One thing to note about LAS is that point coordinates are stored as scaled integers and are then unscaled upon loading to double values with the use of scale factors and offsets stored in the header.
     86LAS is a binary, public file format designed to hold LiDAR points data. One alternative to LAS are ASCII files(.csv or .txt), which are however less efficient in terms of both processing time and file size. \\ \\
     87
     88LAS files consist of several parts: Public Header Block, Variable Length Records, Point Records and in format 1.3 and later Waveform Records. \\ \\
     89
     90One thing to note about LAS is that point coordinates are stored as scaled integers and are then unscaled upon loading to double values with the use of scale factors and offsets stored in the header. \\ \\
    8591
    8692A detailed LAS format specification can be found at:
     
    9197In addition to discrete point records LAS 1.3 files also hold full waveform data which is quite huge in volume. The point records have also been modified to include a number of new attributes related to waveform data. \\ \\
    9298
     99New point format causes a problem to the quadtree library because point objects are much bigger in size thus affecting the performance and memory consumption. We don't currently have tools to process full waveform data, however we do sometimes include it in deliveries so we need some way of handling it.
     100
    93101=== 2.4 What do I need to know about laslib? [=#q2.4] ===
    94102
    95 There is no official documentation for laslib API, however it is a fairly well written C++ which provides an easy to use interface. The best way to learn laslib is to study the source code of programs included in lastools.
    96 
    97 The main classes of our interest are LASreader and LASwriter. A simple program that reads points from a file, filters out points with classification 7 (noise), prints points' coordinates and then saves them to another file may look like this (note there's no error checking or exception handling for simplicity, but you should always include it the real code):
     103There is no official documentation for laslib API, however it is a fairly well written C++ which provides an easy to use interface. The best way to learn laslib is to study the source code of programs included in lastools. It is quite powerful and even provides its own tools for spatial indexing (totally undocumented though). \\ \\
     104
     105The main classes of our interest are {{{LASreader}}} and {{{LASwriter}}}. A simple program that reads points from a file, filters out points with classification 7 (noise), prints points' coordinates and then saves them to another file may look like this (note there's no error checking or exception handling for simplicity, but you should always include it the real code):
    98106{{{
    99107#!div style="font-size: 100%"
     
    159167=== 2.5 How do I make laslib a shared library? [=#q2.5] ===
    160168
    161 It seems that laslib is mainly developed for Windows users so there are no targets for shared libraries in the Makefiles by default. At the same time shared libraries are needed by LAG to work correctly. To fix this you're going to need to modify the Makefiles and add -fPIC option to the compiler. You'll also have to change the name of the library from laslib to //liblaslib// for the linker to detect it.
     169It seems that laslib is mainly developed for Windows users so there are no targets for shared libraries in the Makefiles by default. At the same time shared libraries are needed by LAG to work correctly. To fix this you're going to need to modify the Makefiles and add {{{-fPIC}}} option to the compiler. You'll also have to change the name of the library from laslib to //liblaslib// for the linker to detect it. \\ \\
    162170
    163171You'll normally find modified Makefiles somewhere around, so you can copy them after downloading a new version of laslib and hopefully they will work. If this is not the case, below is the part that needs to be added to the Makefile inside laslib/src folder.
     
    195203}}}
    196204
    197 You're not likely to have to alter this part and if a newly downloaded version of laslib fails to build with modified Makefiles, the first thing to check is if OBJ_LAS (the list of object files) hasn't changed since the previous release.
    198 
    199 
    200 == 3. Quadtreee [#q3] ==
     205You're not likely to have to alter this part and if a newly downloaded version of laslib fails to build with modified Makefiles, the first thing to check is if {{{OBJ_LAS}}} (the list of object files) hasn't changed since the previous release.
     206
     207----
     208
     209== 3. Lidar Quadtreee [=#q3] ==
    201210
    202211== 3.1 How does Quadtree work? [=#q3.1] ==
    203212
    204 The Lidar Quadtree library provides a [http://en.wikipedia.org/wiki/Quadtree quadtree] data structure for storing and indexing points. It also contains a caching mechanism for loading and unloading points from memory.
    205 
    206 Each node of the quadtree represents a bounding box in x and y coordinates. Whenever a point is inserted into the quadtree it is determined which of the four nodes it falls into and then the same happens for each node recursively until a leaf node is found. Whenever the number of points in a node goes above the maximum capacity specified, the node splits into four.
     213The Lidar Quadtree library provides a [http://en.wikipedia.org/wiki/Quadtree quadtree] data structure for storing and indexing points. It also contains a caching mechanism for loading and unloading points from memory. \\ \\
     214
     215Each node of the quadtree represents a bounding box in x and y coordinates. Whenever a point is inserted into the quadtree it is determined which of the four nodes it falls into and then the same happens for each node recursively until a leaf node is found. Whenever the number of points in a node goes above the maximum capacity specified on creation of the quadtree, the node splits into four. \\ \\
    207216
    208217When retrieving points a bounding box is passed to the quadtree and all nodes which intersect with it are queried for the points. It's quite simple really.
     
    217226  Represents a single point in space with three coordinates: x, y, z.
    218227LidarPoint.cpp
    219   Represents a lidar point with coordinates (inherits from Point) and other attributes like time, intensity, classification etc. Note that size of the point directly affects the size and performance of the quadtree, thus only necessary attributes are held inside this class.
     228  Represents a lidar point with coordinates (inherits from {{{Point}}}) and other attributes like time, intensity, classification etc. Note that size of the point directly affects the size and performance of the quadtree, thus only necessary attributes are held inside this class.
    220229PointBucket.cpp
    221   A flat data structure which actually holds the points. Each quadtree node holds a pointer to a PointBucket which holds the points in an array of LidarPoints. The PointBucket is also responsible for caching and uncaching of data.
     230  A flat data structure which actually holds the points. Each quadtree node holds a pointer to a {{{PointBucket}}} which holds the points in an array of {{{LidarPoint}}} objects. The {{{PointBucket}}} is also responsible for caching and uncaching of data.
    222231PointData.cpp
    223232  A struct to hold LAS 1.3+ point attributes which are stored separately, outside the quadtree.
     
    227236=== 3.3 What is the reason behind subbuckets? [=#q3.3] ===
    228237
    229 The PointBucket class may hold copies of the same point in several arrays called sub-buckets. There are two parameters which control of how many sub-buckets are created: Resolution Depth and Resolution Base. The Resolution Depth determines the number of sub-buckets for each bucket and Resolution Base determines the number of points at each level by specifying the interval between included points based on a formula of Resolution Base^(Level - 1)^. For example the default LAG values of 5 and 4 create 4 sub-buckets per bucket: one containing every point (5^0^), second containing every 5th point (5^1^), third containing every 25th point (5^2^) and fourth with every 125th point (5^3^). //
    230 The reason for this is rendering of points. For example when viewing a whole flightline in the overview window there is no need to render every single point or store all the points in memory at once. Thus sub-buckets containing every n-th point are used instead. Note that you could just pass every-nth point for rendering, but it is really the memory usage and caching issue that sub-buckets solve. //
    231 Because each resolution level effectively slowers the quadtree (some points need to be inserted n times into n arrays) for applications other then LAG, which make no use of sub-buckets Resolution Depth and Resolution Base should be set to 1.
     238The {{{PointBucket}}} class may hold copies of the same point in several arrays called sub-buckets. There are two parameters which control of how many sub-buckets are created: //Resolution Depth// and //Resolution Base//. The //Resolution Depth// determines the number of sub-buckets for each bucket and //Resolution Base// determines the number of points at each level by specifying the interval between included points based on a formula of Resolution Base^(Level - 1)^. For example the default LAG values of 5 and 4 create 4 sub-buckets per bucket: one containing every point (5^0^), second containing every 5th point (5^1^), third containing every 25th point (5^2^) and fourth with every 125th point (5^3^). \\ \\
     239
     240The reason for this is rendering of points. For example when viewing a whole flightline in the overview window there is no need to render every single point or store all the points in memory at once. Thus sub-buckets containing every n-th point are used instead. Note that you could just pass every-nth point for rendering, but it is really the memory usage and caching issue that sub-buckets solve. \\ \\
     241
     242Because each resolution level effectively slowers the quadtree (some points need to be inserted n times into n arrays) for applications other then LAG, which make no use of sub-buckets, //Resolution Depth// and //Resolution Base// should be set to 1.
    232243
    233244=== 3.4 How does caching work? [=#q3.4] ===
     
    235246Upon creation of the quadtree the user specifies a maximum number of points to be held in cache.
    236247
     248
     249
     250
    237251=== 3.5 What is the reason behind data compression? [=#q3.5] ===
    238252
    239 To make caching faster. //
     253To make caching faster. \\ \\
     254
    240255PointBucket uses [http://www.oberhumer.com/opensource/lzo/ lzo compression] to compress buckets before writing them to disk. It may seem like a sub-optimal solution but, since quadtree's performance is mainly IO bound, the compression time is still much lower then reading/writing to the hard drive. Since the volume of data gets smaller thanks to compression in the end it speeds up IO operations which are the major bottleneck.
    241256
    242257=== 3.6 What are LasSaver and LasLoader classes in the quadtree? [=#q3.6] ===
    243258
    244 These should be removed at some point together with geoprojectionconverter. They are currently there only for backwards compatibility with other programs that use the quadtree (classify_las). \\ \\
    245 
    246 From design point of view these classes should not be a part of the quadtree which should only serve as a data container. In practice, when these classes were used, making any changes in LAG regarding saving or loading files required alterations to the quadtree which was less then convenient. Now that loading and saving has been moved to separate classes inside LAG it is much more maintainable. Points inspection. \\
     259These should be removed at some point together with {{{geoprojectionconverter}}}. They are currently there only for backwards compatibility with other programs that use the quadtree (classify_las). \\ \\
     260
     261From design point of view these classes should not be a part of the quadtree which should only serve as a data container. In practice, when these classes were used, making any changes in LAG regarding saving or loading files required alterations to the quadtree which was less then convenient. Now that loading and saving has been moved to separate classes inside LAG it is much more maintainable. Additionally it allows individual point inspection during loading and saving, which was not possiblebe before.
     262
     263----
    247264
    248265== 4. LAG [=#q4] ==