Changes between Version 5 and Version 6 of Processing/lagdevelopersfaq


Ignore:
Timestamp:
Jul 4, 2012 6:43:50 PM (7 years ago)
Author:
jaho
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Processing/lagdevelopersfaq

    v5 v6  
    44[#q3 3. How do I edit the GUI?] \\
    55[#q4 4. What are LAS files?] \\
     6
     7[#q4.1 4.1 What's the deal with LAS 1.3 files? \\
     8
    69[#q5 5. What do I need to know about laslib?] \\
    710[#q6 6. How do I make laslib a shared library?] \\
    8 
    9 [#q5 Why is LAG's code so messy?] \\
    10 [#q5 How can I make it better?] \\
    11 [#q2 How do I edit the GUI?] \\
    12 [#q3 What are LAS files?] \\
    13 [#q5 What is laslib?] \\
    14 [#q4 How does Quadtree work?] \\
    15 [#q5 How does Cacheminder work?] \\
    16 [#q5 How does loading of files work?] \\
    17 [#q5 How does saving of files work?] \\
    18 [#q5 How does rendering work?] \\
    19 [#q5 What about the LASsaver and LASloader in the quadtree?] \\
    20 [#q5 What are some major issues with LAG that need work?] \\
    21 [#q5 What are some additional features that can be added to LAG?] \\
    22 [#q5 What tools are there to help me with LAG development?] \\
     11[#q7 7. Why is LAG's code so messy?] \\
     12[#q8 8. How can I make LAG's code better?] \\
     13[#q9 9. Where can I learn good C++ coding practices?] \\
     14[#q10 10. Who should I ask for help?] \\
     15[#q11 11. How does Quadtree work?] \\
     16[#q12 12. How is Quadtree structured?] \\
     17[#q13 13. What is the reason behind subbuckets?] \\
     18[#q14 14. How does caching work?] \\
     19[#q15 15. What is the reason behind data compression?] \\
     20[#q16 16. What are LasSaver and LasLoader classes in the quadtree?] \\
     21
     22[#q17 17. How is LAG's code structured?] \\
     23[#q18 18. How does loading of files work?] \\
     24[#q19 19. How does saving of files work?] \\
     25[#q21 21. How is LAS 1.3 point format handled?] \\
     26[#q22 22. How does rendering work?] \\
     27
     28[#q23 23. What are main ideas for further LAG development?] \\
     29[#q24 24. What are some major issues with LAG that need fixing?] \\
     30[#q25 25. What are some additional features that can be added to LAG?] \\
     31[#q26 26. What tools are there to help me with LAG development?] \\
     32
    2333[#q5 Question X?] \\
    2434[#q5 Question X?] \\
     
    6878== 4. What are LAS files? [=#q4] ==
    6979
    70 LAS is a binary, public file format designed to hold LiDAR point data. An alternative to LAS are ASCII files, which are however much less efficient in terms of both processing time and file size.
    71 
    72 LAS files consist of several parts: Public Header Block, Variable Length Records, Point Records and in format 1.3 and later Waveform Records.
     80LAS is a binary, public file format designed to hold LiDAR point data. An alternative to LAS are ASCII files, which are however much less efficient in terms of both processing time and file size. \\
     81
     82LAS files consist of several parts: Public Header Block, Variable Length Records, Point Records and in format 1.3 and later Waveform Records. \\
     83
     84One thing to note about LAS is that point coordinates are stored as scaled integers and are then unscaled upon loading to double values with the use of scale factors and offsets stored in the header.
    7385
    7486A detailed LAS format specification can be found at:
     
    143155}}}
    144156
    145 == 6. Ho do I make laslib a shared library? [=#q6] ==
     157== 6. How do I make laslib a shared library? [=#q6] ==
    146158
    147159It seems that laslib is mainly developed for Windows users so there are no targets for shared libraries in the Makefiles by default. At the same time shared libraries are needed by LAG to work correctly. To fix this you're going to need to modify the Makefiles and add -fPIC option to the compiler. You'll also have to change the name of the library from laslib to //liblaslib// for the linker to detect it.
     
    182194
    183195You're not likely to have to alter this part and if a newly downloaded version of laslib fails to build with modified Makefiles, the first thing to check is if OBJ_LAS (the list of object files) hasn't changed since the previous release.
     196
     197== 7. Why is LAG's code so messy? [=#q7] ==
     198
     199The lag has been in development since 2009 and there were at least four different people working on it, each with different coding style, ideas and barely any supervision. It is now much better then it used to be, but it's easy to see that different parts have been written by different people. There also still seems to be a lot of temporary solutions which were never implemented properly, since they worked at the time. //
     200
     201== 8. How can I make LAG's code better? [=#q8] ==
     202
     203Think before you code. Take users of your classes into consideration and try to design easy to use interfaces. Do some refactoring if you think it's going to make things easier to maintain, and to understand the code better. Use consistent coding style and comments. Try to learn good coding practices before you start writing the code (eg. implementing const correctness from the start is much easier then adding it once everything has been written).
     204
     205== 9. Where can I learn good C++? [=#q9] ==
     206
     207Whether you already know C++, come from C or are just starting to learn, it is a good idea to familiarise yourself with good programming practice before writing production code. Assuming you already know the basics of the language I highly recommend these two sources:
     208
     209[http://www.parashift.com/c++-faq-lite/ C++ FAQ] - every C++ programmer should read this at least once. It's well written, to the point and covers a lot of things from OO design to freestore management.
     210
     211[http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml Google C++ Style Guide] - you don't have to blindly follow it, but it should give you an idea of what a good coding style looks like.
     212
     213== 10. Who can I ask for help? [=#q10] ==
     214
     215One problem with LAG is that it has been entirely developed by students, so the only people that know how it really works are most likely gone (thus this FAQ). For general help with programming you should probably ask Mike Grant (he knows a bit about LAG) or Mark Warren as they seem to be the best programmers in ARSF. They also tend to be quite busy though so if you don't feel like bugging them [http://stackoverflow.com/ Stack Overflow] is always a good place to look for help.
     216
     217== 11. How does Quadtree work? [=#q11] ==
     218
     219The Lidar Quadtree library provides a [http://en.wikipedia.org/wiki/Quadtree quadtree] data structure for storing and indexing points. It also contains a caching mechanism for loading and unloading points from memory.
     220
     221Each node of the quadtree represents a bounding box in x and y coordinates. Whenever a point is inserted into the quadtree it is determined which of the four nodes it falls into and then the same happens for each node recursively until a leaf node is found. Whenever the number of points in a node goes above the maximum capacity specified, the node splits into four.
     222
     223When retrieving points a bounding box is passed to the quadtree and all nodes which intersect with it are queried for the points. It's quite simple really.
     224
     225== 12. How is quadtree's code structured? [=#q12] ==
     226
     227Quadtree.cpp
     228  Provides an interface for the quadtree. It is the only class that should be used from the outside. It also holds metadata about the quadtree and a pointer to the root node.
     229QuadtreeNode.cpp
     230  Represents a single quadtree node. Provides methods for inserting and retrieving the points.
     231Point.cpp
     232  Represents a single point in space with three coordinates: x, y, z.
     233LidarPoint.cpp
     234  Represents a lidar point with coordinates (inherits from Point) and other attributes like time, intensity, classification etc. Note that size of the point directly affects the size and performance of the quadtree, thus only necessary attributes are held inside this class.
     235PointBucket.cpp
     236  A flat data structure which actually holds the points. Each quadtree node holds a pointer to a PointBucket which holds the points in an array of LidarPoints. The PointBucket is also responsible for caching and uncaching of data.
     237PointData.cpp
     238  A struct to hold LAS 1.3+ point attributes which are stored separately.
     239CacheMinder.cpp
     240  A class to keep track of how many points are currently loaded into memory.
     241
     242=== 13. What is the reason behind subbuckets? [=#q13] ===
     243
     244The PointBucket class may hold copies of the same point in several arrays called sub-buckets. There are two parameters which control of how many sub-buckets are created: Resolution Depth and Resolution Base. The Resolution Depth determines the number of sub-buckets for each bucket and Resolution Base determines the number of points at each level by specifying the interval between included points based on a formula of Resolution Base^(Level - 1)^. For example the default LAG values of 5 and 4 create 4 sub-buckets per bucket: one containing every point (5^0^), second containing every 5th point (5^1^), third containing every 25th point (5^2^) and fourth with every 125th point (5^3^). //
     245The reason for this is rendering of points. For example when viewing a whole flightline in the overview window there is no need to render every single point or store all the points in memory at once. Thus sub-buckets containing every n-th point are used instead. Note that you could just pass every-nth point for rendering, but it is really the memory usage and caching issue that sub-buckets solve. //
     246Because each resolution level effectively slowers the quadtree (some points need to be inserted n times into n arrays) for applications other then LAG, which make no use of sub-buckets Resolution Depth and Resolution Base should be set to 1.
     247
     248== 14. How does caching work? [=#q14] ==
     249
     250Upon creation of the quadtree the user specifies a maximum number of points to be held in cache.
     251
     252== 15. What is the reason behind data compression? [=#q15] ==
     253
     254To make caching faster. //
     255PointBucket uses [http://www.oberhumer.com/opensource/lzo/ lzo compression] to compress buckets before writing them to disk. It may seem like a sub-optimal solution but, since quadtree's performance is mainly IO bound, the compression time is still much lower then reading/writing to the hard drive. Since the volume of data gets smaller thanks to compression in the end it speeds up IO operations which are the major bottleneck.
     256
     257== 16. What are LasSaver and LasLoader classes in the quadtree? [=#q16] ==
     258
     259These should be removed at some point together with geoprojectionconverter. They are currently there only for backwards compatibility with other programs that use the quadtree (classify_las).
     260From design point of view these classes should not be a part of the quadtree which should only serve as a data container. In practice, when these classes were used, making any changes in LAG regarding saving or loading files required alterations to the quadtree which was less then convenient. Now that loading and saving has been moved to separate classes inside LAG it is much more maintainable.
     261
     262=== 17. How is LAG's code structured? [=#q17] ===
     263
     264lag.cpp
     265  That's were main()function is. Not much happens here. The program looks for glade ui files, instantiates main classes, then starts Gtk thread where everything runs.
     266BoxOverlay.cpp
     267  This is the box (fence or a profile) that you can draw on the screen.
     268ClassifyWorker.cpp
     269  A worker class responsible for classifying points. This is kind of a stub at the moment, since the methods that do points classification are actually elsewhere and are only called from this class in separate thread.
     270Coulour.cpp
     271  Represents an RGB colour.
     272FileUtils.cpp
     273  An utility class which holds several methods used by classes that deal with files.
     274LagDisplay.cpp
     275  An abstract class responsible for rendering. Profile and TwoDeeOverview inherit from this class.
     276LoadWorker.cpp
     277  A worker class responsible for loading files. This is also the point where the quadtree is being created.
     278MathFuncs.cpp
     279  An utility class with some common math functions.
     280PointFilter.h
     281  A struct to hold a filter string which is then passed to the laslib::LASreadOpener.parse() method to create a filter for the points.
     282Profile.cpp
     283  Represents the profile view of the data. The rendering of the profile and classification of points is done here in a particularly messy way.
     284ProfileWorker.cpp
     285  A worker class that loads points selected in the overview into a profile. This is another stub as the actual methods for loading the profile are currently in Profile.cpp.
     286SaveWorker.cpp
     287  A worker class responsible for saving points to a file.
     288SelectionBox.cpp
     289  Holds the coordinates of a selection made on the screen.
     290TwoDeeOverview.cpp
     291  A mess. Represents the 2d overview of the data and handles its rendering.
     292Worker.h
     293  An abstract worker class.
     294ui classes
     295  These classes represent top-level interface elements (windows and dialogs) and are responsible for connecting the UI to signal handlers.
     296
     297=== 18. How does loading of files work? [=#q18] ===
     298
     299Upon pressing //Add// or //Refresh// button an instance of //LoadWorker// is created in the //FileOpener// class with all the parameters from the file opening dialog (like filenames, ASCII parse string, filters etc) passed to its constructor. Then its //run()// method is called which actually does all the loading and then sends a signal back to the GUI thread through //Glib::Dispatcher// when the loading has finished.
     300Inside the //run()// method if the first file is being loaded a new quadtree object is created with its boundary equal to the values taken from the file's header. Every time a new file is loaded this boundary is adjusted to accommodate new points.
     301Once the quadtree has been set up a //load_points()// method is called which loads points from a file one by one and creates //LidarPoint// to insert them into the quadtree. \\
     302If the file is in latlong projection a GeoProjectionConverter class (which comes from lastools) is used to first adjust scale factors and offsets in the header, and then transform point coordinates. The points stored inside the quadtree are always in UTM projection as it is much easier to handle (in latlong x, y are in degrees and z is in metres which causes some difficulties with rendering).
     303
     304=== 20. How does saving of files work? [=#q20] ===
     305
     306Upon pressing //Save// button an instance of //SaveWorker// is created with all necessary parameters from the save dialog passed to its constructor. Then its //run()// method is called where the saving actually happens. \\
     307Inside //run()// method an array of LidarPoints is created and then a query is run on the entire quadtree to get all the buckets. It then iterates through each bucket and through each point and then inserts each point which belongs to a given flightline into an array. Once an array fills up the points are saved to a file and new points are added from the start of the array until everything has been saved.
     308If the output or the input file is in loatlong projecion two GeoProjectionConverter objects are used to get correct scale factor and offset values in the header and then convert points' coordinates. This is because the points inside the quadtree are in UTM projection and if the input file is in latlong then its scale factors and offsets need to be converted to UTM. At the same time if the output file is in latlong point coordinates need to be converted.
     309
     310=== 21. How is LAS 1.3 point format and waveform data handled? [=#q21] ===
     311
     312The problem with points in LAS 1.3+ files is that they contain a number of additional attributes which are used to describe corresponding waveform data, but which are not needed by lag. These values are all 32 or 64 bit and adding them to the //LidarPoint// class would effectively double its size. In turn they would considerably slower the quadtree and make it occupy additional memory. Therefore all these attributes are stored in a temporary file on the hard drive and then retrieved when points are being saved with help of //LidarPoint::dataindex// variable. (If you have any doubts it is much faster add several double values to //LidarPoint// class and profile some quadtree operations. The performance impact is huge and maybe it would be a good idea at some point to try to store coordinates as scaled integers and have the Quadtree unscale them whenever they're requested. It would make get_x() operations a bit slower but the overall quadtree quite faster.) \\
     313The LoadWorker class containst the following two static members:
     314{{{
     315#!div style="font-size: 100%"
     316  {{{#!c++
     317  static std::tr1::unordered_map<uint8_t, std::string> point_data_paths;
     318  static std::vector<int> point_number;
     319  }}}
     320}}}
     321
     322
     323
     324[#q22 22. How does rendering work?] \\
     325
     326[#q23 23. What are main ideas for further LAG development?] \\
     327[#q24 24. What are some major issues with LAG that need fixing?] \\
     328[#q25 25. What are some additional features that can be added to LAG?] \\
     329[#q26 26. What tools are there to help me with LAG development?] \\
     330