Changes between Version 9 and Version 10 of Processing/lagdevelopersfaq


Ignore:
Timestamp:
Jul 6, 2012 4:13:40 PM (7 years ago)
Author:
jaho
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Processing/lagdevelopersfaq

    v9 v10  
    8585
    8686LAS is a binary, public file format designed to hold LiDAR points data. One alternative to LAS are ASCII files(.csv or .txt), which are however less efficient in terms of both processing time and file size. \\ \\
    87 
    8887LAS files consist of several parts: Public Header Block, Variable Length Records, Point Records and in format 1.3 and later Waveform Records. \\ \\
    89 
    9088One thing to note about LAS is that point coordinates are stored as scaled integers and are then unscaled upon loading to double values with the use of scale factors and offsets stored in the header. \\ \\
    91 
    9289A detailed LAS format specification can be found at:
    9390[http://www.asprs.org/Committee-General/LASer-LAS-File-Format-Exchange-Activities.html http://www.asprs.org/Committee-General/LASer-LAS-File-Format-Exchange-Activities.html]
     
    168165
    169166It seems that laslib is mainly developed for Windows users so there are no targets for shared libraries in the Makefiles by default. At the same time shared libraries are needed by LAG to work correctly. To fix this you're going to need to modify the Makefiles and add {{{-fPIC}}} option to the compiler. You'll also have to change the name of the library from laslib to //liblaslib// for the linker to detect it. \\ \\
    170 
    171167You'll normally find modified Makefiles somewhere around, so you can copy them after downloading a new version of laslib and hopefully they will work. If this is not the case, below is the part that needs to be added to the Makefile inside laslib/src folder.
    172168
     
    212208
    213209The Lidar Quadtree library provides a [http://en.wikipedia.org/wiki/Quadtree quadtree] data structure for storing and indexing points. It also contains a caching mechanism for loading and unloading points from memory. \\ \\
    214 
    215210Each node of the quadtree represents a bounding box in x and y coordinates. Whenever a point is inserted into the quadtree it is determined which of the four nodes it falls into and then the same happens for each node recursively until a leaf node is found. Whenever the number of points in a node goes above the maximum capacity specified on creation of the quadtree, the node splits into four. \\ \\
    216 
    217211When retrieving points a bounding box is passed to the quadtree and all nodes which intersect with it are queried for the points. It's quite simple really.
    218212
     
    237231
    238232The {{{PointBucket}}} class may hold copies of the same point in several arrays called sub-buckets. There are two parameters which control of how many sub-buckets are created: //Resolution Depth// and //Resolution Base//. The //Resolution Depth// determines the number of sub-buckets for each bucket and //Resolution Base// determines the number of points at each level by specifying the interval between included points based on a formula of Resolution Base^(Level - 1)^. For example the default LAG values of 5 and 4 create 4 sub-buckets per bucket: one containing every point (5^0^), second containing every 5th point (5^1^), third containing every 25th point (5^2^) and fourth with every 125th point (5^3^). \\ \\
    239 
    240233The reason for this is rendering of points. For example when viewing a whole flightline in the overview window there is no need to render every single point or store all the points in memory at once. Thus sub-buckets containing every n-th point are used instead. Note that you could just pass every-nth point for rendering, but it is really the memory usage and caching issue that sub-buckets solve. \\ \\
    241 
    242234Because each resolution level effectively slowers the quadtree (some points need to be inserted n times into n arrays) for applications other then LAG, which make no use of sub-buckets, //Resolution Depth// and //Resolution Base// should be set to 1.
    243235
     
    252244
    253245To make caching faster. \\ \\
    254 
    255246PointBucket uses [http://www.oberhumer.com/opensource/lzo/ lzo compression] to compress buckets before writing them to disk. It may seem like a sub-optimal solution but, since quadtree's performance is mainly IO bound, the compression time is still much lower then reading/writing to the hard drive. Since the volume of data gets smaller thanks to compression in the end it speeds up IO operations which are the major bottleneck.
    256247
     
    258249
    259250These should be removed at some point together with {{{geoprojectionconverter}}}. They are currently there only for backwards compatibility with other programs that use the quadtree (classify_las). \\ \\
    260 
    261251From design point of view these classes should not be a part of the quadtree which should only serve as a data container. In practice, when these classes were used, making any changes in LAG regarding saving or loading files required alterations to the quadtree which was less then convenient. Now that loading and saving has been moved to separate classes inside LAG it is much more maintainable. Additionally it allows individual point inspection during loading and saving, which was not possiblebe before.
    262252
     
    268258
    269259lag.cpp
    270   That's were main()function is. Not much happens here. The program looks for glade ui files, instantiates main classes, then starts Gtk thread where everything runs.
     260  That's were {{{main()}}} function is. Not much happens here. The program looks for glade ui files, instantiates main classes, then starts Gtk thread where everything runs.
    271261BoxOverlay.cpp
    272262  This is the box (fence or a profile) that you can draw on the screen.
     
    278268  An utility class which holds several methods used by classes that deal with files.
    279269LagDisplay.cpp
    280   An abstract class responsible for rendering. Profile and TwoDeeOverview inherit from this class.
     270  An abstract class responsible for rendering. {{{Profile}}} and {{{TwoDeeOverview}}} inherit from this class.
    281271LoadWorker.cpp
    282272  A worker class responsible for loading files. This is also the point where the quadtree is being created.
     
    284274  An utility class with some common math functions.
    285275PointFilter.h
    286   A struct to hold a filter string which is then passed to the laslib::LASreadOpener.parse() method to create a filter for the points.
     276  A struct to hold a filter string which is then passed to the {{{laslib::LASreadOpener.parse()}}} method to create a filter for the points.
    287277Profile.cpp
    288278  Represents the profile view of the data. The rendering of the profile and classification of points is done here in a particularly messy way.
     
    302292=== 4.2 How do I edit the GUI? [=#q4.2] ===
    303293
    304 The GUI layout is stored in //lag.ui// [http://glade.gnome.org/ Glade] file which is in plain xml. This allows editing the interface without the need for recompiling the source code every time a change is made. The //lag.ui// file can be opened in Glade User Interface Designer which can by run with //glade-3// command.
    305 
    306 The interface of the application is created from xml using Gtk::Builder object, which is instantiated in lag.cpp and then passed by Glib::RefPtr to each of the classes that handle the GUI. All of these classes are located under src/ui folder and each of them has two common methods: //void load_xml(builder)// and //void connect_signals()// which are called in their constructors.
    307 
    308 The //load_xml(const Glib::RefPtr<Gtk::Builder>& builder)// method gets widgets from the builder file and assigns them to the class members (pointers). Widgets instantiated by Gtk::Builder have to be deleted manually. In fact, according to documentation, this only applies to top-level widgets (windows and dialogs), but LAG's authors seem to prefer to delete every single one anyway.
    309 
    310 The //connect_signals()// method is responsible for connecting various input signals to class methods. An example lifetime of a Gtk Widget then looks like this:
    311 
     294The GUI layout is stored in {{{lag.ui}}} [http://glade.gnome.org/ Glade] file which is in plain xml. This allows editing the interface without the need for recompiling the source code every time a change is made. The {{{lag.ui}}} file can be opened in Glade User Interface Designer which can by run with //glade-3// command. \\ \\
     295The interface of the application is created from xml using {{{Gtk::Builder}}} object, which is instantiated in lag.cpp and then passed by {{{Glib::RefPtr}}} to each of the classes that handle the GUI. All of these classes are located under {{{src/ui}}} folder and each of them has two common methods: {{{void load_xml(builder)}}} and {{{void connect_signals()}}} which are called in their constructors. \\ \\
     296The {{{load_xml(const Glib::RefPtr<Gtk::Builder>& builder)}}} method gets widgets from the builder file and assigns them to the class members (pointers to Gtk objects). Widgets instantiated by {{{Gtk::Builder}}} have to be deleted manually. In fact, according to documentation, this only applies to top-level widgets (windows and dialogs), but LAG's authors seem to prefer to delete every single one anyway. \\ \\
     297The {{{connect_signals()}}} method is responsible for connecting various input signals to class methods. An example lifetime of a Gtk Widget then looks like this:
    312298{{{
    313299#!div style="font-size: 100%"
     
    334320=== 4.3 How does loading of files work? [=#q4.3] ===
    335321
    336 Upon pressing //Add// or //Refresh// button an instance of //LoadWorker// is created in the //FileOpener// class with all the parameters from the file opening dialog (like filenames, ASCII parse string, filters etc) passed to its constructor. Then its //run()// method is called which actually does all the loading and then sends a signal back to the GUI thread through //Glib::Dispatcher// when the loading has finished.
    337 Inside the //run()// method if the first file is being loaded a new quadtree object is created with its boundary equal to the values taken from the file's header. Every time a new file is loaded this boundary is adjusted to accommodate new points.
    338 Once the quadtree has been set up a //load_points()// method is called which loads points from a file one by one and creates //LidarPoint// to insert them into the quadtree. \\
    339 If the file is in latlong projection a GeoProjectionConverter class (which comes from lastools) is used to first adjust scale factors and offsets in the header, and then transform point coordinates. The points stored inside the quadtree are always in UTM projection as it is much easier to handle (in latlong x, y are in degrees and z is in metres which causes some difficulties with rendering).
     322Upon pressing //Add// or //Refresh// button an instance of {{{LoadWorker}}} is created in the {{{FileOpener}}} class with all the parameters from the file opening dialog (like filenames, ASCII parse string, filters etc) passed to its constructor. Then its {{{run()}}} method is called which actually does all the loading and then sends a signal back to the GUI thread through {{{Glib::Dispatcher}}} when the loading has finished. \\ \\
     323Inside the {{{run()}}} method if the first file is being loaded a new quadtree object is created with its boundary equal to the values taken from the file's header. Every time a new file is loaded this boundary is adjusted to accommodate new points. \\ \\
     324Once the quadtree has been set up a {{{load_points()}}} method is called which loads points from a file one by one and creates {{{LidarPoint}}} objects to insert them into the quadtree. \\ \\
     325If the file is in latlong projection a {{{GeoProjectionConverter}}} (which comes from lastools) is used to first adjust scale factors and offsets in the header, and then transform point coordinates. The points stored inside the quadtree are always in UTM projection as it is much easier to handle (in latlong x, y are in degrees and z is in metres which causes some difficulties with rendering).
    340326
    341327=== 4.4 How does saving of files work? [=#q4.4] ===
    342328
    343 Upon pressing //Save// button an instance of //SaveWorker// is created with all necessary parameters from the save dialog passed to its constructor. Then its //run()// method is called where the saving actually happens. \\
    344 Inside //run()// method an array of LidarPoints is created and then a query is run on the entire quadtree to get all the buckets. It then iterates through each bucket and through each point and then inserts each point which belongs to a given flightline into an array. Once an array fills up the points are saved to a file and new points are added from the start of the array until everything has been saved.
    345 If the output or the input file is in loatlong projecion two GeoProjectionConverter objects are used to get correct scale factor and offset values in the header and then convert points' coordinates. This is because the points inside the quadtree are in UTM projection and if the input file is in latlong then its scale factors and offsets need to be converted to UTM. At the same time if the output file is in latlong point coordinates need to be converted.
     329Upon pressing //Save// button an instance of {{{SaveWorker}}} is created with all necessary parameters from the save dialog passed to its constructor. Then its {{{run()}}} method is called where the saving actually happens. \\ \\
     330Inside {{{run()}}} method an array of {{{LidarPoint}}} objects is created and then a query is run on the entire quadtree to get all the buckets. It then iterates through each bucket and through each point and then inserts each point which belongs to a given flightline into an array. Once an array fills up the points are saved to a file and new points are added from the start of the array until everything has been saved. \\ \\
     331If the output or the input file is in loatlong projecion two {{{GeoProjectionConverter}}} objects are used to get correct scale factor and offset values in the header and then convert points' coordinates. This is because the points inside the quadtree are in UTM projection and if the input file is in latlong then its scale factors and offsets need to be converted to UTM. At the same time if the output file is in latlong point coordinates need to be converted with a different transformation.
    346332
    347333=== 4.5 How is LAS 1.3 point format and waveform data handled? [=#q4.5] ===
    348334
    349 The problem with points in LAS 1.3+ files is that they contain a number of additional attributes which are used to describe corresponding waveform data, but which are not needed by lag. These values are all 32 or 64 bit and adding them to the //LidarPoint// class would effectively double its size. In turn they would considerably slower the quadtree and make it occupy additional memory. Therefore all these attributes are stored in a temporary file on the hard drive and then retrieved when points are being saved with help of //LidarPoint::dataindex// variable. (If you have any doubts it is much faster add several double values to //LidarPoint// class and profile some quadtree operations. The performance impact is huge and maybe it would be a good idea at some point to try to store coordinates as scaled integers and have the Quadtree unscale them whenever they're requested. It would make get_x() operations a bit slower but the overall quadtree quite faster.) \\
    350 The LoadWorker class containst the following two static members:
     335The problem with points in LAS 1.3+ files is that they contain a number of additional attributes which are used to describe corresponding waveform data, but which are not needed by lag. These values are all 32 or 64 bit and adding them to the {{{LidarPoint}}} class would effectively double its size. In turn they would considerably slower the quadtree and make it occupy additional memory. Therefore all these attributes are stored in a temporary file on the hard drive and then retrieved when points are being saved with help of {{{LidarPoint::dataindex}}} variable. (If you have any doubts it is much faster add several double values to {{{LidarPoint}}} class and profile some quadtree operations. The performance impact is huge and maybe it would be a good idea at some point to try to store coordinates as scaled integers and have the Quadtree unscale them whenever they're requested. It would make get_x/y/z() operations a bit slower but the overall quadtree quite faster.) \\ \\
     336The {{{LoadWorker}}} class containst the following two static members:
    351337{{{
    352338#!div style="font-size: 100%"
     
    356342  }}}
    357343}}}
    358 When 1.3 or higher format is recognised //LoadWorker::load_points_wf()// method is used which creates //LidarPoint// objects to be inserted into the quadtree as well as //PointData// objects which hold the remaining attributes. A temporary file is created for each flightline and //PointData// objects are serially written to it. The //point_data_paths// is used to map a flightline number to a name of a temporary file where //PointData// objects are stored and //point_number// vector holds the number of points currently loaded per flightline. This number is also assigned to //LidarPoint::data_index// so the exact location of its data in the temporary file can be retrieved by using //LidarPoint::data_index// x //sizeof(PointData)//. Not that the actual waveform data is not touched during the process.\\
    359 When LAS 1.3 points are saved the location of temporary file is retrieved inside {{{SaveWorker::save_points_wf()}}} method from {{{LoadWorker::point_data_paths.find(flightline_number)}}}. Then for each saved point it corresponding {{{PointData}}} is retrieved from the temporary file with like this:
     344When 1.3 or higher format is recognised {{{LoadWorker::load_points_wf()}}} method is used which creates {{{LidarPoint}}} objects to be inserted into the quadtree as well as {{{PointData}}} objects which hold the remaining attributes. A temporary file is created for each flightline and {{{PointData}}} objects are serially written to it. The {{{point_data_paths}}} is used to map a flightline number to a name of a temporary file where {{{PointData}}} objects are stored and {{{point_number}}} vector holds the number of points currently loaded per flightline. This number is also assigned to {{{LidarPoint::data_index}}} so the exact location of its data in the temporary file can be retrieved by using {{{LidarPoint::data_index}}} x {{{sizeof(PointData)}}}. Note that the actual waveform data is not touched during the process.\\ \\
     345When LAS 1.3 points are saved the location of temporary file is retrieved inside {{{SaveWorker::save_points_wf()}}} method from {{{LoadWorker::point_data_paths.find(flightline_number)}}}. Then for each saved point it corresponding {{{PointData}}} is retrieved from the temporary file with something like this:
    360346{{{
    361347#!div style="font-size: 100%"
     
    386372Once the files have been loaded, a pointer to the quadtree is passed to {{{LagDisplay}}}, {{{TwoDeeOverview}}} and {{{Profile}}}. First {{{LagDisplay::prepare_image()}}} method is called which sets up OpenGl and gets all the buckets from the quadtree with {{{advSubset()}}} method. Based on the ranges of z and intensity values it then sets up arrays for coloring and brightness of the points in {{{LagDisplay::coloursandshades()}}}. Then {{{resetview()}}} method is called which sets up orthographic projection and conversion of world coordinates to screen coordinates. \\ \\
    387373After that a {{{drawviewable()}}} method is called in {{{TwoDeeOverview}}} and/or {{{Profile}}} which checks what part of the data is currently being viewed on the screen and gets the relevant buckets from the quadtree. It also uses a good number of boolean variables which are changed all over the place to make things harder to understand.If you were wondering how NOT to do threading in C++, here's a great example. Eventually we create a new thread and call {{{TwoDeeOverview::mainimage()}}} method passing it the buckets we've got. \\ \\
    388 
    389374In {{{mainimage()}}} we call {{{drawpointsfrombuckets()}}} which prepares the points for being drawn. For each bucket from the quadtree it determines the resolution level needed (sub-bucket) based on the current zoom and viewing area. Then for each point in that bucket it creates vertices with attributes of the point to be drawn (location, color, brightness) and passes them to {{{DrawGLToCard()}}} method in the main thread which draws the contents of the vertices to the framebuffer. Finally {{{FlushGLToScreen()}}} method draws the contens of framebuffer to the screen. \\ \\
    390 
    391375Simple as that.
    392376
     377
    393378== 5. Further Development [=#q5] ==
     379----
    394380
    395381=== 5.1 What are main ideas for further LAG development? [=#q5.1] ===