Changes between Version 12 and Version 13 of Processing/lagdevelopersfaq


Ignore:
Timestamp:
Jul 6, 2012 6:12:21 PM (7 years ago)
Author:
jaho
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Processing/lagdevelopersfaq

    v12 v13  
    5959=== 1.4 How can I make LAG's code better? [=#q1.4] ===
    6060
    61 Think first, code later. Take users of your classes into consideration and try to design easy to use and logical interfaces. Do some refactoring if you think it's going to make things easier to maintain, and to understand the code better. Use consistent coding style and comments. \\ \\
    62 Try to learn good coding practices before you start writing the code. It is better to spend some time planning and then writing an implementation, then writing a code that just works and then trying to fix it. For example implementing const correctness from the start is much easier then adding it once everything has been written. \\ \\
     61Take users of your classes into consideration and try to design easy to use and logical interfaces. Do some refactoring if you think it's going to make things easier to maintain, and to understand the code better. Use consistent coding style and comments. Use white space and indentation for readability. \\ \\
     62Try to learn good coding practices before you start writing the code. It is better to spend some time planning and then writing an implementation, then writing a code that just works and then trying to fix it. For example implementing things like const correctness or exception handling from the start is much easier then adding it once everything has been written. \\ \\
    6363Profile your code instead of relying on your intuition of what "should be" faster. Use OO principles for the benefit of more robust code (eg. use polymorphism instead of huge blocks of if statements). Use common sense.
    6464
     
    241241Upon creation of the quadtree the user specifies a maximum number of points to be held in memory. Whenever a new bucket is created inside {{{PointBucket::cache()}}} the {{{CacheMinder::updateCache(int requestSize, PointBucket* pointBucket, bool force)}}} method is called with {{{requestSize}}} parameter equal to the number of points that will be held in that bucket. This number is then added to {{{CacheMinder::cacheUsed}}} variable and compared to {{{CacheMinder::totalCache}}} equal to maximum number of points that can be held in memory. If the sum of the to is smaller then maxim cache size the {{{cachUsed}}} variable is updated and a pointer to the bucket is added to the queue ({{{std::deque}}}) of cached buckets. If total amount of cache requested is greater then allowed maximum the buckets are taken from the front of the queue and uncached until there's enough space for the new bucket. (A note here: the {{{force}}} variable in {{{updateCache()}}} method is actually obsolete as it's always set to true). \\ \\
    242242When a bucket needs to be uncached, because there is no more room for new buckets in memory, {{{PointBucket::uncache()}}} method is called which serializes the bucket by compressing it and writes it to a file. Two lso compression is used for this together with two static variables to reduce the memory overhead. The serialization code looks like this:
     243
    243244{{{
    244245#!div style="font-size: 100%"
     
    257258  }}}
    258259}}}
     260
    259261Once the data has been written, the bucket is deleted from memory and {{{CacheMinder::update_cache()}}} method gets called with negative {{{requestSize}}} to update the amount of used cache. \\ \\
    260 If a {{{setPoint()}}} or {{{getPoint()}}} method is called on an uncached bucked it is simply re-created from the file an {{{CacheMinder::updateCache()}}} is called to reflect these changes.
     262If a {{{setPoint()}}} or {{{getPoint()}}} method is called on an uncached bucked it is simply re-created from the file and {{{CacheMinder::updateCache()}}} is called to reflect this change.
    261263
    262264=== 3.5 What is the reason behind data compression? [=#q3.5] ===
     
    316318The {{{load_xml(const Glib::RefPtr<Gtk::Builder>& builder)}}} method gets widgets from the builder file and assigns them to the class members (pointers to Gtk objects). Widgets instantiated by {{{Gtk::Builder}}} have to be deleted manually. In fact, according to documentation, this only applies to top-level widgets (windows and dialogs), but LAG's authors seem to prefer to delete every single one anyway. \\ \\
    317319The {{{connect_signals()}}} method is responsible for connecting various input signals to class methods. An example lifetime of a Gtk Widget then looks like this:
     320
    318321{{{
    319322#!div style="font-size: 100%"
     
    355358The problem with points in LAS 1.3+ files is that they contain a number of additional attributes which are used to describe corresponding waveform data, but which are not needed by lag. These values are all 32 or 64 bit and adding them to the {{{LidarPoint}}} class would effectively double its size. In turn they would considerably slower the quadtree and make it occupy additional memory. Therefore all these attributes are stored in a temporary file on the hard drive and then retrieved when points are being saved with help of {{{LidarPoint::dataindex}}} variable. (If you have any doubts it is much faster add several double values to {{{LidarPoint}}} class and profile some quadtree operations. The performance impact is huge and maybe it would be a good idea at some point to try to store coordinates as scaled integers and have the Quadtree unscale them whenever they're requested. It would make get_x/y/z() operations a bit slower but the overall quadtree quite faster.) \\ \\
    356359The {{{LoadWorker}}} class containst the following two static members:
     360
    357361{{{
    358362#!div style="font-size: 100%"
     
    362366  }}}
    363367}}}
     368
    364369When 1.3 or higher format is recognised {{{LoadWorker::load_points_wf()}}} method is used which creates {{{LidarPoint}}} objects to be inserted into the quadtree as well as {{{PointData}}} objects which hold the remaining attributes. A temporary file is created for each flightline and {{{PointData}}} objects are serially written to it. The {{{point_data_paths}}} is used to map a flightline number to a name of a temporary file where {{{PointData}}} objects are stored and {{{point_number}}} vector holds the number of points currently loaded per flightline. This number is also assigned to {{{LidarPoint::data_index}}} so the exact location of its data in the temporary file can be retrieved by using {{{LidarPoint::data_index}}} x {{{sizeof(PointData)}}}. Note that the actual waveform data is not touched during the process.\\ \\
    365370When LAS 1.3 points are saved the location of temporary file is retrieved inside {{{SaveWorker::save_points_wf()}}} method from {{{LoadWorker::point_data_paths.find(flightline_number)}}}. Then for each saved point it corresponding {{{PointData}}} is retrieved from the temporary file with something like this:
     371
    366372{{{
    367373#!div style="font-size: 100%"
     
    376382  }}}
    377383}}}
     384
    378385Because laslib library doesn't support storing waveform data in the same file as point records it automatically sets {{{start_of_waveform_data_packet_record}}} field in the header to 0. We need to manually change it to the correct value:
     386
    379387{{{
    380388#!div style="font-size: 100%"
     
    384392  }}}
    385393}}}
     394
    386395Finally we copy over the waveform data manually  and attach it to the end of the output file. Note that this requires for LAS 1.3 files to be saved to a separate file after modification. Unfortunately there is no way around this with how laslib library currently works.
    387396
    388 === 4.6 How does rendering work?] [=#q4.6] ===
     397=== 4.6 How does rendering work? [=#q4.6] ===
    389398
    390399It's difficult to tell. It's quite a mess and it definitely needs some refactoring for maintainability. It does work however so I haven't changed it, focusing on other things. \\ \\
     
    402411=== 5.1 What are main ideas for further LAG development? [=#q5.1] ===
    403412
    404 There are few categories of things that need to be done:
    405   * things that have been started but haven't been quite finished
     413There are few categories of things that need to be done: 
    406414  * issues that need fixing
     415  * code improvements
    407416  * additional features
    408417
    409 To add to that there's still plenty of refactoring to be done (possibly a good thing to start with to understand the code better). Particularly the rendering, which works, but is kind of scattered between different classes and methods and its threading which is just bad. Ideally there should be a separate Renderer/RenderingWorker class with all the code in one place.
    410 
    411 === 5.2 What is the unfinished development stuff? [=#q5.2] ===
    412 
    413 That's mainly sorting out the threading. While LoadWorker and SaveWorker have been fully implemented, the ProfileWorker and ClassifyWorker need some improvement (so the job is actually done inside these classes). To add to that a progress indication for classifying points and loading a profile could be added. \\ \\
    414 
    415 Add more things here.
    416 
    417 === 5.3 What are some major issues with LAG that need fixing? [=#q5.3] ===
    418 
    419 Luckily there's not so many any more, however the major one is the memory consumption. This is quite a difficult problem, but it has to be looked at before lag can be released to the public. \\
    420 Another one is handling of the LAS 1.3 files. \\
     418=== 5.2 What are the major issues with LAG that need fixing? [=#q5.2] ===
     419
     420Luckily there's not so many any more, however the major one is the memory consumption. This is quite a difficult problem, but it should be looked at before lag can be released to the public. \\ \\
     421Another one is handling of the LAS 1.3 files, which works, but can be improved (eg. saving to the same file). This may not be possible though as long as laslib just ignores  full waveform data. \\ \\
    421422Also there is an issue with rendering/loading thread where forcing ON_EXPOSE event while loading a file at the same time may sometimes cause a segfault. For example moving some window on top of the TwoDeeOverview window while file is being loaded may cause this. This is most likely cause by both loading and rendering thread trying to access the quadtree at the same time and I believe sorting out the rendering thread should solve this problem.
    422423
    423 === 5.4 What is lag memory issue? [=#q5.4] ===
     424
     425=== 5.3 What is lag memory issue? [=#q5.3] ===
    424426
    425427It's an unknown source of high memory consumption of LAG. It looks like either a memory leak, memory fragmentation or both. The reason to suspect memory fragmentation is that the memory usage keeps growing to some point then stays constant. It's easy to see when loading the same file multiple times. Also when using a heap profiler it will show correct and constant memory usage and will not grow as high, possibly because of custom memory pool from the profiler.
    426428
    427429Probably the best way to fix it would be to create a [http://en.wikipedia.org/wiki/Allocator_(C%2B%2B)#Custom_allocators custom memory allocator] which is an interesting project on its own.
     430
     431=== 5.4 Which parts of LAG's code need improvement? [=#q5.4] ===
     432
     433All that I haven't written. \\
     434On a serious note that's mainly sorting out the threading. While {{{LoadWorker}}} and {{{SaveWorker}}} have been fully implemented, the {{{ProfileWorker}}} and {{{ClassifyWorker}}} need some improvement (so the work is actually done inside these classes). To add to that a progress indication for classifying points and loading a profile could be added. \\ \\
     435{{{PointBucket}}} and {{{CacheMinder}}} may use some work, possibly implementing a custom memory allocator for better memory management. \\ \\
     436There's still plenty of refactoring to be done (possibly a good thing to start with to understand the code better). Particularly the rendering, which works, but is kind of scattered between different classes and methods and its threading which is just bad. Ideally there should be a separate Renderer/RenderingWorker class with all the code in one place. \\ \\
     437Finally there is always some room for optimisations.
    428438
    429439=== 5.5 What are some additional features that can be added to LAG? [=#q5.5] ===