flightline caching and memory optimization

[chrfi]

a feature is desired which allows the user through a pop up box when loading files to be notified that the selection will overflow the available ram and use swap space, they will then be given 3 options, abort, continue, or use hard drive/main memory caching.

abort does what it says on the tin as does continue (which should have a warning message pointing out that your about to have your swap filled and this will make the machine unresponsive and cause serious problems if the machine runs out of swap memory)

the third option will cause the flight lines to be loaded but to be stored on the hard drive rather than in ram. This will provide stable access to full resolution data at the cost of speed.

wish list:
the program will be intelligent about the cashing, loading in the most relevant data to main memory to reduce response time when the user is only viewing a small portion of the data

the program will be multi threaded allowing data to be pulled off the hard drive while the user is still using the gui. for instance while the users is zooming in and out of a small area, a larger area around it is being moved to main memory in anticipation of the user viewing it.

supplemental to this any and all options to further reduce the memory footprint of loaded flight lines need to be looked at

[chrfi]

I think this will mostly fall into the inner workings of the back end data structure so ill take this on.

begun some inital research.

[chrfi]

caching done.

multi threading has been dropped for now, the necessary locks and methods still exist in code but at the moment no clever way to buffer has been better than the basic way of just keeping the most recent buckets in ram.

the caching has been tested on an Ethiopia line and it works albeit very very very slowly

[chrfi]

theres "seems" to be an issue whereby if say 10 large flightlines are loaded at once the program behaves normally for the first 6-8 and then simply stops, network traffic drops off. the program is still running and is still trying to load but nothing happens.

[chrfi]

the cache and memory optimization is currently complete. cacheing is first in first out and the memory footprint has proved generally impossible to shrink.