LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: adjrock

Hi all,
I load a few million of rows at a time in a graph and it takes me a long time. I understand that, but when I close Lavastorm and open it later, the data is still loaded or I have to reload the whole graph. Does anyone have an idea how I can make sure that my data remains and avoid doing reloads which costs a lot of time.

Thanks

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mmarinelli

Here's some material on the subject of temporary file cleanup which should help you:
--
As graphs are executed by the LAE server, the temporary files created by each node are maintained using one of three schemes: �Normal�, �Aggressive�, or �Lazy�, each of which is described below:

� Normal
This cleanup scheme, which is the default and typical setting for interactive BRE sessions, will retain temporary files until one of the following actions occurs:
1. Node status is cleared, either by manual user action or by the clearing of an ancestor node�s status
2. Node is deleted and �Delete Temp Files on Node Deletion� preference is set in the BRE client software
3. The underlying temporary file is deleted using the disk space cleanup utility available through the BRE Tools->Server->Disk Usage... menu
4. The underlying temporary file is deleted by some other mechanism outside of the LAE software

� Aggressive
This cleanup scheme, which should be employed for production execution of graphs, will only retain temporary files so long as they are needed by downstream nodes. Once a downstream node execution has been completed successfully, all data for a parent node will be deleted. Using this scheme, a graph only requires enough temporary file space to store a fraction of the temporary files which would be retained using the �Normal� scheme, since data is only retained for the current working set of nodes. However, since temporary files are deleted during processing, the user no longer has the ability to visit an ancestor node and view that node�s data.

� Lazy
This mode is similar to �Normal�, but does not delete temporary files when ancestor nodes have their status cleared. Instead, temporary files will be retained for a node until the node is itself re-executed, at which time the previous data will be overwritten with refreshed results. This scheme requires the most disk space, and is not recommended unless this functionality is necessary.

The desired scheme can be set by the BRE client (Tools->Preferences->Execution) for interactive sessions or by using the controller properties for production run execution on the server. For the latter case, �Normal� mode is active if ls.brain.controller.cleanInputs=false and �Aggressive� mode is active if ls.brain.controller.cleanInputs=true. �Lazy� mode is not supported by the controller.
--

So in your case, you may have changed the file cleanup scheme to one which is deleting your temporary files, or the files may be being deleted by some external program. If you use the "Normal" setting and no other process is cleaning files out of your temporary space, you should still have your interim data sets between sessions.