LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dglavastorm

Hi Lavastorm - Is there any way to 'Lock" a node so that it remains an active part of the graph but doesn't clear if I highlight the composite and hit Re-Run or Clear Status?

The scenario:

A large graph with many inputs, one of which calculates to approx 10 million rows - this is historical data up to a certain date and won't ever be recalculated but the data is required for each new daily run.

Currently every day I have to individually clear and re-run the nodes around this meganode to ensure it doesn't clear and attempt to recalculate (takes about an hour). Disabling the node doesn't prevent it from clearing if I highlight Composite and hit Re-Run, and of course disabling prevents the rest of the graph from running.

Exporting to Excel or CSV doesn't help because the file ends up being gigabytes big. The processing time to re-import this data daily defeats the purpose of trying to not re-calculate it.

Cheers,

Dave

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ThomasT

You should have a look at check out the node "Checkpoint" in Logistics category.
Is this what you are looking for?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

If you have data that should essentially be considered static, I would not leave it sitting in the TEMP folders. I would make sure that it is saved somewhere other than the temp area.

The Checkpoint node is a good way to handle this.

But.. OPEN the checkpoint node and on the BRDFile but set the 'NoWrite' option to TRUE.
That should help make bringing it back in faster.

If you select NoWrite=False (the default) then the BRDFile node copies the input to a new temporary file, which can take a good amount of time. (and disk space)

If you select NoWrite=True, then the node simply creates a link to the original file and immediately starts reading from it.

Depending upon whether you need the record counts from this one node or not, you can also speed it up another step by specifying NoRecordCount=True which tells the node not to go read the data just to get a record count.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dglavastorm

Thanks - the Checkpoint instructions/helpfiles are really not too clear to me so some clarification would be appreciated on these 4 questions

1 The nodes purpose is simply to publish data to a BRD file just like the Publish node, but with the added function of passing the data along ?

2 If so, can it also export as CSV or Excel ?

3 Is there a benefit to using BRD instead of one of these common formats ? The export file is still 4GB like the CSV, but if the import time is negligible then I can use this.

4 Am I also right in saying that the first time you run it you must set "UseCheckpoint" to False so it writes a file, then change back to True so it changes this to an import node ?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hey,

1 The nodes purpose is simply to publish data to a BRD file just like the Publish node, but with the added function of passing the data along ?

Essentially, yes.
As an example, it can be useful to configure your graph such that you have a few graph level parameters to specify things like "Checkpoint1" "Checkpoint2" or something similar.
You could then place all nodes before your first Checkpoint node into a composite, and set that composite to be disabled if the Checkpoint1 was set to true.
You could then have a Checkpoint node, which has "UseCheckpoint" set to {{^Checkpoint1^}}.
Then, all nodes after that checkpoint node, prior to the next checkpoint, could be set to have their Enabled flag set to (not {{^Checkpoint2^}}).
The second Checkpoint node, would then have "UseCheckpoint" set to "{{^Checkpoint2^}} and so on.
Using this mechanism, you wouldn't need to re-run all of the nodes in your graph, but just make checkpoints from which you could then resume execution.

2 If so, can it also export as CSV or Excel ? 

The checkpoint node only works by producing and consuming BRD files.

3 Is there a benefit to using BRD instead of one of these common formats ? The export file is still 4GB like the CSV, but if the import time is negligible then I can use this.

Yes.
Since BRD is the internal format used to pass data between nodes, if you set "NoWrite" and "NoRecordCount" to true, then the data should just be imported (though I believe this depends on the file system and whether or not symbolic links are allowed) by specifying that the next node downstream will work with a reference to the original checkpoint file.

4 Am I also right in saying that the first time you run it you must set "UseCheckpoint" to False so it writes a file, then change back to True so it changes this to an import node ?

Correct.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dglavastorm

Thanks Tim - Clear on points 2-4 but a bit lost one 1. Do you have an example you could post ?