LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: jonathanm

Is there anything special I need to think about when trying to use a Hash Split node, or any limitations as to their use?

I have two tables that I'm trying to join together with an inner join, one with 100,000 rows and the other with 77,000,000. I'm expecting a result set of around 5,000,000 rows give or take.

When I run the join the graph fails with a "too many open files" error and it's been suggested that I use the Hash Split node to generate smaller data sets and then cat the results together after.

Looking at the Training Session 14 example graph on this forum and playing around with some static data the Hash Split seems straight forward enough. However when I run the node I get the following error:

The table being split is an Invoice Detail file and I have the Invoice Number as the Split Field. I have defined a total of seven output nodes, is there anything else I need to do to split this file?

Thanks

Jonathan

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Hey,

This type of error can happen for a number of reasons and is likely not related to the HashSplit being used.

Probably the first thing to check is that the input data to the node that is failing is still available.

This error will appear if you have executed a node, which provides the input to another node. If you then change server farms, and attempt to run the second node, then the input data isn't available to the second node. Similarly, this can happen if the graph state references files which have previously been deleted from the LAE tmp directory.

A quick way to check this is to double click on the output pins of nodes providing input to the failing node.

If that doesn't solve the problem, then can you post the graph and we can work from there. If the graph contains sensitive information, then either PM the BRG, or post it in a private forum.

Tim.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: jonathanm

Hi Tim

Thanks for that, I checked the outputs and you're right the data's no longer there. I'll need to rerun the graph, but it takes around 4.5 hours to extract the 77 million records.

Hopefully the data will hang around a bit longer this time. If not, is there any way to direct the temporary output to my PC rather than the server?

Cheers

Jonathan

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Running a mixed-mode, between a desktop & enterprise server isn't supported.

If you have the Lavastorm Desktop installed, you could run this all on your local machine.

Alternatively, if you want your data to stay around, you should look at changing the way in which you are running the temp file settings under Tools->Preferences->Execution->Temp File.

Note that this will change the settings for all temp files created during your run, so this might end up creating a lot of temp files that you will need to make sure you clean up later.

Another option is to leave your settings as they currently are, but use a "Checkpoint" node - found in the Lavastorm->Logistics category. You can use this to checkpoint your graphs status at various places during the run. Then, you can clear down all of your temp files when you want, and simply run from the Checkpoint node in the graph. You can specify the checkpoint such that the data passed into the checkpoint node gets saved to a location that won't be cleared, and you can just make sure you take care of these files yourself.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: jonathanm

Thanks Tim, I'll give it a try.

All the best

Jonathan