LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: szegit

Hi!

I'm using a graph, which contains a Hash Split node, to split the data into 10 parts.
After a few hours of using the graph the data loading start to slowing at this node (at beginning the graph runs in 4-6 mins, but after 4-5 hours of loading it produces 20-25 mins runs). Using F1 Help I found, that Hash Split node shall split data to odd parts and if it is possible it shall be a prime number.

I wanna ask if the slowing is because of that I split to 10 parts my dataset or it is rather some issue with a buffer of LEA?

Thx for help!

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: xathras

Are you using Server Edition or Desktop. At gogo we have the Server edition and put through more than a few million records and it processes pretty quickly < few mins

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: szegit

Server version. We're loading more reports, but through 1 run is only worked 10 and it is organized with a hash split node. If the run is over, we start it again (next 10) till we have anything on input...
So it is not big amount of data, just after few hours of loading, the process starts to slow down.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

This seems very odd, and not something I have heard before. I would reach out to your support contact, as I suspect you might have something environmental going on. That node is fairly straightforward and generally one of our fastest nodes.

Cheers
Rich

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: szegit

so... i think we have a bit misunderstanding here
i just started examine a graph, what was made by some of my colleagues and i just try to figure out how it works and give the graph equalized form (some processes are made there different way and i have to optimalize them for faster run and throw out things what we do not need).
when i examined the hash split node, i just read the definition using F1 help and it says: "Takes one input, and either a list of field names or a BRAINscript expression. Splits the input into any number of outputs, based on hashing the values in the specified fields or expression. For optimal performance, the number of outputs should be odd, preferrably a prime number." And i just wanna know why shall i have on output odd and prime number...
Approximately the slowing is not caused by hash split node, but it is caused by something what is after the hash split (there are 10 composites which are very complex)

thx
T

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

Ah, that makes sense. The help suggests that for a good equal "spray" of records. If the output pins of the hash split have a good enough split on each of the 10 (you don't have many more records in one pin, and much fewer on another), than you are fine. If you are getting a significantly uneven distribution, I would only then suggest to change the number of pins, otherwise I wouldn't worry about it.

Cheers
Rich

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: szegit

thx for answer, i'll see how the graph is going to be trnasformed and then maybe the slowing will disappear..


T