LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: santo_dd

hi,

I Am a new user of Lava-storm.

While using the X-ref function of lavastorm, I came across this error " buffer size cannot exceed 536870912 bytes, Aborting ".

I deleted my previous workings on lavastorm along with some temporary files , but still unable to execute the function.

I would appreciated if u could help me with this problem.

Santosh

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: mermacora

Hi Santosh

You�ll get this error with join nodes when the fields/expression you use to join the 2 inputs is non-unique, and with large inputs. When you join with a non-unique key expression the result will be a Cartesian product of matches � e.g. 200 rows with a certain key value on both inputs will results in 40,000 matching output rows. Due to the way joins are implemented in LAE, the 40,000 rows all reside in a memory buffer which is limited by a configuration property and when this exceeded the node will error. This must be what�s happening with your x-Ref.

There are a few ways to work around this limitation:

1. Remove the duplicates in one ore both of your inputs before you join
2. Alter your join key fields/expression to be more specific/unique � reduce the number of Cartesian product of matches
3. Increase the size of the join memory buffer by altering the ls.brain.node.join.maxBufferSize property (this is set in the ls_brain_node.prop file) - but the buffer size is obviously limited by how much available memory you have on your PC

There was a similar question posted the other day which you should also take a look at:
http://community.mda-data.com/showthread.php?t=110#6

Hope this helps.
Mario

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: raylee

Hi,

We're running into this issue with a number of important graphs and they are failing. It would be difficult to modify them to remove duplicates. We'd like to attempt to boost the buffer size. I tried changing ls.brain.node.join.maxBufferSize and ls.brain.node.lookup.maxBufferSize in ls_brain_node.prob back in version 4.5.2 but the server would not start. Going to attempt again in 4.5.3. The question is, is there a best value to set this at? Our 64bit server has 24GB of RAM.

Ray

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

There really isn't a "best value", it depends on the size RAM in your system, and the other processes that you are running, and other environmental factors. As mentioned above, you really should try removing the duplicates, your graph will likely run much faster and not encounter this issue. However, you can always unset the value and go back to the PRE-4.5 behavior, but your error message is much better with a cap, and your behavior is more deterministic...if you don't have a limit, sometimes you could crash your server, sometimes your graph will run, sometimes it will crash, etc. My recommendation would be to set it fairly high, try a few gig. Or you can set it to *UNDEFINED*, (notice no quotes around it), which will unset the value....you need to make sure there is no value anywhere up the property chain though, so you would need to make sure that ls.brain.node.join.maxBufferSize is not set, but also ls.brain.node.maxBufferSize, ls.brain.maxBufferSize, etc.

Cheers
Rich

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Cindy

Hi, I am running into this error on an X-ref node as well. I have no duplicates in one of my inputs, and really need the result from the join. I think the option for me is to increase the buffer size. I need some instruction on the instruction... I don't know where to find this property file.

"Increase the size of the join memory buffer by altering the ls.brain.node.join.maxBufferSize property (this is set in the ls_brain_node.prop file) - but the buffer size is obviously limited by how much available memory you have on your PC".

Cindy Clark

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

For the Desktop version (on XP), this file is located at C:\Program Files\Lavastorm\LAE 4.5\conf\brain\ls_brain_node.prop
For the Server version.. the file is located at {lavahome}/conf/brain/ls_brain_node.prop - you will have to know where your Lavastorm Home folder is.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ThomasT

My Collegue has a similar problem, but it shouldnt have to do with duplicates..

Could you help?
He tried adjusting the buffer size.

Errors for Join
ERROR processing data:
Maximum buffer size of 900870912 bytes exceeded. Aborting.
2013-10-21 10:20:25.000; WARN: Maximum buffer size of 900870912 bytes exceeded. Aborting.
id: 0 chain: 0 group: 0
cppDetail: context: .\Joiner.cpp@71

2013-10-21 10:20:35.000; ERROR: ERROR processing data:
Maximum buffer size of 900870912 bytes exceeded. Aborting.
id: 0 chain: 0 group: 0
cppDetail: context: Salmon::NED::NodeMain::errorMsg() .\NodeMain.cpp@28

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

I had a similar situation once - for that data set, I was able to use a standard JOIN to find records that matched exactly, which dropped the bulk of the records, then I used the fuzzy match on the remaining orphans.
Doing this also cut a significant amount of processing time - the fuzzy match is not fast on large record sets.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: MichalK

I'm running into the same problem. I'm doing a join between a dataset with 380,000 rows and a single value, so it shouldn't be doing a cartesian product. Also, I've used the same node successfully on a much larger dataset.
Any ideas what can be wrong?

buffersize.jpg