LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: MarkCunningham

Hi

I'm Currently trying to Run an X-Ref node in my Graph

the lookup input(Lower) has 256 rows and the Top Input has 1.3 million.

this x-ref is failing and i get an error of
-- Errors for Join
ERROR processing data:
Maximum buffer size of 536870912 bytes exceeded. Aborting.

Installing log (10/28/2011 17:39:36) ...
2011-10-28 17:40:14; Level: 2; Type: 0; Desc: "Maximum buffer size of 536870912 bytes exceeded. Aborting."; File: ".\Joiner.cpp"; Line: 71; Context: ""
Uninstalling log (10/28/2011 17:40:18) ...

but when the same data is put through a different Xref with over 2.3million rows it works fine with no issue with a slightly different match key ..
i have made sure that the Bottom input has no duplicate data so this does not create any additionl rows.

i don't know what is wrong with it can you help at all?

regards
Mark Cunningham
CPW Plc.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: timonk

Mark,

The Xref an Join nodes build their lists of cartesian products in memory. The reason for the buffer error is that the size of the cross product in yoru xref exceeds the size of the memory buffer setting. This parameter and setting was added to LAE to prevent users from completely eating up the memory on their system, and to allow them control.

The important detail here isn't the size of the inputs, its the size of the resulting cartesian cross product your xref is producing while matching data. (e.g. if you have 3 records at both inputs, and each is a match for the other, you end up with 9 records at output. That list is built up in memory)

You note a "slightly different match key" in your working Xref. That difference in matching keys is resulting in smaller matched lists in memory, and so the data passes.

You should review how you are matching your data. Do you have a lot of duplicate values for matching? Is your match key too general?

You can increase the buffer parameter and see if you are just slightly larger than the current setting. It is set in the ls_brain_node.prop file in your installation. The "ls.brain.node.join.maxBufferSize".

Regards,
Timon Koufopoulos
MDS Lavastorm Support.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: MarkCunningham

Hi Tim

Thanks but I'm not understanding this and all I have done is split the data out based on a value then placed the outputs into two different X-refs.

First node is looking at 1.3 million and is failing
Second looks at 2.3 million and works ..

both using the same look up data containing no duplicates with a value of 260.

why would the grater work and the smaller Xref fail?

Thanks
Mark

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: MarkCunningham

Oh and as in addtional note

i would only expect 1.3 Million out as there are no duplicates to create more than 1 output as in the possable issue you explained above.

Thanks Mark

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

Mark, first, to get you going, it sounds like you probably want to use a Lookup node, which will load the second input into memory, and the first input does not need to be sorted. This works if there is no duplicate values on the second input, which yours does not have.

What is going on here with the X-Ref, is as Timon described above. The full set of matches is loaded into memory on the X-Ref. This means that even though you have 260 unique values in the second input, if say, one of the 260 values matched half your records on the first input, they would all be loaded into memory...so if you are changing your match criteria, the smaller set of data might still have issues because more of that set is matching a single row of the 260 values. This is a known design limitation, and will be addressed via some mechanism in a future release.

Cheers
Rich

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: MarkCunningham

Thanks Rich

i'll try a different approch on the data .

regards
Mark