Data360 Analyze

Hi - am receiving an error below:

Node execution terminated while loading lookup hash by error: Maximum buffer size of 536870912 bytes exceeded. Aborting.

Input1 has 5 records

Input 2 has 1.74M records

Is there a setting that needs to be configured?

I am running the desktop version of dataverse.

Thanks

Martin

My suggestion is to do what I refer to as a "reverse lookup".

Add another lookup in front of the one you have.. switch the inputs on it. put the 1.7m records on top and the 5 records on the bottom. 

The code will be simply:

emit 1:*
where matchIsFound

This way, only the records that could possibly match from pin2 will be fed to your existing lookup.

There is a place where you can change the buffer setting, but you may fare better with trying the trick above.

Martin,

As per the documentation https://preview-desktop.lavastorm.com/#e-node-help/Correlation/lookup.html?Highlight=lookup this node 'Enriches a large data set (left table) with a small data set (right table) by joining the two data sets based on common fields.'

This node is meant for one input that is large and one that is small (e.g. a lookup table).  Given you have two large data sets you will need to use the Join or X-Ref node.

Cheers, Tim.

Thanks Both

i have run both options provided and each works for me.

cheers

Martin