LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tris_717684

Hello LAE team!

What can I do to resolve attached error occurring at a Lookup node? I've got LAE 6.1.4 Professional Plus Edition set up in my computer

Situation:

Input key is from a Filter node which has 7,024,637 records across 57 columns

Lookup key is from a Filter node which has 1,362 records across 8 columns

Hope to hear from the team soonest.Lookup node error.jpg

Regards,

tris

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: gmullin

If you open up <LAE Install Dir>/conf/brain/ls_brain_node.prop and find the parameter called ls.brain.node.lookup.maxBufferSize, you can increase the value its equal to. Try 1073741824 (double) for example.

Or if you don't want to edit the file you can also set a parameter in the Lookup node by pressing "Declare Parameters" button and entering a parameter like the screenshot. And set the value within the node.
Attachments:
MaxBufferSize.jpg

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tris_717684

thanks so much for the very quick turnaround @gmullin! much appreciated!

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

Is one or more of those 8 columns exceptionally long? 1300 records should not have caused that problem.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tris_717684

hello @rmullin and @stonysmith!

1. I successfully created the MaxBufferSize parameter within the lookup node; unfortunately, it still shows the same error.
2. I tried to change the Is_brain_node.prop but shows "Access denied" window when I tried to save the recommended change.
3. The contents of the two files I'm trying to do a lookup are not exceptionally long. Just to troubleshoot, I tried to use a sample size of 200 records for the LookupKey and the node worked. Is 7MM records x 57 cols. too much for the node to handle?

Any other option I can do?

Thanks heaps team!

Regards,

tris

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

Originally posted by: Tris_717684
					

Is 7MM records x 57 cols. too much for the node to handle?
			

The size of the input file (pin1) does not matter in any way.
The total byte size of the lookup file (pin2) can not exceed the buffer size.

If there are columns in the lookup (pin2) that you are not using (neither in the lookup key nor sent to the output) then you could remove those columns with a filter node, thus reducing the memory required by the lookup.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tris_717684

Thanks @stonysmith! will play around with it.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tris_717684

Further to @gmullin's suggestion, I have attempted many times to edit and save and always get the "Access denied to edit the Is_brain_node.prop file.". What do i use to edit and save the prop file?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: gmullin

Are you able to open Notepad as Admin and then try to open the ls_brain_node.prop file? Or can you edit the conf/site.prop file, you can add the same parameter there too and it will overwrite the value.

I don't see why creating the parameter in the Lookup node itself wouldn't work, what value did you set the parameter to? Maybe try increasing the value?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tris_717684

I am able to open and edit the *.prop file using Wordpad or Notepad without any problems. I have replicated the original *.prop file and renamed it to show ls_brain_node (old).prop as a fall back (and this confirms that I have admin access to this folder). However, on saving the edited *.prop file, an error window popped up showing that my access is denied. I cannot save it as a *.txt file as well and it's showing the same error.

I have increased it 1,500,000,000 (nearly tripled) and the declared parameter still doesn't work.

I have reduced the number of columns to 27 from 57 (but same number of rows)... and the lookup node still fails.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: gmullin

With the parameter in the node you still get an error saying "Maximum buffer size of 536870912 bytes exceeded"? If so, then it would look like it's not picking up the value of the parameter you created? Can you show screenshots of the parameter declaration?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: khopkins

The easiest solution to this is to use an xref instead of a lookup node as the way the lookup works fills up the available buffer space in memory especially with over 7 million records