LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: xathras

Hi,

Has anyone developed, or got an alternative method of sorting data in LAE quicker than the standard Sort Node and then joining date?
At the moment I have a data set of 7,000,000 reference points to compare against a data set of 10,000 records.

This takes some time to do. Approximately, 10 minutes.

Is it possible to implement something in python or java to do this? if so, has anyone done it and willing to share?

Thanks in advance

Wayne

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

If the 10,000 records does not have a lot of columns, then use a LOOKUP node.. it'll run much quicker.

Lookup does not require sorting, but it requires the entire 2nd input to be read into memory, so it has an upper limit of how much you can process.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: xathras

Hi,

I ran a lookup just now after your recommendation and my sort data hits the maximum buffer. Thanks for the suggestion. Any alternatives?

Regards
Wayne

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: xathras

Bumping this up. Does anyone have any suggestions?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

Um....is there anything you can do to shrink the size of the rows? 10K is not THAT many records to hold in memory. You could also increase the buffer size, if you are running on a machine that can handle it. You need to create a parameter on the lookup node and on the parameter declaration dialog, set the Run Time Property Name to ls.brain.node.lookup.maxBufferSize. Then go back to the node editor, and set that to a big number (it's the number of bytes that will load into memory in the lookup node). That should sort you out.

Cheers
Rich

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: xathras

Hi Rich,

I tried doing that method but found increasing the buffer size worked but the runs varied and didn't really reduce the run time that I was looking for.
As a work around, what I have done is in my source SQL selection for the two data sources is convert to lower case and sort in alphabetical order (which surprisingly didn't add much overhead - but I will be looking for a more efficient way to do this) and then use the X-Reference Node and specify the data is already sorted in respective reference fields.

That gets me down from a run time of 10-15 minutes to 3-4 minutes.

I might start having a look at the python code behind the core brg library to see exactly how the sorts work behind the scenes
Thanks
Wayne

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

I think you'll find that the sort & join nodes aren't implemented in python and there isn't any code in the core library.
However, if you want, you can always try & write something from scratch using a Python or Java node.
I'm pretty sure that the code in the sort & join node has already been optimised, and is a compiled C++ implementation which will likely be faster than what you would be able to achieve in Python or Java anyway.

When you were using the lookup node, did you make sure that the smaller (i.e. 10,000 record) data set was provided to the bottom input of the node?
It is only the bottom input that gets loaded into memory, so you should make sure that the smaller data set gets connected to this input.


Tim.