LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: akumari

Hi,
I have a query. Just wondering if someone else has come across the same problem in the past.

While running a lavastorm component on solaris machine, I am getting an error "ERROR performing sort:ntoo many open files (24): '/u02/lavastorm/ssapp0154/tmp/radtietl/RADTI_ETL_S2C_FLEX_V2/production/sort.ssapp0154.13034.4090.1261395934000.brd'". My assumption is that it is related to the ulimit settings in .profile.lavastorm file. The ulimit is set to 4096 in the file but at the kernel level(/etc/system) the setting is set rlim_fd_cur=4096
set rlim_fd_max=65536.

Just wondering if the current limit of maximum open file descriptor set in the .profile file (4096) is causing the issue and not letting the no. of open files to be higher than 4096 and not reach to the max limit of 65536.

Also is the ulimit setting really required in the .profile.lavastorm and if it can be deleted?

Thanks
Akshu

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rabbott

Hello Akshu,

The "too many open files" error messages relates to your operating system�s limit of file descriptors, which appears to be too low for your current use.

To rectify this you should try setting your kernel parameter to allow more open file descriptors (fds).

To find out your fds limit, type:

$ bash -c "ulimit -n"

The number returned is the number of fds allowed open concurrently. If you are doing sorts (as in this case) on large files, which is a common cause of this error, you can reduce the number of files the Sort node needs to open at once by increasing it's buffer size using the ls.brain.node.sort.maxBufferSize property in the $BRAIN_HOME/conf/ls_brain_node.prop file (by default it is set to 30000000) .

A common default fds setting for UNIX is 1024. An fds size of at least 4096 is recommended to best accommodate the LAE server.

Some extra information about ULIMIT:

1) You generally can�t set it HIGHER within a single session. If you have ULIMIT in your .profile script, then that limits how high it can go.
2) Solaris 8 has a kernel limit set at 1024.. you�d have to recompile the kernel to increase it.
3) Solaris 10 has the ULIMIT set by the administrator.. you have to change it there to get the "user" ULIMIT changeable.

Hope this helps!

Regards,

- Bob

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: akumari

HI Bob

Thanks for the reply. I have one more query.

If the property ls.brain.node.sort.maxBufferSize property is not specified in the $BRAIN_HOME/conf/ls_brain_node.prop file, would brain still consider the default value(30000000) when running?

Also, can i raise the value of the ulimit in the .profile.lavastorm script to the hard limit value specified in the kernel(the solaris9 machine that i am using has got it set up as 65536).

Thanks, Regards
Akshu

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rabbott

Hi Akshu,

I believe the default ls.brain.node.sort.maxbuffersize = "30000000" is set regardless of whether the setting is present in the ls_brain_node.prop file. One way to verify would be to issue the following command:

$ getProperties -d|grep ls.brain.node.sort.maxbuffersize

With regards to raising the value of the ulimit in the .profile.lavastorm script to the hard limit value specified in your Solaris 9 kernel, the present setting of 4096 should be sufficient for most BRAIN server needs. I would recommend first increasing the ls.brain.node.sort.maxbuffersize parameter to see if that alleviates the error condition you've encountered. If the error persists, then try increasing the ulimit setting.

Hope this helps.

- Bob

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: akumari

Thanks for your help Bob.
Just changing the sort buffer size resolved the issue.

Regards
Akshu

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Daniel Stephenson

Hi,
A question regarding the raising of the various maxBufferSize properties. I'm recieving 'buffer size' errors whilst running join nodes. To resolve i'm just replacing with Lookups which works fine. At some point when my input data inceases these will error too I presume. What are the implications of increasing buffer size and to what level can this be done?

Also, these errors are occuring in simple joins (which can be replaced with lookups) whilst in other very similar joins with the same data it works fine? What are the factors to consider that generate these errors?

I'm running the desktop version 4.5.2 on a Duel Core Duo.

thanks
Daniel

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: maria.pislopez

Hello Daniel,

Joins and lookups work a little differently in regards to duplicate records on the key value. Lookups load only one record per key value into memory (and throws out the duplicates) whereas the join will load all records pertaining to the matched key. As a result, the join will output the cartesian product of the matched records on both sides which can expand the number of outputted records quite a bit.

Usually, we find that the maxbuffersize should not have to be increased. The default value is 500 MB. Many times improperly joined data or many duplicate records will cause the buffer errors. You can check for duplicate records using the Duplicate Detector. If you need to filter out duplicates use the Agg Ex, sort the data and output "where lastInGroup." It might also be worth performing the join on a small subset of records to make sure you obtain the expected result.

Please let us know if you need more information or help on this issue. Also, feel free to upload a graph with sample data of the issue you are having.

Good Luck!
Maria

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: timonk

Daniel,

How high is your current buffer size?

The lists built in memory when using JOIN type nodes (or LOOKUP or XREF) are very sensitive to the join statements being used. It is not really a matter of the size of the data sets being worked with, so much as the size of the cartesian product resulting from the join itself. Depending on how unique, or duplicated, the matching values are for the fields you are joining on you can get wildly different results. And explosive growth.

I am not entirely familiar with the differences "under the hood" for LOOKUP vs. JOIN, but yes if the data set increases, and that increase results in more duplicate values, LOOKUPs will hit the same issue as a JOIN. As they both build lists in memory.

I believe the process memory limit on x32 windows systems is 2GB, and on x64 it is 3GB.

-Timon

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Tim Meagher

Just a further clarification on these points ....

A lookup node processes the bottom (or "right" if you prefer that terminology) input first, and loads all of this into memory. It loads this into a hash map, where the lookup key is used as the hash key. Therefore, for records that have duplicate keys, only one unique entry will end up in the hash map, so duplicates are ignored. In this case of a duplicate lookup key, a warning message will be printed.

I think at the moment, the implementation does an overwrite on the hash map, meaning that only the last duplicate in the data set will be used, but this isn't part of the interface and is not guaranteed.

In any case, the lookup will then process on a one-by-one basis the rows in the top (/left) input. Since there are only unique lookup entries in the hash map, it doesn't need to load more than one record from the top (/left) input at a time, so there is never more than one record from the top (/left) input in memory.

In short: If you don't have duplicates on your lookup key, and your lookup data set - the bottom (/right) input - is pretty small, then you can pass as many records as you want through the top (/left) input without it causing any memory issues.

However, you will start to quickly run into memory issues on the lookup node if you have a lot of (unique) records in the bottom (/right) input.

As Maria has rightly pointed out, the join node on the other hand is different, since this will produce a cartesian product (and load this in memory) for cases where the join key matches. However, if there are no duplicates, the join node will generally require less memory than a lookup node, since it doesn't try and load in one side of the input into memory.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: alyon

Hi All,

I have a query related to the same issue.

One of our application failed with the error:

too many open files (24):
'/u03/lavastorm/ssapp9013v/tmp/nrc_user/Mobile_Precursor/NRC_CYCLE_MAY_07_08/sort.ssapp9013v.23469.1018.1336492620000.brd'

We increased the ulimit from 1024 to 10000. And it solved the problem.

But I was wondering if there is drawback of increasing the ulimit.
Why increasing buffer size is preferred over increasing ulimit?
Is there a middle optimum solution?

We are running this application on a server farm.
Some of the servers have ls.brain.node.sort.maxbuffersize set as 50000000 and on other servers it is set as 30000000.
All the servers have ulimit as 10000 now.

Your inputs are highly appreciated.

Thanks
Anshul

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ejones

Based on what I've figured out, the main reason limit exists in the operating system is to protect against a runaway process bringing the system down. There is no optimal choice of value for this. It needs to be high enough that the processes that are running correctly have the resources they need but low enough to cause any out of control processes to be stopped. LAE does open more files than the normal Unix process so it isn't too surprising that you would need to increase the limit.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: rboccuzzi

Ernest is correct. Additionally, many OS's and software systems had a much smaller limit defaulted for them back when typical available memory was much less. In many cases, to prevent a system from using up significant resources, the limit was set to 1024. On many new installations of UNIX flavors, the default is now much higher than 1024, as typical memory for systems is much larger, and that allocation is less likely to be a system resource issue. I also believe, but am not sure, that the allocation used to be fixed for all processes, but is now dynamic, so the larger allocation no longer controls an overhead of how much each process takes up for resources in that context. This again makes increasing this number a viable option.

When you increase the maximum buffer size, you are increasing the amount of data the sort node is holding IN MEMORY, which is a much bigger amount than the amount of memory each open file will take up. If you have a system with a large amount of memory, and you are not running many sort nodes at the same time, you can certainly increase the max buffer size. You should be aware however, this is one of the few nodes that loads large amounts of data in memory, and improper configuration of this parameter can possibly exhaust your system memory and cause system thrashing.

Therefore, it is a system dependent setting as to what your maximum buffer size is, which should factor in system memory, number of typical concurrent nodes, etc. Increasing the number of open files to greater than 1024 generally is safe however, so definitely the first thing we recommend. I hope this helps clarify the relationship a bit more.

Cheers
Rich

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

I don't have scientific evidence for this, but larger buffer sizes make SORT run slightly slower, however, more open files causes your disk I/O channel(s) to work harder. It is a tradeoff where you should experiment a bit to find the optimum solution for your configuration.

A bit of a side note: depending upon how much data you are processing, it may be worthwhile to investigate usage of the HASH SPLIT node such that you can break up / control how much data is processed in a single pass.