Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.
Originally posted by: stonysmithThere are multiple ways to solve this.
When LAE sorts a file, it does so by breaking it into smaller "sort segments" and then sorts each segment individually, merging them back together at the end. If your input file is over a certain size, then the number of "sort segments" can increase to the point that the operating system won't handle that many open files at one time.
Possible solutions are:
a) reduce the amount of rows or columns being sorted
b) increase the size of each sort segment
c) increase the number of open files allowed
====
a) When using the stock histogram node, it only summarizes the column(s) you ask for, but it has to sort ALL other columns of data along with it.. If you put in a filter node in front of the histogram and only emit the column(s) you are going to count, then you'll drastically drop the amount of data going into the sort, but you'll get the same results from the histogram.
b) You can change the size of the sort segments:
locate this file: {install_folder}/conf/brain/ls_brain_node.prop
and in that file, this parameter: ls.brain.node.sort.maxBufferSize=30000000
I most often change that setting to 50000000 - 50mb I would not suggest setting it higher than a gigabyte. Be aware that playing with this setting (up or down) can affect sort performance, so it's a bit of trial and error to get it just right.
c) In Linux, there is a command to set the number of allowed open files:
It most often defaults to 1000. If you set that up to 10000, you might then never see the problem again with SORT. But, there's a small catch.. there is an "master" setting for this value which is owned by the root user and you can't specify a limit higher than the master value. You will have to use root access if you want to need to set the system-wide limit.