LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Feridun

Hey everybody,
Do you know how to change the maximum size of the temp sort files? We are sorting a table of 1,9 billions of rows, it creates about 25k files 15MB each or maybe we can change the max number of temp files, so that it speeds up. The reason whay i'm asking this is because its very slow.. And best practice of fast sorting is appreciated

Another question is: we need to re-locate the temp file folder without reinstalling the LD. The path should exist in one of the .prop files but i could not find. Can anybody please help?

Thank you all and happy new year

Cheers

Feridun

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: xathras

Hi Feridun,

I recently just changed our temp file location. I am going to assume that you're using 4.5.4 Server. Navigate to your install path. Ours is: /opt/software/lae/4.5.4. Under there you'll see a conf directory, followed by brain. There you will see a ls_brain.prop file.

The header of the file will look something like:

#
# General brain properties
#

ls.brain.home= "/opt/software/lae/4.5.4/"
ls.brain.binDir="${ls.brain.home}/bin"
ls.brain.archBinDir="${ls.brain.binDir}/${ls.arch}"


The values to change your temp file location are:

ls.brain.tempDir= "/work/tmp"
ls.brain.outputTempDir= "/work/tmp"
ls.brain.localTempDir= "/work/tmp"
ls.brain.logTempDir="${ls.brain.localTempDir}"

I have mine set to /work/tmp which is a NFS Mount Point. I'll do a deeper dive in some documentation I have from the Lavastorm Team to see how you'd go about changing your temp file size issue.

Regards
Wayne

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: xathras

Hi,

Just checked the config directory and there is also ls_brain_node.prop which might help. Some config below. Still trying to dig out a document

################################################## #############################
# Sort
################################################## #############################

ls.brain.node.sort.maxBufferSize=30000000
ls.brain.node.tempDir=${ls.brain.node.localTempDir }

################################################## #############################
# Join
################################################## #############################
ls.brain.node.join.maxBufferSize=536870912

################################################## #############################
# Lookup
################################################## #############################
ls.brain.node.lookup.maxBufferSize=536870912

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: xathras

Feridun,

No joy in the documentation I had. Didn't specifically mention this scenario. I did however, look at the core.brg file which holds the core Sort node itself. In the tool code it specifies the following:

MinInputs = 1
MaxInputs = 100000
MinOutputs = 1
MaxOutputs = 1

I think what's happening is that the MaxInputs definition at 100,000 is causing the increased files and performance issues, since 25k File Open and Closes Operations is going to be an expensive overhead. Someone from Lavastorm would need to confirm my understanding here. I'm not sure you can do anything about that. Maybe updating the core.brg, but then you're running an unsupported version (even if changing that value worked).

Wayne

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

The 100,000 has nothing to do with the sort segment size. The sort node does NOT try to open 100k files.

change the setting:
ls.brain.node.sort.maxBufferSize=30000000
double the 3 to a 6 and you should see half as many sort segments created. The full answer is a bit more technical than that, but give it a shot.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: xathras

Thanks for clarifying. I have experienced a similar issue where I posted about quick sort because I get slower performance on larger data sets - I tried the lookup node but that didn't help so much. At least my original posting on the cache size was in the right direction. Got a little side tracked when looking at the sort node itself . I am going to assume that the 100,000 is the max inputs that can be pinned to the sort node instead ;-).

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

Yes. A little known feature of the SORT node is that you can send it multiple inputs, and as long as they all have the same columns, it functions like a CAT+SORT all in one step.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: Feridun

Hi Wayne, Smith,
Thank you both for your answers.

1. For changing the directory of the tmp files, as the additional disk provided was exactly the same capacity
2. For the Sorting performance, I can say that multiple inputs is a good way that I did not know. Thank You However, I have only one single data to sort. I don't know what it gives to split it first, then sort+cat again. By the way, I had already set the buffersize to 60000000. I wonder if it is really the number of the files that causes slow sorting, or the disk? In may be the disk. What do you say?

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: xathras

Feridun,

Have you looked at an IOSTAT on your machine when this is running? What is the size of your left input and size of your right input. Have you tried using a lookup node instead? As I mentioned above it helped a little but didn't resolve fully.

Alternatively you can install Sar a nice Linux tool that will give you some performance information. See example below:

See how my system gets busy about 6am

[root@pil-bl-lap-02 ~]# sar
Linux 2.6.18-308.el5 01/03/2013

12:00:01 AM CPU %user %nice %system %iowait %steal %idle
12:10:01 AM all 0.77 0.00 0.56 0.17 0.00 98.50
12:20:01 AM all 0.70 0.02 0.56 0.17 0.00 98.56
12:30:01 AM all 0.67 0.00 0.55 0.17 0.00 98.61
12:40:01 AM all 0.69 0.00 0.56 0.17 0.00 98.59
12:50:01 AM all 0.95 0.00 1.16 3.32 0.00 94.58
01:00:02 AM all 0.71 0.00 0.80 0.17 0.00 98.32
01:10:01 AM all 0.69 0.00 0.56 0.17 0.00 98.58
01:20:01 AM all 8.09 0.02 1.50 0.26 0.00 90.13
01:30:01 AM all 8.41 0.00 1.54 0.34 0.00 89.72
01:40:01 AM all 20.63 0.00 3.36 3.06 0.00 72.95
01:50:01 AM all 10.53 0.00 1.89 1.16 0.00 86.42
02:00:01 AM all 2.91 0.00 1.08 0.55 0.00 95.46
02:10:01 AM all 2.86 0.00 0.99 0.20 0.00 95.95
02:20:01 AM all 0.69 0.02 0.55 0.17 0.00 98.57
02:30:01 AM all 0.69 0.00 0.55 0.16 0.00 98.60
02:40:01 AM all 0.68 0.00 0.56 0.18 0.00 98.59
02:50:01 AM all 0.66 0.00 0.55 0.18 0.00 98.61
03:00:01 AM all 0.68 0.00 0.55 0.17 0.00 98.60
03:10:01 AM all 0.68 0.00 0.55 0.16 0.00 98.61
03:20:01 AM all 0.67 0.02 0.55 0.18 0.00 98.58
03:30:01 AM all 0.69 0.00 0.55 0.16 0.00 98.59
03:40:01 AM all 0.68 0.00 0.54 0.16 0.00 98.61
03:50:01 AM all 0.69 0.00 0.55 0.16 0.00 98.60
04:00:01 AM all 0.66 0.00 0.55 0.18 0.00 98.60
04:10:01 AM all 0.73 0.01 0.59 0.41 0.00 98.27
04:20:01 AM all 0.68 0.02 0.55 0.17 0.00 98.58
04:30:01 AM all 0.67 0.00 0.55 0.17 0.00 98.61
04:40:01 AM all 2.25 0.00 0.81 0.19 0.00 96.75
04:50:01 AM all 1.83 0.00 0.77 0.17 0.00 97.23
05:00:01 AM all 0.69 0.00 0.54 0.16 0.00 98.61
05:10:01 AM all 0.67 0.00 0.55 0.18 0.00 98.60
05:20:01 AM all 0.68 0.02 0.55 0.19 0.00 98.57
05:30:01 AM all 0.68 0.00 0.54 0.17 0.00 98.61
05:40:01 AM all 0.66 0.00 0.54 0.18 0.00 98.62
05:50:01 AM all 0.68 0.00 0.55 0.18 0.00 98.59
06:00:01 AM all 0.68 0.00 0.55 0.16 0.00 98.61
06:10:01 AM all 15.22 0.00 2.00 0.29 0.00 82.50
06:20:01 AM all 10.92 0.02 1.79 0.38 0.00 86.90
06:30:01 AM all 18.15 0.00 2.93 0.46 0.00 78.45
06:40:01 AM all 33.53 0.00 3.25 3.33 0.00 59.89
06:50:01 AM all 11.05 0.00 2.24 0.65 0.00 86.06
07:00:01 AM all 6.66 0.00 0.80 0.22 0.00 92.32
07:10:01 AM all 21.38 0.00 4.00 0.57 0.00 74.04
07:20:01 AM all 22.51 0.03 2.69 0.58 0.00 74.20
07:30:01 AM all 16.57 0.00 2.31 0.32 0.00 80.80
07:40:01 AM all 5.72 0.00 1.11 0.23 0.00 92.94
07:50:02 AM all 22.57 0.00 2.60 0.42 0.00 74.41


Wayne