LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dhrobertson

Hi there,

I have a server running a quad core processor and initially only had it running 4 thread for Lavastorm which was the default. the processor was hardly being used at all and I needed a bunch of processes running at the same time so I bumped the threads up to 8. there was no change in the performance increasing to this number. I am only reading data from Database tables so maybe this is why I see no impact, and if I was actually using nodes with computations etc, then I would see more of an impact?

is it fair to say you should only run the same number of thread as you have processor cores, or, can I happily increase the thread limit without impacting the server too much?

Maybe is it a "how long is a piece of string" question but in general what do folk set the threads as.

I know there is a lavastorm document about benchmark testing out there, but I think the testing there was based on a whole bunch of intensive nodes running at the same time.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

You are 100% correct with your metaphor of "how long is a piece of string". In other words... "It Depends".

First: When you are running a database query, the performance is controlled/throttled by how fast the database itself responds. You could have a infinitely fast machine running LAE, but you'll be waiting on the turtle speed of the database.

To get all technical:
LAE is a rather unique piece of software. Most software maxes out on only one of the following resources: the CPU speed, or the I/O speed, or the amount of RAM memory. LAE is a bit unique in that it can consume large amounts of all three of these resources at once. LAE is not like a web server which sends out a web page, and then waits several lifetimes for the user to respond. Remember, 15 seconds of "human" time is 30,000,000,000 cpu cycles. LAE doesn't hold back, it doesn't "wait" any cycles to get the work done.. which means that it can consume a LOT of resources if you have large volumes of data or complex processing requirements.

But, the one thing that LAE does do - is that one node will stay on one processor core. (Most) nodes will only use one core to do their work. That's why you're seeing no change in performance by increasing your thread count - you're not going to make a single node run faster by increasing the thread count.

My experience is that on most modern multi-threaded operating systems and hardware, you can reasonably expect to run 4 tasks on each core, meaning you could set your thread count to 16 without seeing a degradation in performance. But.. that means your graph logic has to be setup such that you can take advantage of the multiple threads.. If your logic is waiting on a single database query or say a single sort, then the extra threads won't help you any. At the same time though, if you are working with datasets that have 100's of millions of records... you may find that your box is maxed out with only a thread count of 4.

Quote: Your mileage WILL vary. <grin>

Humor: You need to also be careful how high you set the thread count. A coworker of mine once set the thread count on a 6 core box to 200, and when he managed to get 160 nodes all running at once, the hardware rebooted, probably due to heat overload. You don't want to do that.

I am happy to deepen the discussion of performance, there are several techniques that can be applied to improve thruput, but, to continue the discussion, I'd need to start asking some other specifics.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: dhrobertson

Thank you very much for your answer. that does help me a lot. the current project I on is not requiring a lot of processing from a lavastorm perspective, but I am doing a lot of pulling for SAP tables and the time it is taking is huge based on the fact I was initially only using 4 threads. If I can bump up to 16 without degradation, then it speeds the process up 4 fold.

another question..... I currently control the number of threads used on the server in the BRE when running a graph through my laptop attached to the lavastorm server. If I save that graph as a LXA and place on the Lavastorm server to be run as a scheduled task.... how do I make sure that the server will run the graph using the 8 threads. is this info saved in the LXA or is this a configuration I need to make on the server itself? I am pretty new to using the server so sorry if it sounds like a basic question.

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: stonysmith

Originally posted by: dhrobertson
					

bump up to 16 without degradation
			

My comment about "four tasks per core" was not hard number.. you will need to experiment with it.
Remember that SAP might limit your number of open queries at one time, or that asking SAP to send you 16 tables in parallel might not be faster. For this situation, it's more about what the SAP server is doing.

Originally posted by: dhrobertson
					

save that graph as a LXA and place on the Lavastorm server
			

There is a server configuration setting which affects the thread count as well as the server and port used when executing a graph. That setting is completely independent from whatever you have selected in BRE.
The setting is set by default in the file {install folder}/conf/brain/ls_brain.prop and this is the default value:
You would need to modify the "1" to a higher number to get more threads.

The setting can be changed at several other points in the system, but I'm not sure if it can be changed per-graph inside of the Logistics Manager. I'll update this thread when I discover the answer.