Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.
Originally posted by: Tim MeagherHey,
Just a note on the ls.brain.controller.farm and ls.brain.controller.scheduler settings - these will only take effect if you are running the graph as a deployed BRX via the command line controller.
If you are running this through BRE, then the setting changes will need to be made under Tools->Preferences->Server Farms.
There you can set the number of threads and whether or not the graph is using streaming/non-streaming mode if you setup your farm to use the external controller.
If your main problem areas are x-refs and sorts, then changing to streaming mode isn't likely to help too much though.
Streaming ensures that if there are nodes which produce a single output that is consumed by only one other node, then the data transfer between these nodes is done via socket streaming, meaning that both nodes can execute at the same time. In doing so disk I/O overhead is also reduced.
A sort node, however, needs to consume all its input records in order to do the sort on the data prior to outputting any records, so streaming won't help here.
Similarly, an X-Ref node contains two (optionally disabled) sort nodes followed by a Join node which has multiple outputs - so again not much streaming benefit.
There are a couple of things that you can do though.
One potential area for performance issues is unecessary sorting.
Make sure that if the data is coming into the X-Ref sorted - based on the keys that the X-Ref uses - that you aren't re-sorting within the X-Ref.
Also, you should probably consider the use of hash split.
If you are not getting much better performance than on a desktop machine, this is likely because the graph is not heavily parallelized.
Due to the nodes being used (sorts, X-Refs) and graph topology (not many singly-dependent nodes) streaming mode is also not providing much benefit to allow for concurrent processing.
In this situation, what you'll see is that even though you have 8 threads available to be executing, only a few nodes will be executing at any given time.
Each node runs in a single process, therefore unless multiple nodes are able to be run concurrently, the size of the server - in terms of raw number of CPUs - won't make much of a difference to performance.
As a heavily exaggerated example, if you have 10 sort nodes in a chain, only one of them will ever be able to run at one time, in one process.
It won't matter if these are run on a server with 50 CPUs and using 100 LAE threads, or if these are run on a server with 1 CPU and 1 LAE thread (providing nothing else is running on that machine) - the only thing that will really impact performance is the speed of a single CPU, memory and disk speed.
By using a hash split node, you can split the data processing across multiple nodes that can be run in parallel.
In doing so, the amount of work that each node needs to perform is reduced and at the same time, more nodes will be able to run concurrently, meaning that you will be able to make better use of the CPUs and LAE threads available on the server.
Hope this helps,
Tim.