LAE

Welcome to the LAE community!  Please feel free to start a discussion in the discussion tab or join in a conversation.

Discussions

Members

Resources

Events

 View Only

Lavastorm Server Performance

  • 1.  Lavastorm Server Performance

    Employee
    Posted 09-02-2013 19:36

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: markrolfo

    Hi,

    Currently Running lavastorm enterprise server on a Oracle M3000 Sparc64VII server 2.5 quad core (8 total using virtulization and multithreadding), 64gb ram with fast SAN.
    Currently system performance is not fast enough for some business processes taking place, need to speed it to in order to complete some time sensitive critical business activities.

    Has anyone used Oracle T4-1 server. 2.85GHZ 8-core 64gig ram for processing data on lavastorm server? If so how was performance.

    Understand this might not be a question for the forum as it does not relate to the actual software.
    Have sped up graphs using has split and other techniques.


    Cheers,
    Mark.
    Lavastorm User - Melbourne, Australia


  • 2.  RE: Lavastorm Server Performance

    Employee
    Posted 09-11-2013 13:04

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: xathras

    Mark, how many threads is your lavastorm process running in parallel?

    We got a significant performance benefit from increasing that. Let me dig out out config


  • 3.  RE: Lavastorm Server Performance

    Employee
    Posted 09-11-2013 13:11

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: xathras

    Hi Mark,

    Go to your install directory. e.g. mine is /opt/software/lae/4.5.4/
    You will see some folders navigate down to conf/brain and you will see ls_brain.prop.

    Navigate down to the controller properties the file will look something like:

    ################################################## #############################
    # Controller properties
    ################################################## #############################

    # Specifies which messages the controller can handle
    ls.brain.controller.messageRegistry="${ls.brain.ho me}/conf/brain/controller/messages.conf"

    ls.brain.controller.logDir="${ls.brain.logDir}/controller"
    ls.brain.controller.logFile="controller.log"
    ls.brain.controller.farm=( "localhost:${ls.brain.server.port}:20" )
    ls.brain.controller.stateDumpInterval = 300 # seconds. negative-> only dump state at end
    ls.brain.controller.executable = ( "python" "${ls.binDir}/laeController" "--background" )
    ls.brain.controller.windows-x86-32.executable = ( "python.exe" "${ls.binDir}/laeController" )
    ls.brain.controller.brdLogFile="controller.log.brd "
    ls.brain.controller.scheduler = "non-streaming"
    # "debug", "non-streaming", "streaming", "default"

    ls.brain.brxDeploymentDir = "${ls.brain.home}/deploy"
    ################################################## #############################

    We set ls.brain.controller.farm to add 20 to the end. This will be 20 threads a significant performance benefit happened when we did that.
    You can also set ls.brain.controller.scheduler from non-streaming to streaming.


  • 4.  RE: Lavastorm Server Performance

    Employee
    Posted 09-11-2013 18:58

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: markrolfo

    We are running up to 8 threads simultaneously.


  • 5.  RE: Lavastorm Server Performance

    Employee
    Posted 09-12-2013 13:15

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: xathras

    A box of your size, similar to ours can handle much more. Do you have sar installed on your machine and look at the system performance over a 24 hour period?

    Will look something like the below:
    [wtaylor@pil-bl-lap-02 ~]$ sar -u
    Linux 2.6.18-308.el5 (hostname) 09/12/2013

    12:00:01 AM CPU %user %nice %system %iowait %steal %idle
    12:10:01 AM all 5.72 0.00 1.41 2.80 0.00 90.07
    12:20:01 AM all 6.25 0.00 1.42 1.15 0.00 91.18
    12:30:01 AM all 2.37 0.00 0.76 1.00 0.00 95.87
    12:40:01 AM all 6.20 0.27 1.44 7.25 0.00 84.83
    12:50:01 AM all 4.88 0.00 1.32 5.22 0.00 88.58
    01:00:02 AM all 14.39 0.00 1.88 0.94 0.00 82.80
    01:10:01 AM all 22.74 0.00 3.13 0.72 0.00 73.41
    01:20:01 AM all 15.80 0.00 2.83 0.54 0.00 80.83
    01:30:01 AM all 18.96 0.00 3.42 1.00 0.00 76.62
    01:40:01 AM all 25.94 0.28 4.30 4.80 0.00 64.68
    01:50:01 AM all 9.29 0.00 2.28 1.28 0.00 87.14
    02:00:01 AM all 0.70 0.00 0.62 0.16 0.00 98.52
    02:10:01 AM all 8.36 0.00 2.10 0.21 0.00 89.33
    02:20:01 AM all 4.87 0.00 1.34 0.20 0.00 93.59
    02:30:01 AM all 2.65 0.00 0.92 0.19 0.00 96.24
    02:40:01 AM all 6.43 0.27 1.51 0.23 0.00 91.56

    Do you also have anything breakdown down what nodes are taking the longest? We have something that scrapes over the longs and determines what nodes are taking the longest. Typically SQL gathering is our biggest and get a lot of success in optimizing them.


  • 6.  RE: Lavastorm Server Performance

    Employee
    Posted 09-12-2013 18:28

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: markrolfo

    We use cacti, but i can look at the processes through SSH.
    Our SQL gatherings are quite quick due to our fast network connection between servers, our x-refs and sorts are taking the longest, as part of a systems integration we have to verify accounts from a different kenan billing engine before passing accounts into production with our own billing engine (kenan) with a limited time (200k in 11 hours)

    Extensive testing has found that through bench-marking graphs the server performance is the same as a core 2 duo desktop. It could be something weird occurring with the ZFS file system, SAN or memory leak.
    The editing of the number of maximum nodes increased our system performance by 9%.

    Are you using redhat instead of solaris?


  • 7.  RE: Lavastorm Server Performance

    Employee
    Posted 09-13-2013 02:13

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: Tim Meagher

    Hey,

    Just a note on the ls.brain.controller.farm and ls.brain.controller.scheduler settings - these will only take effect if you are running the graph as a deployed BRX via the command line controller.
    If you are running this through BRE, then the setting changes will need to be made under Tools->Preferences->Server Farms.
    There you can set the number of threads and whether or not the graph is using streaming/non-streaming mode if you setup your farm to use the external controller.

    If your main problem areas are x-refs and sorts, then changing to streaming mode isn't likely to help too much though.
    Streaming ensures that if there are nodes which produce a single output that is consumed by only one other node, then the data transfer between these nodes is done via socket streaming, meaning that both nodes can execute at the same time. In doing so disk I/O overhead is also reduced.

    A sort node, however, needs to consume all its input records in order to do the sort on the data prior to outputting any records, so streaming won't help here.
    Similarly, an X-Ref node contains two (optionally disabled) sort nodes followed by a Join node which has multiple outputs - so again not much streaming benefit.

    There are a couple of things that you can do though.
    One potential area for performance issues is unecessary sorting.
    Make sure that if the data is coming into the X-Ref sorted - based on the keys that the X-Ref uses - that you aren't re-sorting within the X-Ref.

    Also, you should probably consider the use of hash split.

    If you are not getting much better performance than on a desktop machine, this is likely because the graph is not heavily parallelized.
    Due to the nodes being used (sorts, X-Refs) and graph topology (not many singly-dependent nodes) streaming mode is also not providing much benefit to allow for concurrent processing.

    In this situation, what you'll see is that even though you have 8 threads available to be executing, only a few nodes will be executing at any given time.

    Each node runs in a single process, therefore unless multiple nodes are able to be run concurrently, the size of the server - in terms of raw number of CPUs - won't make much of a difference to performance.
    As a heavily exaggerated example, if you have 10 sort nodes in a chain, only one of them will ever be able to run at one time, in one process.
    It won't matter if these are run on a server with 50 CPUs and using 100 LAE threads, or if these are run on a server with 1 CPU and 1 LAE thread (providing nothing else is running on that machine) - the only thing that will really impact performance is the speed of a single CPU, memory and disk speed.


    By using a hash split node, you can split the data processing across multiple nodes that can be run in parallel.
    In doing so, the amount of work that each node needs to perform is reduced and at the same time, more nodes will be able to run concurrently, meaning that you will be able to make better use of the CPUs and LAE threads available on the server.

    Hope this helps,
    Tim.


  • 8.  RE: Lavastorm Server Performance

    Employee
    Posted 09-13-2013 04:42

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: stonysmith

    Another small suggestion is to not perform more I/O operations than you need to.

    I've seen many instances where a user joins 200k records to 200k records using an XREF, and "most" of the records come out either side as orphans. Then, they only use the MATCH output. If you have an output pin that has a large number of records on it that you never use (is not connected downstream), then my suggestion is to disable that output like this:

    #disabled for performance
    #output 3 {
    # emit 2:*
    # where join.rightOrphan
    #}

    On one example I saw this week, the user had a node that was running for 49 minutes (22m records) and it was sending 98% of the records to pin #3 which were then not used downstream. That meant that the system had to spend time coping all those unused records to an (unused) temp file. Commenting out the Output statement for that pin dropped the node execution time to something like four minutes. - Your mileage will vary - but, you could try to reduce unused outputs.


  • 9.  RE: Lavastorm Server Performance

    Employee
    Posted 09-13-2013 08:51

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: xathras

    Great Suggestion


  • 10.  RE: Lavastorm Server Performance

    Employee
    Posted 09-13-2013 09:10

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: xathras

    Originally posted by: Tim Meagher
    					

    Hey,

    Just a note on the ls.brain.controller.farm and ls.brain.controller.scheduler settings - these will only take effect if you are running the graph as a deployed BRX via the command line controller.
    If you are running this through BRE, then the setting changes will need to be made under Tools->Preferences->Server Farms.
    There you can set the number of threads and whether or not the graph is using streaming/non-streaming mode if you setup your farm to use the external controller.

    If your main problem areas are x-refs and sorts, then changing to streaming mode isn't likely to help too much though.
    Streaming ensures that if there are nodes which produce a single output that is consumed by only one other node, then the data transfer between these nodes is done via socket streaming, meaning that both nodes can execute at the same time. In doing so disk I/O overhead is also reduced.

    A sort node, however, needs to consume all its input records in order to do the sort on the data prior to outputting any records, so streaming won't help here.
    Similarly, an X-Ref node contains two (optionally disabled) sort nodes followed by a Join node which has multiple outputs - so again not much streaming benefit.

    There are a couple of things that you can do though.
    One potential area for performance issues is unecessary sorting.
    Make sure that if the data is coming into the X-Ref sorted - based on the keys that the X-Ref uses - that you aren't re-sorting within the X-Ref.

    Also, you should probably consider the use of hash split.

    If you are not getting much better performance than on a desktop machine, this is likely because the graph is not heavily parallelized.
    Due to the nodes being used (sorts, X-Refs) and graph topology (not many singly-dependent nodes) streaming mode is also not providing much benefit to allow for concurrent processing.

    In this situation, what you'll see is that even though you have 8 threads available to be executing, only a few nodes will be executing at any given time.

    Each node runs in a single process, therefore unless multiple nodes are able to be run concurrently, the size of the server - in terms of raw number of CPUs - won't make much of a difference to performance.
    As a heavily exaggerated example, if you have 10 sort nodes in a chain, only one of them will ever be able to run at one time, in one process.
    It won't matter if these are run on a server with 50 CPUs and using 100 LAE threads, or if these are run on a server with 1 CPU and 1 LAE thread (providing nothing else is running on that machine) - the only thing that will really impact performance is the speed of a single CPU, memory and disk speed.


    By using a hash split node, you can split the data processing across multiple nodes that can be run in parallel.
    In doing so, the amount of work that each node needs to perform is reduced and at the same time, more nodes will be able to run concurrently, meaning that you will be able to make better use of the CPUs and LAE threads available on the server.

    Hope this helps,
    Tim.
    Hash Splits really helped us out too. We made a library of the common factors downstream and used a hash split to repeat that data x period of times and that CAT it at the end.


  • 11.  RE: Lavastorm Server Performance

    Employee
    Posted 05-27-2014 14:41

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: johnvvu

    Is the suggestion to increase this property valid for LAE 4.6? I'm scheduling brx files from the command line via laecontroller and this property is currently set at 1 on my server.

    ls.brain.controller.farm=( "localhost:${ls.brain.server.port}:1" )