LAE

Welcome to the LAE community!  Please feel free to start a discussion in the discussion tab or join in a conversation.

Discussions

Members

Resources

Events

 View Only
  • 1.  SQL server bulk load

    Employee
    Posted 09-18-2012 02:06

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: khopkins

    Hello all

    I've been using the 'JDBC store' node to insert data into an SQL Server (2008 R2) database.
    I've managed to get it all working but I'm only able to insert approx. 17,000 records every 30mins.

    On investigate it led me to believe that I needed to add the additional line to the connection string 'EnableBulkLoad=true;BulkLoadBatchSize=10000', this doesn't seem to work.

    Do I need to configure/specify anything else to enable bulk load?

    Does anyone have any ideas please?

    Regards,

    Keith


  • 2.  RE: SQL server bulk load

    Employee
    Posted 09-18-2012 13:56

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: xathras

    Hi Keith,

    Try this. On the JDBC Store Node click on Parameters 2, then on Load Method Change from Insert to Optimized and then set your commit frequency to a frequency you're comfortable with. In the Lavastorm help it recommends a few thousands. I've used this at Gogo with great success.

    Regards
    Wayne


  • 3.  RE: SQL server bulk load

    Employee
    Posted 09-19-2012 02:13

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: khopkins

    Thanks Wayne


  • 4.  RE: SQL server bulk load

    Employee
    Posted 10-03-2012 07:50

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: mgreenbank

    Thanks for the reply Wayne - I've tried your solution and I'm now able to insert the same 17,000 records into an SQL Server database in approx. 5 seconds. I'm now coming across a second issue, I've set the commit frequency to 10,000, on inserting just over 2.2 million records (into an SQL Server database) it completes approx. 60K~100K records and fails with an error message of 'Installing Log', have you experienced this at all?

    Thanks

    Matt


  • 5.  RE: SQL server bulk load

    Employee
    Posted 10-07-2012 01:06

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: xathras

    Matt are you running the Desktop or Server Version? It maybe worth while increasing your logging level and seeing if that helps. There should be an option in the node parameters.


  • 6.  RE: SQL server bulk load

    Employee
    Posted 10-11-2012 03:36

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: mgreenbank

    Hi xathra, I'm using the server edition of lavastorm with the Business Rules Editor on the client, I can't seem to find the option to increase the logging level in the node, can you clarify where this is located?

    Thanks

    Matt


  • 7.  RE: SQL server bulk load

    Employee
    Posted 10-11-2012 05:41

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: xathras

    Hi Matt,

    Let me look at our configuration here and i will post an update for you.

    Regards
    Wayne


  • 8.  RE: SQL server bulk load

    Employee
    Posted 10-11-2012 07:44

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: rabbott

    Hi Folks,

    If technical support is needed for this issue, please use one of the following options:

    1. Send a message to helpdesk@mdsuk.com
    2. Report the issue via the on-line incident management system - https://support.mdsuk.com/secure/Dashboard.jspa
    3. Contact the MDS Service Desk by calling 01925 555359

    Cheers,

    - Bob


  • 9.  RE: SQL server bulk load

    Employee
    Posted 10-12-2012 06:33

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: xathras

    Hi Matt,

    I retried this approach and increasing the logging level, but unfortunately I didn't get much from it. I'd suggest you reach out to the Lavastorm Support Team, sorry I couldnt help you any further.

    Regards
    Wayne


  • 10.  RE: SQL server bulk load

    Employee
    Posted 10-13-2012 14:40

    Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

    Originally posted by: stonysmith

    One note: "Installing Log" is not an error. That is simply a message to tell you that the log file was opened.
    Unfortunately, the error message you need seems to be not written out. Have you tried Log Level=3 or 4 ?