LAE

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: khopkins

Hi Everyone,

Has anyone come across the 10,000 record limit using an update into salesforce objects and if so what was the workaround.

This is beacuse the Nodes do not work the same way as the data loader and try to insert every record in one hit which breaches the API limit.

Any help would be appreciated.

Regards,

Keith

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ltolleson

Keith,

Below are the limitations of the SalesForce Bulk API as stated in their documentation. Based on this information it looks like you will need to split out your data into 10,000 record sets and create a new batch for each set of data.

http://www.salesforce.com/us/develop...ynch/index.htm

Batch size

• Batches for data loads can consist of a single CSV or XML file that can be no larger than 10 MB.
• A batch can contain a maximum of 10,000 records.
• A batch can contain a maximum of 10,000,000 characters for all the data in a batch.
• A field can contain a maximum of 32,000 characters.
• A record can contain a maximum of 5,000 fields.
• A record can contain a maximum of 400,000 characters for all its fields.
• A batch must contain some content or an error occurs.

Hope this helps...

Thanks,
Larry

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: khopkins

Hi Larry,

Thanks for those but i already knew about the limits as this is causing the node to fail.
I need a way to get the node to only process 10,000 records at a time and then start a new batch or if not the node than a way of sending data in 10,000 record chunks to the same node without using a hash split and multiple salesforce bulk update/insert nodes.
This is because the number of records can vary between 5,000 and 2,500,000 each time the graph is going to be run.

Regards,

Keith

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: ltolleson

Keith,

You are correct, this is a limitation of SalesForce. The way around this is to break your data up into smaller ( <10,000) record batches. I'm sure you want to do this dynamically and unfortunately we do not have a node that can loop and execute the SalesForce nodes multiple times in your graph. There is a looping node that is scheduled for LAE 5.0 later this year that will be very helpful in these types of situations.

BUT... There is another way to do this, but it requires 2 graphs. The main graph, which has the data you want to process and a 2nd graph that gets called from the main graph that executes the SalesForce functionality. I have attached a zip file that contains an example of these two graphs. You might find other ways using the executeBRX node, but this is how I solved it. There is a graph parameter on the SalesForceProcess graph called BatchSize. You will want to set this to the size of your batch.

Hope this helps...

Thanks,
Larry
Attachments:
SaleForceBatch.zip

Note: This was originally posted by an inactive account. Content was preserved by moving under an admin account.

Originally posted by: khopkins

Perfect thanks Larry this works a charm after tweaking slightly for my requirements.

Regards,

Keith