Bulk loading Data into Netsuite from Amazon S3 ( 6M +) record Approach
Looking for ways to Bulk load about 6M records into a customer stage table ( no scripts, straight loads) using Celigo bulk APIs + 24 concurrent threads. From our current test, looks for a small load of 10K records, ONLY 1 thread is being utilized and is taking about 17 min, and we are looking at ways to force Celigo to use ALL 24 threads, so as to get a a significant uptick in data load throughput
From Netsuite side, the Integration is set to 24 concurrent web service connection. Please advise if there a setting that can be tweaked, or a pattern to build the flow needs to be utilized.
-
John Papavizas, it sounds like you have concurrency for the integration set within NetSuite, but it looks like you don't have it set on the connection settings within Celigo. Here are a couple of good articles to follow that explain how concurrency settings and queing work within Celigo:
https://docs.celigo.com/hc/en-us/articles/360034522472-Govern-concurrency-of-NetSuite-connections
https://docs.celigo.com/hc/en-us/articles/360043926372-Configure-connections-to-optimize-throughput-and-governance
Additionally, you may need to modify batch sizing so that more records get exported per page. Here is a good article about that.0
Please sign in to leave a comment.
Comments
1 comment