Just how does page and batch size effect flow execution - Official Answer Needed!
Hello,
My partner and I were discussing an issue with a client's flow today and we went down the path of page size (exports) and batch size (updates). At one point I stopped and asked him
So does the update process it's whole batch and then move on, or does it hand off each record to the next step?
(or something like that)
I decided I'd ask here, for either someone in the community to answer (if you're POSITIVE) or for Celigo to give an official answer. Consider the following example flow:
- Exports records with page size of 100
- updates the same record with batch size of 20
- updates a related record with batch size of 100
This is a made up scenario to give context for the question.
My questions:
- In stop #1 I'm assuming 100 results (if available) will be pulled down and passed to step #2 as a batch of 100 records - correct?
- Does step #2 process 20 records then pass to step #3 or does it run 5 times in 20 record chunks and then pass all 100 to step #3?
- If #2 passed 20 at a time to step #3, does step #3 buffer until it reaches it's batch size threshold or does it process the 20 as it receives them?
- Sort of asking a repeat question, but in a different way: If #1 has a page size of 1 and the export search yields 100 results, will each records go through the entire flow sequentially or will they be buffered by the first node and batch processed?
Maybe there is a help document that spells this out, but I haven't found it. Hopefully someone has the answer; it would certainly be helpful for me and probably a few others out there.
Regards,
Steve
Comments
Hi Steve Klett, perhaps this is the doc you were looking for?
https://docs.celigo.com/hc/en-us/articles/360043927292-Fine-tune-integrator-io-for-optimal-data-throughput#page-size
Please sign in to leave a comment.