Is there any way to begin processing pages from an export as soon as they are created?
I am running a flow that exports data from the amazon sp-api.
Each api request returns 100 orders, I have set the integrator.io page size for the export to be 1000 orders.
It appears as if my flow waits until all the pages are retrieved from the api before beginning the next step of my flow (lookup).
Is there any way for me to tell integrator to start the lookup after the first page is batched up? I don't see any reason why I should wait until all pages are retrieved before I begin processing the first page retrieved.
0
Comments
As soon as we have enough records to create a page, the page is queued and passed along immediately to the next step in the flow. That said, if the next step is using the same connection as the first step, and the concurrency level on the connection is set to 1, then it will wait to process records because the connection is set to only run 1 thing at a time, etc... I imagine that is the case for your flow pictured above.
Hello Alex Baeza,
Here are some great articles on concurrency that might help you out.
Thanks,
Kathyana
Scott, thanks for reminding me that the concurrency for the export is also used to communicate the result set to the lookup/import steps.
Why is the 1 concurrency not just toggled between api calls and passing the page to the lookup?
If I increase the concurrency from 1 to 2 won't this just increase the rate at which I make calls to the api export? What I would like to know is "is there a priority for using the concurrency?". If export api calls are prioritized, then the lookup will still wait until all api calls are made before it gets a chance to process a page.
Kathayana, Thank you for the material. I have read some of those already, I will read the ones haven't already, and re-read those that I have to help my understanding of the process.
Aha!
Both my export and my lookup use the same credentials for the SP-API but they are essentially hitting two different APIs with different rate limits.
I will create dedicated connections for export and lookup and set the concurrency on both of them to 1, which should fix the issue while still limiting the concurrency to each api.
Thanks.
Please sign in to leave a comment.