Is there any way to begin processing pages from an export as soon as they are created?

Alex Baeza

• November 16, 2022 07:56

I am running a flow that exports data from the amazon sp-api.

Each api request returns 100 orders, I have set the integrator.io page size for the export to be 1000 orders.

It appears as if my flow waits until all the pages are retrieved from the api before beginning the next step of my flow (lookup).

Is there any way for me to tell integrator to start the lookup after the first page is batched up? I don't see any reason why I should wait until all pages are retrieved before I begin processing the first page retrieved.

0

• 

  Scott Henderson CTO

  • November 16, 2022 15:38

  

  

  

  As soon as we have enough records to create a page, the page is queued and passed along immediately to the next step in the flow. That said, if the next step is using the same connection as the first step, and the concurrency level on the connection is set to 1, then it will wait to process records because the connection is set to only run 1 thing at a time, etc... I imagine that is the case for your flow pictured above.

  0
• 

  Kathyana Rule Community moderator Technical Writer

  • November 16, 2022 15:49

  

  

  Hello Alex Baeza, 

  Here are some great articles on concurrency that might help you out. 

  • Fine-tune integrator.io for optimal data throughput
  • Configure connections to optimize throughput and governance
  • Preserve the order of records when using concurrency

  Thanks, 

  Kathyana

  0
• 

  Alex Baeza

  • November 16, 2022 18:51

  

  

  

  Scott, thanks for reminding me that the concurrency for the export is also used to communicate the result set to the lookup/import steps.

  Why is the 1 concurrency not just toggled between api calls and passing the page to the lookup?

  If I increase the concurrency from 1 to 2 won't this just increase the rate at which I make calls to the api export? What I would like to know is "is there a priority for using the concurrency?". If export api calls are prioritized, then the lookup will still wait until all api calls are made before it gets a chance to process a page.

  0
• 

  Alex Baeza

  • November 16, 2022 18:53

  

  

  

  Kathayana, Thank you for the material. I have read some of those already, I will read the ones haven't already, and re-read those that I have to help my understanding of the process.

  0
• 

  Alex Baeza

  • November 16, 2022 19:10

  

  

  

  Aha!

  Both my export and my lookup use the same credentials for the SP-API but they are essentially hitting two different APIs with different rate limits.

  I will create dedicated connections for export and lookup and set the concurrency on both of them to 1, which should fix the issue while still limiting the concurrency to each api.

  Thanks.

  0

Please sign in to leave a comment.