Skip to main content

Is there any way to begin processing pages from an export as soon as they are created?

Comments

5 comments

  • Scott Henderson CTO
    Answer Pro
    Top Contributor

    As soon as we have enough records to create a page, the page is queued and passed along immediately to the next step in the flow. That said, if the next step is using the same connection as the first step, and the concurrency level on the connection is set to 1, then it will wait to process records because the connection is set to only run 1 thing at a time, etc... I imagine that is the case for your flow pictured above.

    0
  • Kathyana Rule Community moderator Technical Writer

    Hello Alex Baeza

    Here are some great articles on concurrency that might help you out. 

    Thanks, 

    Kathyana

    0
  • Alex Baeza
    Answer Pro
    Celigo University Level 4: Legendary
    Awesome Follow-up

    Scott, thanks for reminding me that the concurrency for the export is also used to communicate the result set to the lookup/import steps.

    Why is the 1 concurrency not just toggled between api calls and passing the page to the lookup?

    If I increase the concurrency from 1 to 2 won't this just increase the rate at which I make calls to the api export? What I would like to know is "is there a priority for using the concurrency?". If export api calls are prioritized, then the lookup will still wait until all api calls are made before it gets a chance to process a page.

    0
  • Alex Baeza
    Answer Pro
    Celigo University Level 4: Legendary
    Awesome Follow-up

    Kathayana, Thank you for the material. I have read some of those already, I will read the ones haven't already, and re-read those that I have to help my understanding of the process.

    0
  • Alex Baeza
    Answer Pro
    Celigo University Level 4: Legendary
    Awesome Follow-up

    Aha!

    Both my export and my lookup use the same credentials for the SP-API but they are essentially hitting two different APIs with different rate limits.

    I will create dedicated connections for export and lookup and set the concurrency on both of them to 1, which should fix the issue while still limiting the concurrency to each api.

    Thanks.

    0

Please sign in to leave a comment.