Skip to main content

Snowflake Run Once per Flow ONLY After All Records Leading up to it are Complete

Planned

Comments

4 comments

  • viliandy leonardo

    Jeffery Hill

    Hi,

    Do you have examples or scenarios where you need such functionality? And in this scenario, what's the size of "all records"?

    Thanks

    0
  • Tyler Lamparter Senior Solutions Consultant
    Celigo University Level 4: Legendary
    Engaged

    Hey Viliandy,

    The use case is it's cheaper to make fewer calls to snowflake. So if we use the run once per page, then we make several calls to convert imported temp table data to prod tables. Ideally we can just import all record into a temp table, then run a single merge query on the "run once per flow". The issue now is it will run the once per flow query once the first page or records hits it. I would like it to hold off on making the query until all of the imports are complete.

    0
  • viliandy leonardo Product Management Director

    Tyler Lamparter

     

    Thank you for the additional detail. The "Run once per flow" is kicked off only the first page and we are investigating to enhance this option so that it can run on the last page.

     

    0
  • Scott Henderson CTO

    I am late to the game here, but it is worth noting that there is a simple work around to accomplish the desired setup: use two separate flows.  The first flow should import all the data into the temp table, and then the second flow should run the merge on the full data set imported by the first flow.  The two flows should be linked to always run one after the other using the "Next integration flow" field in the flow settings drawer.

    This is the setup that we use internally at Celigo Inc to sync all our different cloud app data sources into our own instance of Snowflake, and it works great, and is super efficient.

    1

Please sign in to leave a comment.