Snowflake Run Once per Flow ONLY After All Records Leading up to it are Complete




  • Scott Henderson CTO
    Celigo University Level 1: Skilled
    Answer Pro
    Top Contributor

    I am late to the game here, but it is worth noting that there is a simple work around to accomplish the desired setup: use two separate flows.  The first flow should import all the data into the temp table, and then the second flow should run the merge on the full data set imported by the first flow.  The two flows should be linked to always run one after the other using the "Next integration flow" field in the flow settings drawer.

    This is the setup that we use internally at Celigo Inc to sync all our different cloud app data sources into our own instance of Snowflake, and it works great, and is super efficient.

  • viliandy leonardo

    Jeffery Hill


    Do you have examples or scenarios where you need such functionality? And in this scenario, what's the size of "all records"?


  • Jeffery Hill Senior Solutions Consultant
    Celigo University Level 4: Legendary

    Hey Viliandy,

    The use case is it's cheaper to make fewer calls to snowflake. So if we use the run once per page, then we make several calls to convert imported temp table data to prod tables. Ideally we can just import all record into a temp table, then run a single merge query on the "run once per flow". The issue now is it will run the once per flow query once the first page or records hits it. I would like it to hold off on making the query until all of the imports are complete.

  • viliandy leonardo Product Management Director
    Answer Pro
    Top Contributor

    Jeffery Hill


    Thank you for the additional detail. The "Run once per flow" is kicked off only the first page and we are investigating to enhance this option so that it can run on the last page.



Please sign in to leave a comment.