Snowflake Run Once per Flow ONLY After All Records Leading up to it are Complete

Planned

Jeffery Hill

• February 17, 2021 01:16

There is an option to run a snowflake import once per flow, but sometimes that once per flow needs to be triggered only after all records have been completed in the steps prior.

1

• 

  viliandy leonardo

  • June 30, 2021 00:10

  Jeffery Hill

  Hi,

  Do you have examples or scenarios where you need such functionality? And in this scenario, what's the size of "all records"?

  Thanks

  0
• 

  Jeffery Hill Senior Solutions Consultant

  • July 01, 2021 15:35

  

  

  Hey Viliandy,

  The use case is it's cheaper to make fewer calls to snowflake. So if we use the run once per page, then we make several calls to convert imported temp table data to prod tables. Ideally we can just import all record into a temp table, then run a single merge query on the "run once per flow". The issue now is it will run the once per flow query once the first page or records hits it. I would like it to hold off on making the query until all of the imports are complete.

  0
• 

  viliandy leonardo Product Management Director

  • August 09, 2021 20:34

  

  

  Jeffery Hill

   

  Thank you for the additional detail. The "Run once per flow" is kicked off only the first page and we are investigating to enhance this option so that it can run on the last page.

   

  0
• 

  Scott Henderson CTO

  • January 27, 2022 00:04

  

  

  

  I am late to the game here, but it is worth noting that there is a simple work around to accomplish the desired setup: use two separate flows.  The first flow should import all the data into the temp table, and then the second flow should run the merge on the full data set imported by the first flow.  The two flows should be linked to always run one after the other using the "Next integration flow" field in the flow settings drawer.

  This is the setup that we use internally at Celigo Inc to sync all our different cloud app data sources into our own instance of Snowflake, and it works great, and is super efficient.

  1

Please sign in to leave a comment.