Planned
Snowflake Run Once per Flow ONLY After All Records Leading up to it are Complete
There is an option to run a snowflake import once per flow, but sometimes that once per flow needs to be triggered only after all records have been completed in the steps prior.
1
Comments
I am late to the game here, but it is worth noting that there is a simple work around to accomplish the desired setup: use two separate flows. The first flow should import all the data into the temp table, and then the second flow should run the merge on the full data set imported by the first flow. The two flows should be linked to always run one after the other using the "Next integration flow" field in the flow settings drawer.
This is the setup that we use internally at Celigo Inc to sync all our different cloud app data sources into our own instance of Snowflake, and it works great, and is super efficient.
Jeffery Hill
Hi,
Do you have examples or scenarios where you need such functionality? And in this scenario, what's the size of "all records"?
Thanks
Hey Viliandy,
The use case is it's cheaper to make fewer calls to snowflake. So if we use the run once per page, then we make several calls to convert imported temp table data to prod tables. Ideally we can just import all record into a temp table, then run a single merge query on the "run once per flow". The issue now is it will run the once per flow query once the first page or records hits it. I would like it to hold off on making the query until all of the imports are complete.
Jeffery Hill
Thank you for the additional detail. The "Run once per flow" is kicked off only the first page and we are investigating to enhance this option so that it can run on the last page.
Please sign in to leave a comment.