Schedule Webhooks
I would like the ability to schedule webhooks. When populating a database with data from webhooks, you have to constantly have the connection open to the database since the webhook is live. Other solutions I've used let you schedule webhooks so they will basically hold the webhook data in storage until the scheduled run.
For example, I bring data into Snowflake, but I don't use webhooks to Snowflake because we would eat through our credits by constantly having the warehouse running.
-
Hi Tyler Lamparter!
Just to clarify, since we can't schedule webhooks per se (since that's on the "client" side and we can't control when they issue their events), are you envisioning something more like integrator.io "collecting" a batch of records from those webhook calls and sending them to Snowflake as a single batch (either once the batch reaches a certain size, or at a certain frequency, such as once per day) instead of record-by-record as they arrive? -
Hi Courtney,
Yes I envision you all collecting them in storage and then having the flow on a schedule to pull all the events that have aggregated in batch. Webhooks are great since you don't have to go fetch the data, but if you don't need the data updated in realtime to other systems then it puts some unnecessary load.
-
Snowflake does suspend the warehouse, but if the webhook constantly has data coming into it, then the warehouse will never get a chance to suspend. Also I don't think I should have to use a different database to stage the data, then import to snowflake. This is an enhancement request for you all to stage the events and hold them for scheduling.
Please sign in to leave a comment.
Comments
5 comments