Skip to main content

Schedule Webhooks

Comments

5 comments

  • Courtney Jordan

    Hi Tyler Lamparter!

    Just to clarify, since we can't schedule webhooks per se (since that's on the "client" side and we can't control when they issue their events), are you envisioning something more like integrator.io "collecting" a batch of records from those webhook calls and sending them to Snowflake as a single batch (either once the batch reaches a certain size, or at a certain frequency, such as once per day) instead of record-by-record as they arrive?
     
    Comment actions Permalink
  • Tyler Lamparter

    Hi Courtney,

     

    Yes I envision you all collecting them in storage and then having the flow on a schedule to pull all the events that have aggregated in batch. Webhooks are great since you don't have to go fetch the data, but if you don't need the data updated in realtime to other systems then it puts some unnecessary load. 

    Comment actions Permalink
  • viliandy leonardo

    Hi Tyler Lamparter

    Does Snowflake not suspend the datawarehouse when it is not running?

    Would you consider staging data you receive from a webhook to a database and then kick off another flow that loads data from the staging to Snowflake?

    Comment actions Permalink
  • Tyler Lamparter

    Snowflake does suspend the warehouse, but if the webhook constantly has data coming into it, then the warehouse will never get a chance to suspend. Also I don't think I should have to use a different database to stage the data, then import to snowflake. This is an enhancement request for you all to stage the events and hold them for scheduling.

    Comment actions Permalink
  • viliandy leonardo

    Tyler Lamparter

    I acknowledge the idea put forward here. The database staging route was a suggestion until we have built a proper solution to address the needs.

     

    Comment actions Permalink

Please sign in to leave a comment.