This article provides a conceptual overview of webhook technology as it relates to the Celigo platform and includes the steps necessary to create listeners in your flow.
Webhooks send an HTTP POST message over the web in real-time to an endpoint when an event occurs in an application. When an event triggers a webhook request, the source application sends a message to a specified URL, and a webhook listener receives the message and then executes operations according to the message's content.
Before webhook technology was available, if you wanted to sync information in two places across the web, one system would have to send repeating poll messages to review the status of the data in the system it needed to retrieve new information from, and then update data based on the responses to the poll requests. Webhook technology eliminates the necessity for constant polling by allowing one system to automatically send a notification and all relevant information of a given change to another system as soon as the change occurs.
integrator.io supports webhook technology on source application exports. If you set up webhook settings within the user interface of your source application, you can configure listeners in the Celigo platform to receive messages from the source app and trigger integration flows based on the message content. Setting up webhooks requires two steps:
-
Configure a webhook in the source application for an event that triggers the HTTP POST message from the source application to the Celigo platform.
-
Create a webhook listener in the Celigo platform to receive incoming information from the source application and initiate the flow with the data sent.
Celigo platform webhook listeners can receive the following content types:
-
application/json
-
*/xml
-
application/x-www-form-urlencoded
Webhook listeners can't parse CSV data or any other content types besides those listed above.
Tip
Organizations may use different terms for webhook technology. Consult the API documentation for your source app to determine how it refers to these concepts.
These instructions apply to the universalWebhook connection. If you are listening for a webhook from a built-in integrator.io connection, instead choose that Application and then configure the export to use a webhook.
-
From the Resources menu, select Export.
-
In the resulting Create export panel, select Webhook for the Application setting.The Create listener panel opens.
-
Enter a Name and a brief Description for the webhook.
-
Click Next.
Choose the verification type that will provide security for the webhook listener.
The simplest verification method, relying on transport-level encryption for each request.
Enter your Username and Password.
A hash function and a secret key authenticate requests, configured with the following settings:
-
Algorithm: SHA-1, SHA-256, SHA-512, or SHA-384
-
Encoding: Base64 or Hexadecimal
-
Key (secret): This is the secret key provided to you for the webhook. This is required to generate the hash value.
-
Header (containing hmac): This is the header name in your HTTP request. For example, the header name for Slack webhooks is "X-Slack-Signature". To get a signature, encode the request body using the specified algorithm and secret.
Provide a generated token and a secret temporary URL to authenticate requests in the exposed settings.
Authenticate token-based requests with a provided or generated token.
Token (required): Generate a new token or enter the token that will be sent by the webhook origin so that the Celigo platform can authenticate the request to receive data from that endpoint. Only share this token with the webhook provider.
Path (required): Enter the JSON path of the field, such as myAuth.token, in the request body that will contain the token value.
Public URL (required): If a URL has not been generated yet, click the plus icon (+) to generate a public URL for this webhook listener. You can then share this URL with the webhook provider by entering it in the provider's user interface or API. Each webhook provider has a methodology to configure a destination URL where the webhook will send data, so use the public URL as the destination URL when you are configuring the webhook in the host application.
Sample data: Click Generate sample data to populate the sample data field with the webhook data most recently received by the public URL endpoint. If the public URL endpoint has not yet received any data, you can manually enter sample JSON data in the field provided.
Note
XML is supported for webhooks, but the sample data in the webhook allows only JSON. You can convert XML to JSON in
Dev playground with the XML parser helper.The Celigo platform uses a leaky-bucket algorithm for rate limiting, with a capacity of 1,000 tokens and a refill rate of 300 tokens per second, supporting up to 1,080,000 requests per hour. Remember that each connected application may have its own rate limits.
-
Bucket size: Holds 1,000 tokens.
-
Fill rate: Adds 300 tokens per second.
This setup regulates request flow, preventing system overload.
-
Understand limits: Know both integrator.io's and external applications' rate limits.
-
Optimize requests: Minimize API calls.
-
Handle errors: Prepare for rate limit breaches.
Rate limiting is vital for smooth integrations, and Celigo's algorithm combined with an understanding of external limits ensures efficient data exchange.
You can also save mock output data to generic webhook listeners . This works the exact same way as for Mock output for Exports and Lookups , except that it is not possible to fetch preview data to populate the mock data field. Click the Expand Window () button to open and view the single pane JSON editor.
Open the Advanced section for additional webhook configuration options.
Override HTTP status code for the success responses: integrator.io returns a 204 HTTP status code for all successful webhook requests by default. Select an alternate successful HTTP status code for the listener.
Override media type for success responses: JSON is the default media type for success responses. Select an alternate media type from this list.
Override HTTP response body for success responses: The Celigo platform returns an empty response body message for all successful webhook requests by default. Use this field to customize an HTTP response body returned for successful requests made to the listener. Copy and paste the data you sent to integrator.io into the input field to configure a handlebars expression.
Page size: When an export runs in the context of a data flow (where the data from the export is sent right away to an import queue) integrator.io will break the data being exported into one or more smaller pages of records. Saying this another way, integrator.io uses streaming to export data out of one app and import it into another app. The Page size field allows you to specify how many records you want in each page of data. If you leave this field blank, the default page size is 20 records. There is no maximum record limit, but a page of data is limited to 5 MB. Most of the time, the application that you are importing data into will bottleneck the page size value. For example, if you are importing data into NetSuite or Salesforce they each specify (in their API guides) a maximum number of records that can be submitted in any single request.
Data URI template: When your flow runs but has data errors this field can be really helpful in that it allows you to make sure that all the errors in your job dashboard have a link to the original data in the export application. This field uses a handlebars template to generate the dynamic links based on the data being exported. For example, if you are exporting a customer record from Shopify, you would most likely set this field to the following value https://your-store.myshopify.com/admin/customers/{{{id}}}. Or, if you are just exporting a CSV file from an FTP site then this field could simply be one or more columns from the file: {{{internal_id}}, {{{email}}}, etc...
Do not store retry data: Check this box if you do NOT want integrator.io to store retry data for records that fail in your flow. Storing retry data can slow down your flow's overall performance if you are processing very large numbers of records that are failing. Storing retry data also means that anyone with access to your flow's dashboard can see the retry data in clear text.
Override trace key template: Define a trace key that integrator.io will use to identify a unique record. You can use a single field such as {{{field1}}} or use a handlebar expression. For example, this syntax {{join “_” field1 field2}} will generate a trace key template field1_field2. When this field is set, you will override the platform default trace key field.
Rate this featureIncluded in the June 2021 release integrator.io allows you to capture requests received by a webhook listener within a specified time period. You can then view the logs to verify that the data includes all expected information.
Note
See debug flow steps for information on debug logs for other flow step types.
Use the following steps to review data sent from a webhook:
-
Open the flow that contains a listener export step.
-
Click Listener to open the Edit listener panel.
-
Click View debug logs in the upper right corner of the Edit listener panel.
-
Click Start debug and choose a period of time for the debugger to capture data.
-
Perform an action that generates the trigger to send data to the webhook listener within the external application.
-
Select a log to review data received by the webhook listener.
Comments
10 comments
What content types will the Integrator webhook accept? Will it accept multipart/form-data?
Hi, Shane Brown. Thanks for the question, which isn't directly addressed in this article yet.
And sorry for the delay. I needed to track down the answer as well as the relative priority for adding this feature to integrator.io.
Webhooks that receive multipart/form-data are not yet supported in the platform, but we are looking at adding this format, hopefully within the year.
How many concurrent requests can a webhook handle? we are looking to have a single webhook listener exposed for a number of partners, but we would like to understand its capacity, can you please provide some details about it? thanks.
Is there a method to define a Resource Path if we are using a Webhook Listener with XML?
XML Structure is as follows and we want to make sure this is send as 2 records. If we cannot define a path of /Shipments, then this is seen as one record.
<shipments>
<shipment>
.....
</shipment>
<shipment>
.....
</shipment>
</shipments>
Hello Martin Jahn,
Our webhook exports don't yet support the selection of specific fields via resource path, but in your import step, you can access any specific field returned from the webhook export. Please let us know if you have any trouble with that solution.
Tom
As I understood, the WebHook will immediately return a response to the source app upon receiving the payload. Is there any way we can delay sending the response until the flow has been completed?
Basically, I would like to send a 200 response if the flow ran without any errors and send a 4XX/5XX response if there were any errors in the flow.
Hi Zanfer Hussain,
I'm having a little trouble understanding your request. A webhook kicks off the flow with the source data sent from the export. At that point the data begins passing through your flow. I'm not sure what you're attempting by delaying the response. Could you add more clarity?
I've set up a few listeners for various applications. A consistent issue I'm running into when testing and troubleshooting is that I cannot get sample data to populate even after the listener has already received one or many payloads and I click the button to show sample data. Is this a known bug in the application? I find it difficult to work on these without the ability to quickly see the payload being sent to the listener URL. I'd appreciate any help with this!
Hi, Chris. I asked the Flow builder Product leader for a quick read.
We are not aware of any bugs in this scenario. Since you said that the listener received the data, then the webhook must be triggering. Could you please share a screen capture here or initiate a support ticket so that we can understand the issue a little better?
Thanks Stephen. I'll open up a ticket.
Please sign in to leave a comment.