Fundamentals of HTTP connections

Simply put, the HTTP(S) protocol allows you to connect to another server. In integrator.io, you will use HTTP connections to provide credentials and perform requests to different cloud applications that expose an API based on HTTP standards.

Tip: If you are accessing a universal REST API, you are actually relying on the underlying HTTP protocol. An HTTP connection can be used for a RESTful endpoint, as well. 

Contents

• General – section settings
• Application details – section settings
• Configure authentication – section options
• Nonstandard API rate limiter – section settings
• How to test this connection? – section settings
• Advanced – section settings
• HTTP connection example

Before continuing, you must be familiar with the app’s API, including its authentication method. Refer to the online or vendor-supplied documentation.

The universal HTTP connector is by far the most flexible endpoint type with the widest variety of settings, to cover the many data transfer configurations that cloud apps employ. In most cases, you will not have to modify the optional sections in the HTTP Connection panel.

General – section settings

Several settings are available for configuring an integrator.io HTTP connection.

Name (required): Enter a unique name for your connection so that you can easily reference it from other parts of the application.

Application (required, non-editable): A reminder of the app you’re editing.

Mode (required): Two options are available for this field:

• Cloud – This is the default and most common option. It means that integrator.io servers directly talk to your API. You may need to whitelist the integrator.io IP addresses in the server’s firewall settings
• On-premise – Select when you are using an agent, a small application that you can install on your computers to connect to data that is behind your firewall.

Application details – section settings

This section is expanded by default, since it contains required settings.

Base URI (required): Enter the common part of an API’s URL, used across all of the HTTP endpoints you will invoke.

Configure HTTP headers (optional): In some cases, it may be necessary to include custom HTTP headers with your token refresh requests. integrator.io automatically adds the appropriate content-type header based on the Media type of the connection (typically, and by default application/json). Note that this request header value automatically includes the Auth type of this connection. You can reference dynamic path field names for the connection using handlebars {{{placeholders}}}.

Media type (required): This setting specifies the data format to use in the HTTP request body and response body. For example, if your API uses the JSON media type to send HTTP responses, select JSON.

Override media type for success responses (optional): Select an alternate format if successful HTTP responses return a different media type than the original HTTP request body.

Override media type for error responses (optional): Select an alternate format if unsuccessful HTTP responses return a different media type than the original HTTP request body.

Configure authentication – section options

Each Auth type (documented independently) requires unique settings for account authorization, depending on your selection:

• Basic – When this option is selected, integrator.io will include in the request header field Authorization: Basic <credentials>, where credentials is the Base64 encoding of your ID and password joined by a single colon (:).
• Token – Token-based authentication allows you to fetch a specific resource using a token – without providing a username and password – for a period of time.
• OAuth 2.0 – The OAuth 2.0 authorization framework enables integrator.io to obtain limited access to an HTTP service on behalf of an account. Scope is a mechanism in OAuth 2.0 to limit an application’s access to an account.
• Cookie – When integrator.io makes a request, it will include the cookies that contain authentication information in the request.
• Digest – When integrator.io makes an HTTP request, it will include an Authorization header for which the value contains an MD5 hash calculated using the username, password, URL, and other fields. The RFC2617 specification applies for calculating this header.
• WSSE – When integrator.io makes an HTTP request, it will include an X-WSSE header that is calculated on the basis of the Web Services Security (WSS) specification.
• Custom – More fluid option for APIs that implement custom authentication different from any of the above. 

Nonstandard API rate limiter – section settings

Most APIs limit the number of requests that can be sent for a given time period. The following settings in this section of the connection help you to define the custom rate-limit settings for connecting to your API.

Wait time between HTTP requests (optional): This field lets you set the speed at which integrator.io makes requests. In other words, how long should the platform wait before making subsequent HTTP requests calls to the API? Wait time between HTTP requests should be set if the API does not implement rate-limiting and integrator.io should manually regulate call frequency. Enter the value for this field setting in milliseconds (integers, without the unit). Note that this field can be used only when the Concurrency level is set to 1.

Override HTTP status code for rate-limit errors (optional): The HTTP specification states that rate-limit response errors should return a 429 status code. Some APIs have custom rate-limit implementations that rely on other status codes or even throttle errors within the HTTP body. Enter a value if the service you are connecting to uses a status code other than 429.

Path to rate-limit error field in HTTP response body (optional): If the service you are connecting to embeds rate limit errors within the HTTP body, provide the path (JSON path or XPATH) within the response body where integrator.io should look to identify a throttled response. If there is a specific value (or set of values) that indicates rate-limit response at this path, use the Rate-limit error values setting for further instructions on how to identify this type of error.

Rate-limit error values (optional): If you provided a path in the HTTP response body field (above), then enter the exact values that the API will return to indicate rate-limit errors. Separate multiple values with a comma.

Override retry-after HTTP response header name (optional): HTTP services commonly return status code 429 when too many requests are made in a short period of time. This code typically accompanies a retry-after HTTP header response that specifies the time to wait until a successful request can be made. If the API returns this response header with a different name, specify the custom name in this setting.

Example A

Let’s assume an API that returns the response below with status code 500 if we hit the API endpoint continuously with a lot of requests. In these cases, there is a possibility that the API we are connecting to doesn’t have rate-limiting on their side and we have to regulate the requests.

Failed request

HTTP Response Code: 500
HTTP Response Body: 
  {
    status: "Internal Server Error"
  }

So, in the above case, if we specify the value of Wait time between HTTP requests as 500, integrator.io will wait 500 ms before making subsequent HTTP requests.

Example B

Take an API which returns the following response in the case of a rate-limiting error.

Failed request

HTTP Response Code: 400
HTTP Response Body: 
  {
    status: "Throttled"
  }
HTTP Response Headers: 
  {
    try-after: "1000"
  }

In this example, enter the following values:

• Override HTTP status code for rate-limit errors: 400
• Path to rate-limit error field in HTTP response body: status
• Rate-limit error values: Throttled
• Override retry-after HTTP response header name: try-after

How to test this connection? – section settings

If you describe the connection’s test, or ping, values, then integrator.io will show you the connection’s status based on the test location’s response.

Some non-REST–based APIs return a successful HTTP status code even though the HTTP request has failed. These HTTP connect settings allow you to handle status codes that do not follow the REST standard completely (2xx status codes for successful requests and 4xx status codes for failed requests).

HTTP method (optional): Select the HTTP method to use when making the ping request.

Relative URI (optional): Enter a URI to an authenticated endpoint – relative to your Base URI, above – that integrator.io will use in an HTTP request to verify that a connection is working properly (for example: /me, /tokenInfo, or /currentTime. Whenever a connection is saved, integrator.io will send a ping request to this URI and then save this connection if the request succeeds. Click the handlebars button () to open the Advanced field editor and create or edit a template. Note that handlebars {{placeholders}} may be used to reference any connection fields.

An automated batch process runs several times per hour to ping all offline connections to try to bring them back online (and to resume any data flows that were paused as a result). When you set the Relative URI for all your custom connections, then integrator.io can do more to identify offline connections and automatically restore them.

If your API has the endpoint https://abc123.com/v1/order, enter the following values for each setting:

• Base URI: https://abc123.com/
• Ping relative URI: v1/order

HTTP request body (optional, enabled when HTTP method is POST or PUT): Enter an HTTP request body if the API does not accept the GET method for ping requests. integrator.io will send this value to the endpoint in the HTTP request body.

The format depends on the API being used: url-encoded, JSON, or XML. Click the handlebars button () to open the Advanced field editor and create or edit a template.Note that this field can contain handlebars {{{placeholders}}} that will be resolved according to your connection and export data. For example if the export request body requires an authentication token to be embedded, you could enter {{connection.http.auth.token.token}}.

Path to error field in HTTP response body (optional): Enter a value if the API always returns a successful HTTP status code, but then passes an error response within a field in the HTTP response body. For example, if the API always returns a 200 (success) HTTP status code and indicates failure via an error.message field in the HTTP response body, then enter error.message.

Error values (optional): If you provided a path in the HTTP response body field (above), then enter the exact values that the API will return to indicate a failed ping response. Separate multiple values with a comma.

Path to success field in HTTP response body (optional): Enter a value if the API always returns a successful HTTP status code, but then passes a successful response within a field in the HTTP response body. For example, if the API always returns a 200 (success) HTTP status code and indicates success via a success Boolean field in the HTTP response body, then enter success.

Success values (optional): If you provided a path in the HTTP response body field (above), then enter the exact values that the API will return to indicate a successful ping response. Separate multiple values with a comma.

Path to detailed error message field in HTTP response body (optional): Provide a field only if the API returns a field in the HTTP response body to indicate ping response errors.

Any detailed error message will be displayed on the error management dashboard. If this field is not set, then the full HTTP response body will be used as the error message on the dashboard.

Example A

Let’s take an API that returns the following responses for successful (Path to success field in HTTP response body and Success values) and failed requests.

Successful request

HTTP Response Code: 200
HTTP Response Body:
  {
    status: "success"
  }

Failed request

HTTP Response Code: 200
HTTP Response Body:
  {
    status: "failed"
  }

In this case, provide the following values:

• Path to success field in HTTP response body: status
• Success values: success

Example B

Now, consider an API that returns the following responses for successful and failed requests (Path to error field in HTTP response body, Error values, and Path to detailed error message field in HTTP response body).

Successful request (empty response body)

HTTP Response Code: 200
HTTP Response Body:
  {}

Failed request

HTTP Response Code: 200
HTTP Response Body:
  {
    Error: "Request_Failed",
    "Error_Description": "Service is unavailable. Please try after some time."
  }

For the example above, provide the following values:

• Path to error field in HTTP response body: Error
• Error values: Request_Failed
• Path to detailed error message field in HTTP response body: Error_Description

Advanced – section settings

Disable strict SSL (optional): An optional flag to skip verifying the SSL certificate, allowing self-signed or expired certificates. It is highly recommended that you never set this flag to override encryption for any production data connections. Disable at your own risk.

Borrow concurrency from (optional): If you have multiple connections to the same API, you can set the related connections to have a shared concurrency level using this field. This field is applicable only when you have the field Concurrency level set for one connection.

Concurrency level (optional): Set this field to limit the number of concurrent HTTP requests allowed by the connection resource at any one time, or leave this field blank to use burst mode. With burst mode, integrator.io will make HTTP requests as fast as possible, with very high levels of concurrency.

Some APIs respond well with burst mode and can typically handle high volumes. Conversely, other APIs are much stricter when it comes to the number of API requests being sent to their servers, and burst mode may not be recommended.

Also, every connection in integrator.io is mapped to a first-in/first-out (FIFO) queue, from which messages are processed to complete the flow execution. If you set Concurrency level to 5, at most 5 simultaneous messages are processed from the queue. See Example B below for practical guidance.

SSL client key (optional): Enter the private key associated with the certificate provided for the SSL client certificate. Typically, any SSL certificate is associated with a private key, which is used for encrypting/decrypting the data in SSL communication. This value is expected to be in PEM format.

SSL certificate (optional): Some APIs use “certificate-based authentication,” in which a digital certificate is used to identify a user, machine, or device before granting access to a resource, network, or application. If your API requires that a certificate be sent while making a HTTP request, enter the certificate in PEM format.

SSL passphrase (optional): In cases where the private key is encrypted, enter the passphrase, which is used to decrypt the private key. The key can be entered in plain text.

Encrypted (optional): Use this encrypted JSON setting to store all the security-sensitive fields needed by your imports and exports (to access the application being integrated). For example, {'password': 'ayTb53Img!do'} or {'token': 'x7ygd4njlwerf63nhg'}.

Unencrypted (optional): Use this JSON field to store all the non-security–sensitive fields needed by your imports and exports (to access the application being integrated). For example: {'email':'my_email@company.com', 'accountId': '5765432', 'role': 'admin'}.

Concurrency example A

If your API allows a maximum of 10 parallel HTTP requests at once, then set the Concurrency level to 10, which will ensure that integrator.io makes no more than 10 parallel requests.

Concurrency example B

Let’s say that you have two flows – Flow1 and Flow2 – in your integration with the following configurations. Note that both flows’ imports are using the same connection object – Connection1 – with has a Concurrency level of 10.

Flow	Export	Import and connection	Concurrency level
Flow1	Export1	Import1, using Connection1	10 (Connection1)
Flow2	Export2	Import2, using Connection1	10 (Connection1)

If Flow1 is a high-volume flow, having a lot of records to be synced, then it might stall the execution of Flow2 altogether.

If Flow2’s execution is important and you want to have some concurrency always available to Flow2, then you can have a second connection – Connection2 – with a Concurrency level of 5, while reducing the Concurrency level of Connection1 to 5. This configuration ensures that Connection1 and Connection2 are processed in different queues, which makes the flows (Flow1 and Flow2) run independently, as in the table below:

Flow	Export	Import and connection	Concurrency level
Flow1	Export1	Import1, using Connection1	5 (Connection1)
Flow2	Export2	Import2, using Connection2	5 (Connection2)

Concurrency example C

Imagine three connections to the same API with the names shown below:

Connection	Access level	Flows	Integration
Connection1	Full access to the API	Flow1	Integration1
Connection2	Partial access (example: Orders API)	Flow2	Integration2
Connection3	Partial access (example: Payment API)	Flow3	Integration3

As you can see in the above connections, Connection1 has full access to the API, while the other two connections have lower access. In this case, if we want to invite any users to manage Integration2 and Integration3 and maintain the maximum of 2 parallel HTTP requests from all 3 connections, then we need to set shared concurrency level to 2 for all connections. That is, we want to have a maximum of only 2 simultaneous connections to the API from these connections. The best solution is to modify the advanced settings as follows:

• Set Concurrency level for Connection1 to 2
• Set Borrow concurrency from for Connection2 and Connection3 to point to Connection1

HTTP connection example

Follow the steps below to create an HTTP connection, to the Zendesk API in this example. 

1. Select Connections from the Resources menu. 
2. Click + Create connection at the top right corner of the screen.
3. Under Application, choose HTTP from the Universal connectors section. 
4. Enter the sample values provided for each field:
   • Name: Zendesk HTTP connection
   • Mode: Cloud
   • Auth type: Token
   • Base URI: https://<my subdomain>.zendesk.com/api/v2 
   • Media type: JSON
   • Token: <authorization token in Zendesk control panel>
   • Send token via: HTTP header
     • Header name: Authorization (the default header name, which is applicable in the case of Zendesk
     • Header scheme: Bearer (the default value, which is applicable in the case of Zendesk
   • How to test connection?
     • Relative URI: /organizations
     • HTTP method: GET
5. Click Test connection to see if your connection is working properly. You should see the following message:

6. Once you have tested the connection, save and close it.