Fundamentals of HTTP connections

Simply put, the HTTP(S) protocol allows you to connect to another server. In integrator.io, you will use HTTP connections to provide credentials and perform requests to different cloud applications that expose an API based on HTTP standards.

Tip: If you are accessing a universal REST API, you are actually relying on the underlying HTTP protocol. An HTTP connection can be used for a RESTful endpoint, as well. 

Contents

• integrator.io connection options
  • Available fields
  • Example
• Refresh token functionality
• Success and failure paths
  • Ping requests
  • All export and import requests
• Concurrency and rate-limiting
  • Concurrency
  • Rate-limiting
• SSL configuration

integrator.io connection options

Before continuing, you must be familiar with the app’s API, including its authentication method. Refer to the online or vendor-supplied documentation.

The universal HTTP connector is by far the most flexible endpoint type with the widest variety of settings, to cover the many data transfer configurations that cloud apps employ. In most cases, you will not have to modify the optional sections in the HTTP Connection pane. 

Available fields

Several settings are available for configuring an integrator.io HTTP connection.

Name

Enter a unique name for your connection so that you can easily reference it from other parts of the application.

Mode

Two options are available for this field:

• Cloud – This is the default and most common option. It means that integrator.io servers directly talk to your API. You may need to whitelist the integrator.io IP addresses in the server’s firewall settings
• On-premise – Select when you are using an agent, a small application that you can install on your computers to connect to data that is behind your firewall.

Authentication type

• Basic – When this option is selected, integrator.io will include in the request header field Authorization: Basic <credentials>, where credentials is the Base64 encoding of your ID and password joined by a single colon (:).
• Token – Token-based authentication allows you to fetch a specific resource using a token – without providing a username and password – for a period of time.
• OAuth 2.0 – The OAuth 2.0 authorization framework enables integrator.io to obtain limited access to an HTTP service on behalf of an account. Scope is a mechanism in OAuth 2.0 to limit an application’s access to an account.
• Cookie – When integrator.io makes a request, it will include the cookies that contain authentication information in the request.
• Digest – When integrator.io makes an HTTP request, it will include an Authorization header for which the value contains an MD5 hash calculated using the username, password, URL, and other fields. The RFC2617 specification applies for calculating this header.
• WSSE – When integrator.io makes an HTTP request, it will include an X-WSSE header that is calculated on the basis of the Web Services Security (WSS) specification.
• Custom – More fluid option for APIs that implement custom authentication different from any of the above. 

Configure HTTP headers

Add any additional HTTP headers that need to be sent to the API during the connection.

Authentication fail...status code, path, and values

These settings are explained below, in All export and import requests. 

Base URI

Enter the common part of an API’s URL, used across all of the HTTP endpoints you will invoke.

Media type

This setting specifies the media type required for your API. For example, if your API uses the JSON media type to send HTTP responses, select JSON.

Encrypted

Use this encrypted JSON setting to store all the security-sensitive fields needed by your imports and exports (to access the application being integrated). For example, {'password': 'ayTb53Img!do'} or {'token': 'x7ygd4njlwerf63nhg'}.

Unencrypted

Use this JSON field to store all the non-security–sensitive fields needed by your imports and exports (to access the application being integrated). For example: {'email':'my_email@company.com', 'accountId': '5765432', 'role': 'admin'}

API rate limits

The settings under Nonstandard API rate limiter are described below, in Rate-limiting. 

How to test connection?

• Ping relative URI – Provide the path relative to the URI provided in the Base URI, above.
  If your API has the endpoint https://abc123.com/v1/order, enter the following values for each setting:
  • Base URI: https://abc123.com/
  • Ping relative URI: v1/order
• Ping method – The HTTP method (such as GET and PUT) to use when making the ping request.
• Ping...success paths, success values, fail path, fail values, and error path – See Ping requests, below.

Advanced

• Borrow concurrency from and Concurrency level – See Concurrency, below.
• Disable strict SSL, SSL client certificate, SSL client key, and SSL passphrase – See SSL configuration, below.

Example

Follow the steps below to create an HTTP connection, to the Zendesk API in this example. 

1. Select Connections from the Resources menu. 
2. Click + Create connection at the top right corner of the screen.
3. Under Application, choose HTTP from the Universal connectors section. 

4. Enter the sample values provided for each field:
   • Name: Zendesk HTTP connection
   • Mode: Cloud
   • Authentication type: Token
   • Base URI: https://<my subdomain>.zendesk.com/api/v2 
   • Media type: JSON
   • Token: <authorization token in Zendesk control panel>
   • How to send token?
     • Location: Header
     • Header name: Authorization (the default header name, which is applicable in the case of Zendesk
     • Scheme: Bearer (the default value, which is applicable in the case of Zendesk
   • How to test connection?
     • Ping relative URI: /organizations
     • Ping method: GET

5. Click Test to see if your connection is working properly. You should see the following message:

6. Once you have tested the connection, save and close it.

Refresh token functionality

Refresh tokens apply only when connecting to APIs that have token-based authentication (TBA) and the access tokens expire after a period of time.

Some cloud applications like squareup have access tokens that expire. In these cases, How to refresh token? metadata can be provided to tell integrator.io how to refresh the token whenever it expires. This section is visible only when Configure refresh token is checked.

How to refresh token? contains the following settings:

Refresh token

Token to be used in the HTTP request options to get the new access token, whenever it expires. This value is provided by the cloud app you are connecting to.

Refresh relative URI

Endpoint to reach when refreshing the access token. This URL should be provided in the API documentation.

Refresh media type

HTTP request media type to be used while making refresh token request. Select from the list of possible values: URLENCODED, JSON, or XML.

Refresh method

Select HTTP method to be used while making the refresh token request: GET, POST, or PUT.

Refresh token path

Enter the JSON path or XPATH to extract the new/refreshed access token in the HTTP response body.

Refresh token headers

Provide the HTTP headers to be sent in the refresh token request.

Success and failure paths

Some non-REST–based APIs return a successful HTTP status code even though the HTTP request has failed. These HTTP connect settings allow you to handle status codes that do not follow the REST standard completely (2xx status codes for successful requests and 4xx status codes for failed requests).

Ping requests

This configuration is applicable to the ping requests made for your HTTP connection.

Ping success path

Provide the JSON path or XPath appropriate for your Media type. This path is used to extract the value from the ping HTTP response to determine whether the request has been successful.

Ping success values

Provide the expected success values for corresponding to the Ping success path.

Ping fail path

Provide the JSON path or XPath appropriate for your Media type. This path is used to extract the value from the ping HTTP response to decide whether the HTTP request has failed.

Ping fail values

Provide the expected success values for corresponding to the Ping fail path.

Ping error path

Provide the JSON path or XPath appropriate for your Media type. This path is used to extract the error message in case of failed requests from the HTTP response.

Example A

Let’s take an API that returns the following responses for successful (Ping success path and Ping success values) and failed requests.

Successful request

HTTP Response Code: 200
HTTP Response Body:
  {
    status: "success"
  }

Failed request

HTTP Response Code: 200
HTTP Response Body:
  {
    status: "failed"
  }

In this case, provide the following values:

• Ping success path: status
• Ping success values: success

Example B

Now, consider an API that returns the following responses for successful and failed requests (Ping fail path, Ping fail values, and Ping error path).

Successful request (empty response body)

HTTP Response Code: 200
HTTP Response Body:
  {}

Failed request

HTTP Response Code: 200
HTTP Response Body:
  {
    Error: "Request_Failed",
    "Error_Description": "Service is unavailable. Please try after some time."
  }

For the example above, provide the following values:

• Ping fail path: Error
• Ping fail values: Request_Failed
• Ping error path: Error_Description

All export and import requests

This configuration is applicable to all export and import requests made through an HTTP connection.

In the case of failed HTTP requests, there might be a lot of reasons for the failure. To isolate whether the request has failed due to an authentication error, configure the Authentication fail... settings in the connection’s Application details section. For example, you may need to refresh the expired access tokens for some APIs.

Authentication fail status code

Provide the HTTP status code that indicates the request has failed due to an authentication error.

Authentication fail path

Provide the JSON path or XPath appropriate for your Media type. This path is used to extract the value from the HTTP response.

Authentication fail values

Provide the expected success values corresponding to the Authentication fail path.

Example B

Picture an API that returns the HTTP status code 200 for both successful and failed requests. And, for the failed requests, it returns the reason for the error in the HTTP response body.

Successful request

HTTP Response Code: 200
HTTP Response Body:
  {
    status: "success"
  }

Failed request due to authentication

HTTP Response Code: 200
HTTP Response Body:
  {
    status: "failed",
    reason: "Authentication"
  }

Failed request due to unavailable service

HTTP Response Code: 200
HTTP Response Body:
  {
    status: "failed",
    reason: "Service_Unavailable"
  }

In this example, provide the following values:

• Authentication fail status code: 200
• Authentication fail path: reason
• Authentication fail values: Authentication 

Concurrency and rate-limiting

Concurrency

Concurrency is set in the connection’s Advanced section in two settings: Concurrency level and Borrow concurrency from.

Concurrency level

Set this field to limit the number of concurrent HTTP requests allowed by the connection resource at any one time, or leave this field blank to use burst mode. With burst mode, integrator.io will make HTTP requests as fast as possible, with very high levels of concurrency.

Some APIs respond well with burst mode and can typically handle high volumes. Conversely, other APIs are much stricter when it comes to the number of API requests being sent to their servers, and burst mode may not be recommended.

Also, every connection in integrator.io is mapped to a first-in/first-out (FIFO) queue, from which messages are processed to complete the flow execution. If you set Concurrency level to 5, at most 5 simultaneous messages are processed from the queue. See Example B below for practical guidance.

Example A

If your API allows a maximum of 10 parallel HTTP requests at once, then set the Concurrency level to 10, which will ensure that integrator.io makes no more than 10 parallel requests.

Example B

Let’s say that you have two flows – Flow1 and Flow2 – in your integration with the following configurations. Note that both flows’ imports are using the same connection object – Connection1 – with has a Concurrency level of 10.

Flow	Export	Import and connection	Concurrency level
Flow1	Export1	Import1, using Connection1	10 (Connection1)
Flow2	Export2	Import2, using Connection1	10 (Connection1)

If Flow1 is a high-volume flow, having a lot of records to be synced, then it might stall the execution of Flow2 altogether.

If Flow2’s execution is important and you want to have some concurrency always available to Flow2, then you can have a second connection – Connection2 – with a Concurrency level of 5, while reducing the Concurrency level of Connection1 to 5. This configuration ensures that Connection1 and Connection2 are processed in different queues, which makes the flows (Flow1 and Flow2) run independently, as in the table below:

Flow	Export	Import and connection	Concurrency level
Flow1	Export1	Import1, using Connection1	5 (Connection1)
Flow2	Export2	Import2, using Connection2	5 (Connection2)

Borrow concurrency from

If you have multiple connections to the same API, you can set the related connections to have a shared concurrency level using this field. This field is applicable only when you have the field Concurrency level set for one connection.

Example

Imagine three connections to the same API with the names shown below:

Connection	Access level	Flows	Integration
Connection1	Full access to the API	Flow1	Integration1
Connection2	Partial access (example: Orders API)	Flow2	Integration2
Connection3	Partial access (example: Payment API)	Flow3	Integration3

As you can see in the above connections, Connection1 has full access to the API, while the other two connections have lower access. In this case, if we want to invite any users to manage Integration2 and Integration3 and maintain the maximum of 2 parallel HTTP requests from all 3 connections, then we need to set shared concurrency level to 2 for all connections. That is, we want to have a maximum of only 2 simultaneous connections to the API from these connections. The best solution is to modify the advanced settings as follows:

• Set Concurrency level for Connection1 to 2
• Set Borrow concurrency from for Connection2 and Connection3 to point to Connection1

Rate-limiting

Most APIs limit the number of requests that can be sent for a given time period. The following settings in the Nonstandard API rate limiter section of the connection help you to define the custom rate-limit settings for connecting to your API. 

Wait time between HTTP requests

This field lets you set the speed at which integrator.io makes requests. In other words, how long should the platform wait before making subsequent HTTP requests calls to the API?

Wait time between HTTP requests should be set if the API does not implement rate-limiting and integrator.io should manually regulate call frequency. Enter the value for this field setting in milliseconds (integers, without the unit). Note that this field can be used only when the Concurrency level is set to 1.

HTTP status code for rate-limit errors

The HTTP specification states that rate-limit response errors should return a 429 status code. Some APIs have custom rate-limit implementations that rely on other status codes or even throttle errors within the HTTP body. Use this field if the service you are connecting to uses a status code other than 429.

Path to rate-limit errors in HTTP response body

If the service you are connecting to embeds rate limit errors within the HTTP body, provide the path (JSON path or XPATH) within the response body where integrator.io should look to identify a throttled response. If there is a specific value (or set of values) that indicates rate-limit response at this path, use the Rate-limit error values setting for further instructions on how to identify this type of error.

Rate-limit error values

This setting is used only if Path to rate-limit errors in HTTP response body is set. It indicates to integrator.io which specific values to test when determining if the requests have been rate-limited.

Retry-after HTTP response header name

It is common for an HTTP service to return status code 429 if too many requests are made in a short period of time. These 429 responses are typically paired with a retry-after response HTTP header that specifies the time to wait until a successful request can be made. Some services send this response header with a different name, and you can specify this custom name in this field. 

Example A

Let’s assume an API that returns the response below with status code 500 if we hit the API endpoint continuously with a lot of requests. In these cases, there is a possibility that the API we are connecting to doesn’t have rate-limiting on their side and we have to regulate the requests.

Failed request

HTTP Response Code: 500
HTTP Response Body: 
  {
    status: "Internal Server Error"
  }

So, in the above case, if we specify the value of Wait time between HTTP requests as 500, integrator.io will wait 500 ms before making subsequent HTTP requests.

Example B

Let’s take an API which returns the following response in the case of a rate-limiting error.

Failed request

HTTP Response Code: 400
HTTP Response Body: 
  {
    status: "Throttled"
  }
HTTP Response Headers: 
  {
    try-after: "1000"
  }

In this example, enter the following values:

• HTTP status code for rate-limit errors: 400
• Path to rate-limit errors in HTTP response body: status
• Rate-limit error values: Throttled
• Retry-after HTTP response header name: try-after

SSL configuration

Disable strict SSL

An optional flag to skip verifying the SSL certificate, allowing self-signed or expired certificates. It is highly recommended that you never set this flag to override encryption for any production data connections. Disable at your own risk.

SSL certificate

Some APIs use “certificate-based authentication,” in which a digital certificate is used to identify a user, machine, or device before granting access to a resource, network, or application. If your API requires that a certificate be sent while making a HTTP request, enter the certificate in PEM format.

SSL client key

Enter the private key associated with the certificate provided for the SSL client certificate. Typically, any SSL certificate is associated with a private key, which is used for encrypting/decrypting the data in SSL communication. This value is expected to be in PEM format.

SSL passphrase

In cases where the private key is encrypted, enter the passphrase, which is used to decrypt the private key. The key can be entered in plain text or PFX format.