This article explains how to create an export for an HTTP connection. While it applies to other HTTP connection types, the export described below uses an API with token-based authentication.
Contents
Create the HTTP connection
- Sign in to integrator.io.
- From the Tools & resources menu, select Flow Builder.
- Click the Source application search bar, and select HTTP from the list of connection types. (The list filters according to the text you enter).
- To the right of the Connection setting, click the + icon. The Create connection pane opens.
Name the connection and choose its mode
Note: For on-premise mode, you must first install and configure an agent. If the system you want to connect to is behind a firewall, the agent allows you to connect without whitelisting any IP addresses.
Name (required): Give your connection a name that is meaningful to your integration and that can help you identify it later in a list of connections.
Mode (required): Choose one of the following:
- Cloud – Connect to an app on the cloud.
- On-premise – An on-premise app runs locally on the company’s servers. This mode requires an integrator.io agent. From the Agent drop-down menu, select an installed agent.
Describe the connection
Continuing in the Establish your HTTP connection dialog, provide information about the token.
Configure HTTP headers (optional): Click this button to pass additional parameters in the header. Configure any headers needed for your integration, providing the name-value pairs.
Token (required): Enter your API token. Multiple layers of protection (including AES 256 encryption) are in place to secure your API token. For a list of supported encryption types, see Supported encoding algorithms.
Base URI (required): Enter the path, as documented in the source app’s API guide.
Media type (required): Choose the format for your records.
Specify how to send the token
Next, the Establish your REST API connection dialog asks how to send the token. Again refer to the API guide for the service you want to connect to, in order to specify where the token should go.
The How to send token? options differ, depending on whether you select HTTP Header or URL parameter for the Location field.
Put the token in the header
Header name (optional): By default, integrator.io will send all authentication info in the Authorization: HTTP header field. If the REST API you are connecting to requires a different HTTP header, enter just the name here, without the colon (:).
Scheme (optional): Select an authentication scheme, per the API documentation.
Configure token refresh (optional): Check this box if your token expires after a period of time. A series of new fields will appear in the form:
Refresh token URI (optional): Enter the relative URI for refreshing the token.
Refresh token media type (optional): Select a data format.
Refresh token method (optional): Select the HTTP method to use.
Refresh token path (optional): Specify the path to the token, if the refresh token will be in the body of the response.
Refresh token headers (optional): Click to set custom headers for refreshing your token. The editor works like the Configure HTTP headers editor, above.
Include the token in the URL parameter
Parameter name (required): Specify the name of the URL parameter that will hold the API token value. For example, if you specify myAPITokenURLParam, then all HTTP requests will include the following value in the URL:
?myAPITokenURLParam=bearerToken
Include the token in the body
Configure token refresh (optional): Check this box if your token expires after a period of time. A series of new fields will appear in the form:
Refresh token URI (optional): Enter the relative URI for refreshing the token.
Refresh token media type (optional): Select a data format.
Refresh token method (optional): Select the HTTP method to use.
Refresh token path (optional): Specify the path to the token, if the refresh token will be in the body of the response.
Refresh token headers (optional): Click to set custom headers for refreshing your token. The editor works like the Configure HTTP headers editor, above.
Test and save the connection
Ping method (optional): Select the HTTP method to use when making the ping request.
Ping URI (optional): Enter the relative URI to a specific resource, as documented in your app’s API guide. (The ping URI is relative to the base URI.)
Ping success path (optional): Provide the location of custom error codes, as documented in the error schema in your source app’s API guide.
A success path is necessary only if your app returns errors outside of the standard 4xx and 5xx status codes. For example, Slack sets a field in the response body and returns a 200 HTTP status code, whether or not the ping HTTP request failed. In this case, in Ping success path specify the JSON path of a field in the response body that should instead be used to determine if a ping request succeeded. For example, if you are building a connection for Slack’s API, set this field to ok (see Slack API docs for more info).
Ping success values (optional): Enter the values to test whether a connection succeeded. This optional field is used in conjunction with the Ping success path field. The value found in the HTTP response at the path provided is compared against this list of success values. If there is an exact case-sensitive match of any of the values, then the request is considered successful.
Ping fail path: If the service you are connecting to embeds authentication errors within the HTTP body, use this field to set the path within the response body where integrator.io should look to identify a failed auth response. If there is a specific value (or set of values) that indicate a failed auth response at this path, use the Ping fail values field to further instruct our platform on how to identify this type of error.
Ping fail values: Use this field only if the Ping fail path field is set. It indicates to integrator.io which values to test to determine if the requests failed for authentication reasons.
Ping error path: Use this optional field if the response from a failed ping request is large and only part of the response should return as the reason for a failed ping response. If no value is given, then the full HTTP response is used as a description of the failure. If the media-type of the failed response is XML, this value should be an XPATH. Conversely, if the media-type is JSON, then use a JSON path. Note that if failed responses to ping requests have no body, then a text version of the HTTP status code returns as the reason for failure.
Click Test to try connecting before you save these values. If the connection fails, double-check the provided settings, and test again.
Create source
- Click Next and configure the source (export):
- Enter the source name and provide the HTTP action that the external API supports (like GET, PUT, and POST) In some cases, you must include custom HTTP headers with your API requests; in such cases, provide header information in the form. Also, enter the relative URI from which integrator.io can export the data:
- Use the advanced field editor to create the relative URL where you can use various handlebar expressions and configure the URL based on the parameter that needs to be passed in the API request call:
Note: See Handlebars library for a complete list of options.
- If you need to pass anything in the request body, click Build HTTP request body, and provide your custom values in the editor:
- Enter the success media type based on the response from the API (check the source app’s API documentation for this value). Currently, integrator.io supports XML, CSV, and JSON media types.
Error media type
Select the error media type as JSON or XML. The error media type indicates the format of the response data received from all unsuccessful (HTTP) requests. This refers to all responses with non-2xx HTTP status codes. Most APIs support one one media type (data format), as documented in their API guides. This is an optional override of the media type associated with the connection in case the API has route-specific data formats. Amazon MWS, for example, has endpoints that return CSV data, but sends error responses in XML.
Resource path
This optional field allows integrator.io to locate the resource (or set of resources) returned from an API call. If the HTTP response from an API contains the resource(s) at the root, then no value is necessary for this field, but if the response from an API contains a deeper response structure that contains paging information, you must provide the path to the resource(s). If no value is provided, integrator.io assumes that the resources can be found at the root of the response. Note that the format of this path is specific to the media type. If the response is represented in JSON structure, then use "dot" notation to describe the path to your resources.
Example JSON path
results.customers
Example XPATH for XML
/SearchResults/Customers/Customer
Success path
Not all services return HTTP status codes to determine the success or failure of a request. If a service always returns 200 HTTP status code, even for some failures, then use this field to tell integrator.io where to look for a success indicator. This field is used in combination with the Success values field to further refine how integrator.io can identify a successful response. If no success values are provided, any value found in the response body at this path is considered a success.
Success values
Use this optional field along with the Success path field. The value found in the HTTP response at the path provided by Success path is compared against the provided list of success values. If there is an exact case-sensitive match of any of the values, then the request is considered successful.
Fail path
If the service you are connecting to embeds authentication errors within the HTTP body, use this field to set the path within the response body where integrator.io should look to identify a failed auth response. If there is a specific value (or set of values) that indicate a failed auth response at this path, use the Fail values field for further instructions on how to identify this type of error.
Fail values
Use this field only if Fail path is set. It indicates to integrator.io which values to test when determining if the requests failed for authentication reasons.
Error path
Use this optional field to help identify where in the body of a failed HTTP response integrator.io can find the error message. You can provide the field path to the property/node containing the error message. If no value is given, then the full HTTP response body is used as the description of the failure in the dashboard. Use XPATH if the media type of the failed response is XML, and use JSON if the media type is JSON. Note that if failed responses from an application have no body, then a text version of the HTTP status code is used as the reason for failure.
Example JSON path
result.error.message
Example XPATH for XML
/result/error.message/text()
Export type
The following export types are available to help support common integration patterns:
- All: Exports all data, always.
- Delta: Only exports data that has changed since the last time the data flow was run.
- Once: Only exports data that has not been exported already, and also automatically updates records to mark them as exported.
- Test: Only exports one record by default, and is primarily used for testing to avoid syncing lots of data.
Once you have configured all of the export settings, click Preview to review the response from the API call. If you see any errors, then revisit the steps above to fix the problems.
Paging method
Some APIs offer paging functionality in order to limit their HTTP response to a manageable size if the total set of resources is large. If the API supports pagination then you can configure the Does this API supports paging section. It means that instead of providing data in one go, it will provide it in chunks which integrator.io will handle
integrator.io supports the following paging methods:
- Page: Choose page if the API uses a page number parameter to paginate the results.
- Token: Choose token if the API returns a token to the next page within the response body.
- Skip: Choose skip if the API uses a skip-take paging mechanism. Other names for this paging method are offset and limit.
Page method
Use this optional field when the paging method is set to page. It only needs to be set with an initial value if the export should start at a page offset. This value automatically increments with each subsequent page request. The value can be referenced as a placeholder in the body, headers, or relative URI of the export request.
Example
/products?page={{{export.http.paging.page}}}
Max page path
Some APIs return the total number of pages available in each of their page responses. Use this optional field as a trigger to stop making page requests. This field tells integrator.io where to find the maximum page value within each page response. When this page count is met, integrator.io stops making page requests. If omitted, integrator.io continues to make requests until no resources return or a 404 (not found) response occurs.
Max count path
Some APIs return the total number of resources available in each of their page responses. Use this optional field as a trigger to stop making page requests. This field tells integrator.io where to find the total resource count within each page response. When this resource count is met, integrator.io stops making page requests. If omitted, integrator.io continues to make requests until no resources return or a 404 (not found) response occurs.
Last page status code
Some APIs return an error status code (less than 200 or greater than 300) as the last page response. Use this optional field to tell integrator.io the response code to expect in these situations and not to treat this response as an error. For example, if an API returns a generic 500 status code, set this value to 500. Use this field in conjunction with Last page path and Last page value. Note that integrator.io automatically treats error 404 as the last page response, so it is not necessary to set this field for that case.
Last page path
If Last page status code is set, then you can set this field to tell integrator.io to check a specific field in the response object to determine if paging is complete. For example, the specified field can be "error.message". If the Last page path field is not set, then paging will be considered complete; otherwise, this response is considered an error.
Last Page Values
If the Last Page Status Code and Last Page Path fields are both set, then you can set this field to make integrator.io check a specific field’s value in the response object to determine if paging is complete. If the value in the response object matches this field (such as No records found), then paging will be considered complete; otherwise, this response is considered an error.
Token method
Use this optional field when the paging method is set to Token. It needs to be set only if the initial export should contain a token. This value is then automatically set on each subsequent page request. You can reference the value as a placeholder in the body, headers, or relative URI of the export request.
Example
/products?token={{{export.http.paging.token}}}
Path
If the paging method is set to token, integrator.io needs to be able to find the token in the HTTP response of each page in order to construct the next page request. This field lets integrator.io know where to look for the next page token. If no value is found at this path, then paging is terminated (final page reached). When the paging method is set to Next page URL, integrator.io uses this Path field to find where in the HTTP response to look to find a URL to use when requesting the next page of data. The format of the path is dependent on the media type (content type) of the original (first) page response.
Example JSON response path
result.paging.nextPageUrl
Example XPATH for XML path
/result/paging/nextPageUrl
Paging terminates when no URL is found at the specified path.
Relative URI
Use this optional field in combination with any paging method. In some cases subsequent page requests need to use a different relative URI from the first page request.
Token path after first request
Use this optional field in combination with any paging method. In some cases, subsequent page requests need to use a different relative URI than the one used for the first page request.
Resource path
This is an optional override to the resource path if subsequent page responses have a different response structure. Note that the format of this path is specific to the media type.
Example JSON resource path
results.customers
Example XPATH for XML resource path
/SearchResults/Customers/Customer
Skip paging method
Use this optional field when the paging method for the export needs to start at an offset from the first record. This value is automatically set on each subsequent page request. You can reference the value as a placeholder in the body, headers, or relative URI of the export request.
Example
/products?offset={{{export.http.paging.skip}}}
Link header relation method
When the paging method is set to Link header, integrator.io uses HTTP conventions to look for the next page URL within a dedicated link header value. You can include multiple URLs in the link header to facilitate page navigation (next, previous, first, or last page). In cases where multiple values are found, integrator.io uses the following conventions for these relation values:
- prev
- next (default)
- last
- first
Some APIs that use this paging mechanism may not comply with these defaults. In such cases, this field allows you to override the default next relation to a value used by the application you are connecting too.
Relative URI
Use this optional field in combination with any paging method. In some cases subsequent page requests need to use a different relative URI from the first page request.
The following image displays the pagination section for this example:
The API responds back with the current page (starting from 1) and the total number of pages via the total_pages property in the JSON.
Once the above conditions are met, integrator.io stops exporting data from the API.
Configure the Advanced section
Page size
When an export runs in the context of a data flow (meaning integrator.io immediately sends the data from the export to an import queue), integrator.io breaks the exported data into one or more smaller pages of records. You can use the Page size field to specify how many records you want in each page of data. If you leave this field blank, the default system value is 20. There is no maximum value, but a page of data is automatically capped when it exceeds 5 MB. Most of the time, the application that you are importing data into will bottleneck the page size value. For example, NetSuite and Salesforce both document a maximum number of records that can be submitted in any single request, upon import.
Override data URL template
When your flow runs but has data errors, this field allows you to verify that all the errors in your job dashboard have a link to the original data in the export application. This field uses a handlebars template to generate the dynamic links based on the data being exported.
For example, if you are exporting a customer record from Shopify, you would typically set this field to the following value:
https://your-store.myshopify.com/admin/customers/{{{id}}}
Or, if you are just exporting a CSV file from an FTP site, then this field could be one or more columns from the file:
{{{internal_id}}, {{{email}}}
Do not store retry data
Check this if you do not want integrator.io to store retry data for records that fail in your flow. You should do this if you have sensitive data that you don’t want stored on our platform. You should also check this box if you are exporting large data sets. Storing the retry data for large data sets isn’t necessary because the flow is idempotent and storing retry data slows the process.
Once you have completed these settings, click Save.
Comments
0 comments
Please sign in to leave a comment.