This article describes the options available for exporting data from an HTTP source application.
Contents
- Create the export
- Configure HTTP authentication settings
- What would you like to export
- Configure export type
- Does this API support paging?
- Preview data
- Non-standard API response patterns
- Advanced
Create the export
- Sign in to integrator.io.
- From the Tools menu, select Flow Builder.
- Click Add source in the Sources section of the new flow.
- The Create source pane opens.
- From the Application list, select HTTP from the list of connection types.
- After the HTTP Application is added, the Create source pane prompts you for additional information about your export and connection settings. Select an existing connection or click the + button to create one.
Configure HTTP authentication settings
Choose one of the following authentication types:
- Basic: Select Basic if your service implements the HTTP basic authentication strategy. This authentication method adds a Base64-encoded username and password values in the authentication HTTP request header.
- Cookie: Select Cookie if your service relies on session-based authentication. Session-based authentication is typically implemented by including a unique cookie into the HTTP request header. By selecting this option, the platform will automatically create and insert this cookie into every HTTP request it sends to your application.
- Custom: Select Custom for all other types. If you select the Custom authentication method, integrator.io will not perform any special authentication; you can freely configure the HTTP request fields (method, relativeUri, headers, and body) of the import and export models to include {{placeholders}} for any authentication values stored in Encrypted and Unencrypted fields of this connection.
- Digest: Select Digest if your service relies on digest authentication. With “Digest” auth, the client sends a first request to the API, and the server responds with a few details, including a number that can be used only once (nonce), a realm value, and a 401 unauthorized response. An encrypted array of data including username and password combined with the data received from the server in the first request is then sent back. The server uses the passed data to generate an encrypted string and compares it against what is sent in the previous step to authenticate requests.
- Token: Select Token if your service relies on token-based authentication. The token may exist in the header, URL, or body of the HTTP request. This method also supports refreshing tokens if the API supports it.
- OAuth 2.0: You can select this option to connect to any OAuth 2.0-compliant app that does not have a standard integrator.io connection, or you can build your own OAuth 2.0 connection for a supported app and exercise finer control over each setting.
- WSSE: Use this authentication type for connectors that authorize API calls based on WSSE tokens sent in the request header. Web Services Security Environment must be used where the connecting application expects crypto hash generated out of the username and the password. This value is submitted in the header, and the default name of the header is X-WSSE (but this is customizable).
What would you like to export?
In the What would you like to export? section, configure the following:
HTTP method (required): Provide the HTTP action required by your source application to retrieve the export data (options include GET, PUT, or POST).
Configure HTTP headers (optional): In some cases, you must include custom HTTP headers with your API requests; provide the name and value as required by the API.
Relative URI (required): Enter the relative URI that integrator.io can use to export the data.
Build HTTP request body (optional): If you need to pass anything in the request body, you can use the Build HTTP request body editor to create complex relative URLs based on the parameters to be passed to the API. The Build HTTP request body editor allows you to use handlebars expressions. Click Launch to open the Build HTTP request body editor.
Configure export type
The following export types are available to help support common integration patterns:
- All: Exports all data, always
- Delta: Exports data changed since the last time the flow ran
- Once: Exports data that has not already been exported, and also automatically updates records to mark them as exported
- Test: Exports a single record by default, primarily used for testing to avoid syncing lots of data
Does this API support paging?
Some APIs offer paging functionality in order to limit their HTTP response to a manageable size if the total set of resources is large. If the API supports pagination, then you can configure the Does this API supports paging? section. Instead of providing data in a single large payload, the source application will provide the data in multiple smaller payloads for integrator.io to handle.
integrator.io supports the following paging methods:
Choose this method if the API points to subsequent pages in the link header of the API response. When you select this paging method, by default integrator.io uses HTTP conventions to look for the next page URL within a dedicated link header value. You can include multiple URLs for page navigation (forward, back, first, last, etc.).
Link header relation: If the API uses multiple values, you can define those values here. The conventions for these relation values include prev, next, last, and first (integrator.io defaults to next). Since some APIs may not comply with the default "next" relation, you can override it by entering the value expected by your HTTP app. For more information on link headers, see the IETF standards for web linking.
Use this paging method when the API provides the complete URL to which integrator.io must make the request for getting the next page.
Path: Enter the path used to retrieve the response.
Choose this paging method if the API uses a page number parameter to paginate the results.
Page: Set an initial Page value if the export should start at a custom page offset. This value automatically increments with each subsequent page request. The value can be referenced as a placeholder in the body, headers, or relative URI of the export request.
Example
/products?page={{{export.http.paging.page}}}
Max page path: Some APIs return the total number of pages available in each of their page responses. Use this optional field as a trigger to stop making page requests. This field tells integrator.io where to find the maximum page value within each page response. If omitted, integrator.io continues to make requests until no resources return or a 404 (not found) response occurs.
Max count path: Some APIs return the total number of resources available in each of their page responses. Use this optional field as a trigger to stop making page requests. This field tells integrator.io where to find the total resource count within each page response. If omitted, integrator.io continues to make requests until no resources return or a 404 (not found) response occurs.
Chose this paging method if paging is tracked by a value included in the POST request body.
Build paging post body: Click Launch to open the Build paging post body editor. Here you can enter handlebars expressions to reference a value in the body of the POST request.
Use this paging method if the API provides the ID or token as part of a page response. This ID or token is used as part of the URL in the following request in order to retrieve the next page response.
Relative URI: Enter the relative URI that retrieves the next page response.
Use this paging method if the API uses a skip-take paging mechanism (also known as offset or limit).
Skip: Use this field if the export should start at an offset from the first record. This value is automatically set on each subsequent page request. The value can be referenced as a placeholder in the body, headers, or relativeURI of the export request.
Example:
/products?offset={{{export.http.paging.skip}}}
Max page path: Some APIs return the total number of pages available in each of their page responses. This field tells integrator.io where to find the maximum page value within each page response, and when this page count is met, no more page requests will be made. If left blank, integrator.io will continue to make requests until no resources are returned, or a 404 (not found) response returns.
Max count path: Some APIs return the total number of resources available in each of their page responses. This field tells integrator.io where to find the total resource count within each page response, and when this resource count is met, no more page requests will be made. If left blank, integrator.io continues to make requests until no resources are returned, or a 404 (not found) response returns.
Choose this paging method if the API returns a token to the next page within the response body.
Token: If the initial export should contain a token, this value is automatically set on each subsequent page request. You can reference the value as a placeholder in the body, headers, or relative URI of the export request.
For example:
/products?token={{{export.http.paging.token}}}
Path: If the paging method is set to token, integrator.io needs to be able to find the token in the HTTP response of each page in order to construct the next page request. This field lets integrator.io know where to look for the next page token. If no value is found at this path, then paging is terminated (final page reached). When the paging method is set to Next page URL, integrator.io uses this Path field to find where in the HTTP response to look to find a URL to use when requesting the next page of data. The format of the path is dependent on the media type (content type) of the original (first) page response.
Example JSON response path
result.paging.nextPageUrl
Example XPATH for XML path
/result/paging/nextPageUrl
Paging terminates when no URL is found at the specified path.
Relative URI: Use this optional field in combination with any paging method. In some cases subsequent page requests need to use a different relative URI from the first page request.
Token path after first request: Use this optional field in combination with any paging method. In some cases, subsequent page requests need to use a different relative URI than the one used for the first page request.
Resource path: This is an optional override to the resource path if subsequent page responses have a different response structure. Note that the format of this path is specific to the media type.
Example JSON resource path
results.customers
Example XPATH for XML resource path
/SearchResults/Customers/Customer
Last page status code: Some APIs return an error status code (less than 200 or greater than 300) as the last page response. Enter the response code integrator.io should expect in these situations so the system doesn't interpret the response as an error. You can use this field in conjunction with the Last page path and Last page values fields. integrator.io assumes error 404 is the last page response, so if the API you are connecting to also uses that convention, you can leave this field blank.
Last page path: If you have entered a Last page status code value, you use Last page path to search for a specific field in the response object to determine if paging is complete. For example, the specified field can be "error.message". If Last page path is empty, then paging will be considered complete; otherwise, the response will be considered an error.
Last page values: If you have entered values for the Last page status code and Last page path fields, then use this field to check a specific field's value in the response object to determine if paging is complete. If the value in the response object matches the value in the Last page values field, then paging will be considered complete; otherwise, this response will be considered an error.
Preview data
Once you have configured all of the export settings, click Preview to review the response from the API call. If you see any errors, then revisit the steps above to fix the problems.
For more information on Preview panel functionality, see Preview export data.
Non-standard API response patterns
Some APIs may have response patterns that are unusual when compared to most APIs. Use the settings below to accommodate outlying response codes.
Resource path: This optional field allows integrator.io to locate the resource (or set of resources) returned from an API call. If the HTTP response from an API contains resources at the root, then no value is necessary for this field. If the response from an API contains a deeper response structure that contains paging information, you must provide the path to the resources. If no value is provided, integrator.io assumes that the resources can be found at the root of the response. Note that the format of this path is specific to the media type. If the response is represented in JSON structure, then use "dot" notation to describe the path to your resources.
Example JSON path
results.customers
Example XPATH for XML
/SearchResults/Customers/Customer
Error path: Use this optional field to help identify where in the body of a failed HTTP response integrator.io can find the error message. You can provide the field path to the property/node containing the error message. If no value is given, then the full HTTP response body is used as the description of the failure in the dashboard. Use XPATH if the media type of the failed response is XML, and use JSON if the media type is JSON. Note that if failed responses from an application have no body, then a text version of the HTTP status code is used as the reason for failure.
Example JSON path
result.error.message
Example XPATH for XML
/result/error.message/text()
Success path: Not all services return HTTP status codes to determine the success or failure of a request. If a service always returns 200 HTTP status code, even for some failures, then use this field to tell integrator.io where to look for a success indicator. This field is used in combination with the Success values field to further refine how integrator.io can identify a successful response. If no success values are provided, any value found in the response body at this path is considered a success.
Success values: Use this optional field along with the Success path field. The value found in the HTTP response at the path provided by Success path is compared against the provided list of success values. If there is an exact case-sensitive match of any of the values, then the request is considered successful.
Fail path: If the service you are connecting to embeds authentication errors within the HTTP body, use this field to set the path within the response body where integrator.io should look to identify a failed auth response. If there is a specific value (or set of values) that indicate a failed auth response at this path, use the Fail values field for further instructions on how to identify this type of error.
Fail values: This field is used only if the Fail path field is set. It indicates which specific values to test for when determining if the requests failed for authentication reasons. Use this field only if Fail path is set. It indicates to integrator.io which values to test when determining if the requests failed for authentication reasons.
Override success media type: The success media type indicates the format of the data received from all successful (HTTP) requests (CSV, JSON, or XML). Typically, APIs will only support one media type (data format) and will publish that info at the top of their API guides. This is an optional override of the media type associated with the connection in case the API has route-specific data formats. Amazon MWS, for example, is an XML service that returns CSV data for some of its reports.
Override error media type: (JSON or XML)The error media type indicates the format of the response data received from all unsuccessful (HTTP) requests. This refers to all responses with non-2xx HTTP status codes. Typically, APIs support one media type (data format) and publish that info at the top of their API guides. This is an optional override of the media type associated with the connection in case the API has route specific data formats. Amazon MWS, for example, has some endpoints that return CSV data, but send error responses in XML.
Advanced
Page size: When an export runs in the context of a data flow (meaning integrator.io sends the data from the export to an import queue), integrator.io breaks the exported data into one or more smaller pages of records. You can use the Page size field to specify how many records you want in each page of data. If you leave this field blank, the default system value is 20. There is no maximum value, but a page of data is automatically capped when it exceeds 5 MB. Most of the time, the application that you are importing data into will bottleneck the page size value. For example, NetSuite and Salesforce both document a maximum number of records that can be submitted in any single request, upon import.
Data URI template: When your flow runs but has data errors, this field allows you to verify that all the errors have a link to the original data in the export application. This field uses a handlebars template to generate the dynamic links based on the data being exported.
For example, if you are exporting a customer record from Shopify, you would typically set this field to the following value:
https://your-store.myshopify.com/admin/customers/{{{id}}}
Or, if you are just exporting a CSV file from an FTP site, then this field could be one or more columns from the file:
{{{internal_id}}, {{{email}}}
Do not store retry data: Check this if you do not want integrator.io to store retry data for records that fail in your flow, for example, if you have sensitive data that you don’t want stored on our platform. You should also check this box if you are exporting large data sets. Storing the retry data for large data sets isn’t necessary because the flow is idempotent and storing retry data slows the process.
Invoke: This resource can be invoked via an HTTP POST request to this unique URL. Click Copy URL to save the URL to the clipboard.
Configure async helper: If data is exported asynchronously, check this box to select the async helper configuration.
Once you have completed these settings, click Save.
Comments
0 comments
Please sign in to leave a comment.