Export data from an HTTP source app

This article describes the options available for exporting data from an HTTP source application. 

Contents

• Create the export
• Configure HTTP authentication settings
• What would you like to export
• Configure export type
• Does this API support paging?
• Preview data
• Non-standard API response patterns
• Advanced

Create the export

1. Sign in to integrator.io.
2. From the Tools menu, select Flow Builder.
3. Click Add source in the Sources section of the new flow.
4. The Create source pane opens.
5. From the Application list, select HTTP from the list of connection types.  
   
6. After the HTTP Application is added, the Create source pane prompts you for additional information about your export and connection settings. Select an existing connection or click the + button to create one.

Configure HTTP authentication settings

Choose one of the following authentication types:

• Basic: Select Basic if your service implements the HTTP basic authentication strategy. This authentication method adds a Base64-encoded username and password values in the authentication HTTP request header.
• Cookie: Select Cookie if your service relies on session-based authentication. Session-based authentication is typically implemented by including a unique cookie into the HTTP request header. By selecting this option, the platform will automatically create and insert this cookie into every HTTP request it sends to your application.
• Custom: Select Custom for all other types. If you select the Custom authentication method, integrator.io will not perform any special authentication; you can freely configure the HTTP request fields (method, relativeUri, headers, and body) of the import and export models to include {{placeholders}} for any authentication values stored in Encrypted and Unencrypted fields of this connection.
• Digest: Select Digest if your service relies on digest authentication. With “Digest” auth, the client sends a first request to the API, and the server responds with a few details, including a number that can be used only once (nonce), a realm value, and a 401 unauthorized response. An encrypted array of data including username and password combined with the data received from the server in the first request is then sent back. The server uses the passed data to generate an encrypted string and compares it against what is sent in the previous step to authenticate requests.
• Token: Select Token if your service relies on token-based authentication. The token may exist in the header, URL, or body of the HTTP request. This method also supports refreshing tokens if the API supports it.
• OAuth 2.0: You can select this option to connect to any OAuth 2.0-compliant app that does not have a standard integrator.io connection, or you can build your own OAuth 2.0 connection for a supported app and exercise finer control over each setting.
• WSSE: Use this authentication type for connectors that authorize API calls based on WSSE tokens sent in the request header. Web Services Security Environment must be used where the connecting application expects crypto hash generated out of the username and the password. This value is submitted in the header, and the default name of the header is X-WSSE (but this is customizable). 

What would you like to export?

In the What would you like to export? section, configure the following:

HTTP method (required): Provide the HTTP action required by your source application to retrieve the export data (options include GET, PUT, or POST).

Configure HTTP headers (optional): In some cases, you must include custom HTTP headers with your API requests; provide the name and value as required by the API.

Relative URI (required): Enter the relative URI that integrator.io can use to export the data.

Build HTTP request body (optional): If you need to pass anything in the request body, you can use the Build HTTP request body editor to create complex relative URLs based on the parameters to be passed to the API. The Build HTTP request body editor allows you to use handlebars expressions. Click Launch to open the Build HTTP request body editor.

Configure export type

The following export types are available to help support common integration patterns:

• All: Exports all data, always
• Delta: Exports data changed since the last time the flow ran. Add {{lastExportDateTime}} to either the relative URI or HTTP request body to complete the setup. The {{lastExportDateTime}} field is automatically calculated based on the last time the flow ran and successfully exported data. You can reference this read-only date/time field from other fields using handlebars. For example: /products?after={{lastExportDateTime}}.
• Once: Exports data that has not already been exported, and also automatically updates records to mark them as exported
• Test: Exports a single record by default, primarily used for testing to avoid syncing lots of data

Does this API support paging?

See Pagination for details on which paging method to use.

Preview data

Once you have configured all of the export settings, click Preview to review the response from the API call. If you see any errors, then revisit the steps above to fix the problems.

For more information on Preview panel functionality, see Preview export data.

Non-standard API response patterns

Some APIs may have response patterns that are unusual when compared to most APIs. Use the settings below to accommodate outlying response codes.

Resource path: This optional field allows integrator.io to locate the resource (or set of resources) returned from an API call. If the HTTP response from an API contains resources at the root, then no value is necessary for this field. If the response from an API contains a deeper response structure that contains paging information, you must provide the path to the resources. If no value is provided, integrator.io assumes that the resources can be found at the root of the response. Note that the format of this path is specific to the media type. If the response is represented in JSON structure, then use "dot" notation to describe the path to your resources.

Example JSON path

results.customers

Example XPATH for XML

/SearchResults/Customers/Customer

Error path: Use this optional field to help identify where in the body of a failed HTTP response integrator.io can find the error message. You can provide the field path to the property/node containing the error message. If no value is given, then the full HTTP response body is used as the description of the failure in the dashboard. Use XPATH if the media type of the failed response is XML, and use JSON if the media type is JSON. Note that if failed responses from an application have no body, then a text version of the HTTP status code is used as the reason for failure.

Example JSON path

result.error.message

Example XPATH for XML

/result/error.message/text()

Success path: Not all services return HTTP status codes to determine the success or failure of a request. If a service always returns 200 HTTP status code, even for some failures, then use this field to tell integrator.io where to look for a success indicator. This field is used in combination with the Success values field to further refine how integrator.io can identify a successful response. If no success values are provided, any value found in the response body at this path is considered a success.

Success values: Use this optional field along with the Success path field. The value found in the HTTP response at the path provided by Success path is compared against the provided list of success values. If there is an exact case-sensitive match of any of the values, then the request is considered successful.

Fail path: If the service you are connecting to embeds authentication errors within the HTTP body, use this field to set the path within the response body where integrator.io should look to identify a failed auth response. If there is a specific value (or set of values) that indicate a failed auth response at this path, use the Fail values field for further instructions on how to identify this type of error.

Fail values: This field is used only if the Fail path field is set. It indicates which specific values to test for when determining if the requests failed for authentication reasons. Use this field only if Fail path is set. It indicates to integrator.io which values to test when determining if the requests failed for authentication reasons.

Override success media type: The success media type indicates the format of the data received from all successful (HTTP) requests (CSV, JSON, or XML). Typically, APIs will only support one media type (data format) and will publish that info at the top of their API guides. This is an optional override of the media type associated with the connection in case the API has route-specific data formats. Amazon MWS, for example, is an XML service that returns CSV data for some of its reports.

Override error media type: (JSON or XML)The error media type indicates the format of the response data received from all unsuccessful (HTTP) requests. This refers to all responses with non-2xx HTTP status codes. Typically, APIs support one media type (data format) and publish that info at the top of their API guides. This is an optional override of the media type associated with the connection in case the API has route specific data formats. Amazon MWS, for example, has some endpoints that return CSV data, but send error responses in XML.

Advanced

Page size: When an export runs in the context of a data flow (meaning integrator.io sends the data from the export to an import queue), integrator.io breaks the exported data into one or more smaller pages of records. You can use the Page size field to specify how many records you want in each page of data. If you leave this field blank, the default system value is 20. There is no maximum value, but a page of data is automatically capped when it exceeds 5 MB. Most of the time, the application that you are importing data into will bottleneck the page size value. For example, NetSuite and Salesforce both document a maximum number of records that can be submitted in any single request, upon import.

Data URI template: When your flow runs but has data errors, this field allows you to verify that all the errors have a link to the original data in the export application. This field uses a handlebars template to generate the dynamic links based on the data being exported.

For example, if you are exporting a customer record from Shopify, you would typically set this field to the following value:

https://your-store.myshopify.com/admin/customers/{{{id}}}

Or, if you are just exporting a CSV file from an FTP site, then this field could be one or more columns from the file:

{{{internal_id}}, {{{email}}}

Do not store retry data: Check this if you do not want integrator.io to store retry data for records that fail in your flow, for example, if you have sensitive data that you don’t want stored on our platform. You should also check this box if you are exporting large data sets. Storing the retry data for large data sets isn’t necessary because the flow is idempotent and storing retry data slows the process.

Invoke: This resource can be invoked via an HTTP POST request to this unique URL. Click Copy URL to save the URL to the clipboard.

Configure async helper: If data is exported asynchronously, check this box to select the async helper configuration.

Once you have completed these settings, click Save.