Export data from an HTTP source application

This article describes the options available for exporting data from an HTTP source application or configuring an export for async helpers. 

Create the export

1. Sign in to integrator.io.
2. From the Tools menu, select Flow Builder.
3. Click Add source in the Sources section of the new flow.
4. The Create source pane opens.
5. From the Application list, select HTTP from the list of connection types.  
   
6. After the HTTP Application is added, the Create source pane prompts you for additional information about your export and connection settings. Select an existing connection or click the + button to create one.

7. Click Next. (The Create export page opens).

General

Name (required): Give the export an unique and identifiable name that is unlikely to be confused with other exports in your integrator.io environment.

Description (optional): Describe your resource so that other users can quickly understand what it is doing without having to read through all the fields and settings. Be sure to highlight any nuances that a user should be aware of before using your resource in their flows. Also, as you make changes to the resource be sure to keep this field up to date.

Connection: The specific connection you would like to use for your export or import. You can pre-establish and save your connections using Menu > Connections. Each stored connection contains credentials along with other related information needed to access the desired application. In addition, you can click the + icon beside this field to create a new connection.

What would you like to export?

In the What would you like to export? section, configure the following:

HTTP method (required): Provide the HTTP action required by your source application to retrieve the export data (options include GET, PUT, or POST).

Relative URI (required): Enter the relative URI that integrator.io can use to export the data.

Configure HTTP headers (optional): In some cases, you must include custom HTTP headers with your API requests; provide the name and value as required by the API.

HTTP request body (optional): If you need to pass anything in the request body, you can use the Build HTTP request body editor to create complex relative URLs based on the parameters to be passed to the API. The Build HTTP request body editor allows you to use handlebars expressions. Click Launch to open the Build HTTP request body editor.

Configure export type

The following export types are available to help support common integration patterns:

• All: Exports all data, always.
• Delta: Exports data changed since the last time the flow ran.

url?from={{lastExportDateTime}}&to={{currentExportDateTime}}

url?from=10:00&to=10:02

url?from=10:02&to=10:04

url?from=10:04&to=10:06

• Once: Exports data that has not already been exported, and also automatically updates records to mark them as exported.
• Limit: Exports a limited set of records, primarily used for testing to avoid syncing lots of data. 
  

  Note: When you select the export type as Limit - export a set number of records, another setting How many records would you like to export? is enabled for you to enter the maximum number of records to export each time this flow runs. The default value is set to one. You can export a maximum of 100 records.

Does this API use paging?

See Pagination for details on which paging method to use.

Preview data

Once you have configured all of the export settings, click Preview to review the response from the API call. If you see any errors, then revisit the steps above to fix the problems.

For more information on Preview panel functionality, see Preview export data.

Non-standard API response patterns

Some APIs may have response patterns that are unusual when compared to most APIs. Use the settings below to accommodate outlying response codes.

Path to records in HTTP response body: This optional field allows integrator.io to locate the resource (or set of resources) returned from an API call. If the HTTP response from an API contains resources at the root, then no value is necessary for this field. If the response from an API contains a deeper response structure that contains paging information, you must provide the path to the resources. If no value is provided, integrator.io assumes that the resources can be found at the root of the response. Note that the format of this path is specific to the media type. If the response is represented in JSON structure, then use "dot" notation to describe the path to your resources.

Important: You must manually define the namespace if you are using XML/XPATH. You can also use the following function:

/*[local-name()='OrderDto']

Example JSON path

results.customers

Example XPATH for XML

/SearchResults/Customers/Customer

Path to error field in HTTP response body: Use this optional field to help identify where in the body of a failed HTTP response integrator.io can find the error message. You can provide the field path to the property/node containing the error message. If no value is given, then the full HTTP response body is used as the description of the failure in the dashboard. Use XPATH if the media type of the failed response is XML, and use JSON if the media type is JSON. Note that if failed responses from an application have no body, then a text version of the HTTP status code is used as the reason for failure.

Important: You must manually define the namespace if you are using XML/XPATH.

Example JSON path

result.error.message

Example XPATH for XML

/result/error.message/text()

Error values: This field is used only if the Fail path field is set. It indicates which specific values to test for when determining if the requests failed for authentication reasons. Use this field only if Fail path is set. It indicates to integrator.io which values to test when determining if the requests failed for authentication reasons.

Path to success field in HTTP response body: Not all services return HTTP status codes to determine the success or failure of a request. If a service always returns 200 HTTP status code, even for some failures, then use this field to tell integrator.io where to look for a success indicator. This field is used in combination with the Success values field to further refine how integrator.io can identify a successful response. If no success values are provided, any value found in the response body at this path is considered a success. You can also use the following function for namespaces:

/[local-name()='OrderDto']/[name()='diffgr:diffgram']/[local-name()='OrderDto']/[local-name()='Order']/*[local-name()='OrderId']

Success values: Use this optional field along with the Success path field. The value found in the HTTP response at the path provided by Success path is compared against the provided list of success values. If there is an exact case-sensitive match of any of the values, then the request is considered successful.

Path to detailed error message field in HTTP response body: If the service you are connecting to embeds authentication errors within the HTTP body, use this field to set the path within the response body where integrator.io should look to identify a failed auth response. If there is a specific value (or set of values) that indicate a failed auth response at this path, use the Fail values field for further instructions on how to identify this type of error.

Override media type for success responses: The success media type indicates the format of the data received from all successful (HTTP) requests (CSV, JSON, or XML). Typically, APIs will only support one media type (data format) and will publish that info at the top of their API guides. This is an optional override of the media type associated with the connection in case the API has route-specific data formats. Amazon MWS, for example, is an XML service that returns CSV data for some of its reports.

Override media type for error responses: (JSON or XML)The error media type indicates the format of the response data received from all unsuccessful (HTTP) requests. This refers to all responses with non-2xx HTTP status codes. Typically, APIs support one media type (data format) and publish that info at the top of their API guides. This is an optional override of the media type associated with the connection in case the API has route specific data formats. Amazon MWS, for example, has some endpoints that return CSV data, but send error responses in XML.

XML parser helper: The XML parser will give you immediate feedback on how your parse options are applied against raw XML data. See XML parser helper.

Path to file URLs in HTTP response body: Use this optional field to to help identify where in the body of a HTTP response, integrator.io can find URLs to download files. If you want to transfer files to subsequent flow steps, you should enter a field here. If you enter more than one field, use a comma to separate them. Each field can contain one URL or an array of URLs. If you preview the export data, you can see the field and URLs in the HTTP Response.

Important: The file URLs must be public or self-signed, not requiring authentication.

Note: You can receive Gzip compressed responses from any app endpoint. The HTTP connector can parse these responses because it accepts Gzip encoded data by default for all media formats.

Would you like to group records?

You can group HTTP export or lookup records to aggregate your data based on your chosen field(s). The suggested list of fields contains the top-level properties in JSON fields.

integrator.io creates a new grouped record based on your chosen field(s). Select the fields you want to group, based on your sample response data. Fields other than basic data types (such as string and number) are unavailable. If the source application does not sort the exported data, then grouping may not work as expected. The order of each grouping is always respected.

The Sort and group content article provides more examples, though you should note that sorting is not available for universal HTTP exports or lookups at this time. Take the following JSON record:

{
  "page_of_records": [
    {
      "record": {
        "NAME": "Sarah",
        "AGE": 97,
        "PURCHASE": "Beach towel",
        "CATEGORY": "Summer"
      }
    },
    {
      "record": {
        "NAME": "John",
        "AGE": 98,
        "PURCHASE": "Ski gloves",
        "CATEGORY": "Winter"
      }
    },
    {
      "record": {
        "NAME": "Ana",
        "AGE": 99,
        "PURCHASE": "Beach ball",
        "CATEGORY": "Summer"
      }
    },
    {
      "record": {
        "NAME": "James",
        "AGE": 100,
        "PURCHASE": "Snowboard",
        "CATEGORY": "Winter"      
       }
    }
  ]
}

Using the example JSON record, you can group by the Category field. Your data is grouped into Summer and Winter rows. Of course, integrator.io can handle more detailed groupings, say if you want to group records by Category and Purchase, or by Category, Age, and Purchase.

{
  "page_of_records": [
    {
      "rows": [
        {
          "CATEGORY": "Summer",
          "NAME": "Sarah",
          "AGE": 97,
          "PURCHASE": "Beach towel"
        },
        {
          "CATEGORY": "Summer",
          "NAME": "Ana",
          "AGE": 98,
          "PURCHASE": "Beach ball"
        }
      ]
    },
    {
      "rows": [
        {
          "CATEGORY": "Winter",
          "NAME": "John",
          "AGE": 99,
          "PURCHASE": "Ski gloves"
        },
        {
          "CATEGORY": "Winter",
          "NAME": "James",
          "AGE": 100,
          "PURCHASE": "Snowboard"
        }
      ]
    }
  ]
}

Tip: Remember that the Page size property under the Advanced section determines how many records/groups can go into a page. If a single group exceeds the page size, that group is discarded from the export, and an error indicating this issue is reported.

Mock Output

You can add mock output data to exports and lookups in the Mock output section of the Edit export page as simulated output data when configuring a flow.

Advanced

Configure async helper: If data is exported asynchronously, check this box to select the async helper configuration.

Page size: When an export runs in the context of a data flow (meaning integrator.io sends the data from the export to an import queue), integrator.io breaks the exported data into one or more smaller pages of records. You can use the Page size field to specify how many records you want in each page of data. If you leave this field blank, the default system value is 20. There is no maximum value, but a page of data is automatically capped when it exceeds 5 MB. Most of the time, the application that you are importing data into will bottleneck the page size value. For example, NetSuite and Salesforce both document a maximum number of records that can be submitted in any single request, upon import.

Data URI template: When your flow runs but has data errors, this field allows you to verify that all the errors have a link to the original data in the export application. This field uses a handlebars template to generate the dynamic links based on the data being exported.

For example, if you are exporting a customer record from Shopify, you would typically set this field to the following value:

https://your-store.myshopify.com/admin/customers/{{{id}}}

Or, if you are just exporting a CSV file from an FTP site, then this field could be one or more columns from the file:

{{{internal_id}}, {{{email}}}

Override trace key template: Define a trace key that integrator.io will use to identify a unique record. You can use a single field such as {{{field1}}} or use a handlebar expression. For example, this syntax {{join “_” field1 field2}} will generate a trace key template field1_field2. When this field is set, you will override the platform default trace key field.

Once you have completed these settings, click Save.