Articles in this section

Export Microsoft Azure Blob Storage files

This article describes the configuration options for exporting files from Microsoft Azure Blob Storage.

General

Setting

Description

Name your transfer

Name your resource so that you can easily reference it from other parts of the application.

Description

Describe your resource so that other users can quickly understand what it is doing without having to read through all the fields and settings. Be sure to highlight any nuances that a user should be aware of before using your resource in their flows. Also, keep this field up to date as you make changes to the resource.

Connection

Find an existing connection to apply to this export or lookup, or you can click Create connection to define a new resource that you can use here and in other flow steps. Each connection contains securely stored credentials and information needed to access this application. All connections displayed in this list are specific to this application and registered to you for this integration.

Parse files being transferred

Do you need to parse CSV, XML, JSON, etc... files into records before sending them to other applications, or should the files simply be transferred raw as-is (i.e. PDFs, JPGs, etc...)?

How would you like to parse files?

File type (required): Choose the type of file to be exported from Google Drive. For example, choose CSV if you are exporting a flat, delimited text file, or XLSX for a binary Microsoft Excel file. The file type you select changes the fields available on the Create export panel. Acceptable file types include:

  • CSV (or any delimited text file)

  • EDI X12

  • EDIFACT

  • Fixed Width

  • JSON

  • XLSX

  • XML

CSV (or any delimited text file)

Sample file (that would be parsed) (required): This field only displays if you choose CSV, JSON, XLSX, or XML as a file type. Choose a sample file to define the record structure. Click Choose file and navigate to a sample version of the files you will be exporting.

CSV parser helper: The CSV parser helper can be used to visualize and experiment with how integrator.io parses CSV files (or any other delimited text files) into the JSON records/rows that then get processed by your flow. See CSV parser helper.

EDI X12

EDI x12 format (required): Select the EDI X12 file format that matches the files you are exporting from Google Drive.

EDIFACT

EDIFACT format (required): Select the EDIFACT file format that matches the files you are exporting from Google Drive.

Fixed Width

Format (required): Select the Fixed width file format that matches the files you are exporting from Google Drive.

JSON

Sample file (that would be parsed) (required): This field only displays if you choose CSV, JSON, XLSX, or XML as a file type. Choose a sample file to define the record structure. Click Choose file and navigate to a sample version of the files you will be exporting.

Resource path: You can use this field optionally to define the JSON path to the resources you are exporting from the JSON file with handlebars syntax.

XLSX

Sample file (that would be parsed) (required): This field only displays if you choose CSV, JSON, XLSX, or XML as a file type. Choose a sample file to define the record structure. Click Choose file and navigate to a sample version of the files you will be exporting.

File has header: Check this box if the files you are exporting contain a top level header row. If the first row of your XLSX file is reserved for column names (and not actual data), then check this box.

XML

Sample file (that would be parsed) (required): This field only displays if you choose CSV, JSON, XLSX, or XML as a file type. Choose a sample file to define the record structure. Click Choose file and navigate to a sample version of the files you will be exporting.

XML parser helper: The XML parser will give you immediate feedback on how your parse options are applied against raw XML data. See XML parser helper.

Where would you like to transfer from?

Setting

Description

Container name

Specify the Azure Blob Storage container and optional virtual path prefix for the files to be transferred. Click Browse to open a container and prefix picker, and select a path — the selected value is written back to the field automatically. integrator.io will transfer all files and delete them from the container once the transfer completes. You can also (optionally) configure integrator.io to leave files in the container or transfer files that match a certain 'starts with' or 'ends with' file name pattern. Note that Azure Blob Storage uses a flat storage model — folder-like paths are virtual prefixes formed by forward slashes in blob names, not real directories.

File filtering conditions

Add one or more conditions (or groups of conditions) to define the criteria used to determine which files are retrieved when the export runs. If no conditions are set, the export retrieves all files in the specified path. See Create file filtering conditions for file transfer exports

Sort and group records

The Sorting and Grouping option allows you to manage your flow’s files by sorting and grouping records by field. This feature allows you to aggregate your data based on your chosen field(s), and sort the records in ascending or descending order (ASC or DESC).

Sort by

This feature allows you to sort by multiple fields (ASC or DESC) or in the order that the fields were initially entered. You can enter the “Sort records by fields” by a field ID or by using a dot notation.

Note

  • This feature does not sort numeric fields, including dates, in descending or ascending values.

  • Sorting only applies to file provider exports. You can’t sort records for HTTP, database, and data warehouse exports or lookups at this time.

Group by

You can group records when parsing files from any file source, or when exporting records from HTTP, databases, or data warehouses. This feature allows you to aggregate your data based on your chosen field(s). Select the fields you want to group, based on your sample response data. Fields other than basic data types (such as string and number) are unavailable. If the source application does not sort the exported data, then grouping may not work as expected. The order of each grouping is always respected. You can enter records by field id, by using a dot notation, or one field at a time. When you input at least one field, the platform will group records in the order that the fields were initially entered, from top to bottom.

Note

The dot notation starts from “record”: { in the example below.

Take the following JSON record:

{
  "page_of_records": [
    {
      "record": {
        "NAME": "Sarah",
        "AGE": 97,
        "PURCHASE": "Beach towel",
        "CATEGORY": "Summer"
      }
    },
    {
      "record": {
        "NAME": "John",
        "AGE": 98,
        "PURCHASE": "Ski gloves",
        "CATEGORY": "Winter"
      }
    },
    {
      "record": {
        "NAME": "Ana",
        "AGE": 99,
        "PURCHASE": "Beach ball",
        "CATEGORY": "Summer"
      }
    },
    {
      "record": {
        "NAME": "James",
        "AGE": 100,
        "PURCHASE": "Snowboard",
        "CATEGORY": "Winter"      
       }
    }
  ]
}

Using the example JSON record, you can group by the Category field. Your data is grouped into Summer and Winter rows. Of course, integrator.io can handle more detailed groupings, say if you want to group records by Category and Purchase , or by Category , Age , and Purchase. If you’re working with a file provider export, you can sort the records by Ascending or Descending.

{
  "page_of_records": [
    {
      "rows": [
        {
          "CATEGORY": "Summer",
          "NAME": "Sarah",
          "AGE": 97,
          "PURCHASE": "Beach towel"
        },
        {
          "CATEGORY": "Summer",
          "NAME": "Ana",
          "AGE": 98,
          "PURCHASE": "Beach ball"
        }
      ]
    },
    {
      "rows": [
        {
          "CATEGORY": "Winter",
          "NAME": "John",
          "AGE": 99,
          "PURCHASE": "Ski gloves"
        },
        {
          "CATEGORY": "Winter",
          "NAME": "James",
          "AGE": 100,
          "PURCHASE": "Snowboard"
        }
      ]
    }
  ]
}

In some cases, you might have CSV file exports or lookups:

Name

Age

Purchase

Category

Sara

97

Beach towel

Summer

Anna

98

Ski gloves

Winter

John

99

Beach ball

Summer

James

100

Snowboard

Winter

In this case, you’d group by column (Category), and each row would be grouped into the Summer or Winter category. Logically, it would look something like this:

Category

Name

Age

Purchase

Summer

Sara

97

Beach towel

John

99

Beach ball

Winter

James

100

Snowboard

Anna

98

Ski gloves

Tip

Remember that the Page size property under the Advanced section determines how many records/groups can go into a page. If a single group exceeds the page size, that group is discarded from the export, and an error indicating this issue is reported.

Sorting and grouping matrix

This matrix details all possible sorting and grouping outcomes.

Group records – not provided

Group records – provided

Sort records – not provided

File processed as is.

Content is grouped by group record fields.

If a record has a missing field, then the record is grouped in a “missing fields” bucket.

Notice

group records by firstName field, but a record does not have a firstName field.

Sort records – provided

Content is sorted by the created sort rules.

Content is first sorted, then grouped.

Sorting and grouping with new and existing flows

This section provides details regarding sorting and grouping new and existing flows.

Sort and group in a new flow

When sorting and grouping files in a new lookup or export:

  • Select a Sorting and grouping option to specify the file transfer requirements.

Note

Deprecated grouping fields are not available when creating a flow.

  1. Specify whether the file has headers.

  2. Sort and group records as needed.

  3. Select Save and close.

Edit sort and group in a new flow

When editing sorting and grouping in a new flow:

  • Select a Sorting and grouping option to specify the file transfer requirements.

Note

Deprecated grouping fields are not available when creating a flow.

  1. Specify whether the file has headers.

  2. Sort and group records as desired.

  3. Select Save and close.

Edit sort and group using the deprecated feature

When editing the sorting and grouping in an existing flow:

  • The deprecated fields are available until you start using the new feature.

  • Existing, deprecated group fields are automatically generated in the new feature, though they may require further configuration.

  • After the new fields are transferred and saved, the deprecated fields will disappear permanently. The only way to use the old fields is through the integrator.io API.

Advanced

Setting

Description

Decrypt files

Use this option to decrypt files. This option is enabled only when you have configured at least one cryptographic system in the connection and selected the parse file option. If it is unchecked, then decryption will not be performed when files are parsed.

Decompress files

If you're exporting compressed files, set this field to True. Once you check this field, a dropdown field appears for you to choose the compression format

Leave file on server

Check this box to override the default behavior, in which integrator.io deletes the files after transferring them from the source location. If selected, the files will be left on the server, and the same files will be transferred when your flow runs again. For example, if you are downloading or parsing files from an FTP folder and checked Leave file on server, then integrator.io will transfer all files on the FTP folder but leave them in the FTP folder. If the transfer runs again, then integrator.io will transfer the same files.

Note

When running a flow in test mode, integrator.io will not delete files from the transfer application, regardless of this setting.

File encoding

The file encoding indicates how the individual characters in your data are represented on the file system. The default encoding is utf-8. Depending on the source system of the data, the encoding can take on different formats. Current supported formats are: utf-8, win-1254 and utf-16le. If you do not know what encoding your data is, in most cases it will be utf-8.

Backup container name

Where to store backup copies of processed files using a relative path. Resolved at runtime via the provider's listing API. Leave empty to skip backups. Click Browse to open a container and prefix picker, and select a path — the selected value is written back to the field automatically. integrator.io will transfer all files and delete them from the container once the transfer completes. You can also (optionally) configure integrator.io to leave files in the container or transfer files that match a certain 'starts with' or 'ends with' file name pattern. Note that Azure Blob Storage uses a flat storage model — folder-like paths are virtual prefixes formed by forward slashes in blob names, not real directories.

Page size

When an export runs in the context of a data flow (where the data from the export is sent right away to an import queue) integrator.io will break the data being exported into one or more smaller pages of records. Saying this another way, integrator.io uses streaming to export data out of one app and import it into another app. The 'Page size' field can be used to specify how many records you want in each page of data. The default system value (when you leave this field blank) is 20. There is no max value, but a page of data will automatically get capped when it exceeds 5 MB. Most of the time, the application that you are importing data into will bottleneck the page size value. For example, if you are importing data into NetSuite or Salesforce they each specify (in their API guides) a maximum number of records that can be submitted in any single request.

Data URI template

When your flow runs but has data errors this field can be really helpful in that it allows you to make sure that all the errors in your job dashboard have a link to the original data in the export application. This field uses a handlebars template to generate the dynamic links based on the data being exported. For example, if you are exporting a customer record from Shopify, you would most likely set this field to the following value 'https://your-store.myshopify.com/admin/customers/{{{id}}}'. Or, if you are just exporting a CSV file from an FTP site then this field could simply be one or more columns from the file: {{{internal_id}}, {{{email}}}, etc...

Do not store retry data

Check this box if you do NOT want integrator.io to store retry data for records that fail in your flow. Storing retry data can slow down your flow's overall performance if you are processing very large numbers of records that are failing. Storing retry data also means that anyone with access to your flow's dashboard can see the retry data in clear text.

Override trace key template

Define a trace key that integrator.io will use to identify a unique record. Any value you provide overrides the default trace key for your app. You can specify a single field, such as {{record.field1}}, or use a handlebars expression.For example, {{join "_" record.field1 record.field2}} generates a trace key such as 123_456.

Note

If you have applied a transformation to exported data, reference its fields in the trace key template without the path record. – for example, {{field1}}.

Save your export

Click Save to save your current settings

Click Save & close to save your changes and exit the Create export page.

Click Close to exit the Create export page without saving.