Export files from an FTP server

This article describes how to export files from an FTP server.

Create FTP exports

Start creating an export from an FTP server in either of the following ways:

• From the Tools menu, select Flow builder. Then, click Add source.
      – or –
• From the Resources menu, select Exports. Then, click + Create export.

From the Application drop-down menu, select FTP and your available FTP connection.

Edit FTP export details

Name your export (required): Provide a clear and distinguishable name. You will have the option to choose this export throughout integrator.io, and a unique identifier will prove helpful later when selecting among a list of imports that you’ve created.

Description (optional): Describe your export so that you and others can quickly understand its purpose. Be sure to highlight any nuances that a user should be aware of before using this import in a flow. As you make changes to the resource, be sure to keep this description up to date.

Parse files being transferred: If you need to parse CSV, XML, or JSON files into records before sending them to other applications, select Yes. Select No if you want to transfer the files in the exact same format as they are stored on the FTP server.

Note: FTP exports can transfer any file type if no parsing is required. See How would you like to parse files? for a list of file types that can be parsed during transfer.

How would you like to parse files?

File type (required): Choose the type of file to be exported from the FTP server. For example, choose CSV if you are exporting a flat, delimited text file, or XLSX for a binary Microsoft Excel file. The file type you select changes the fields available on the Create export panel. Acceptable file types include:

• CSV (or any delimited text file)
• EDI X12
• EDIFACT
• Fixed Width
• JSON
• XLSX
• XML

CSV (or any delimited text file)

Sample file (that would be parsed) (required): This field only displays if you choose CSV, JSON, XLSX, or XML as a file type. Choose a sample file to define the record structure. Click Choose file and navigate to a sample version of the files you will be exporting. 

CSV parser helper: The CSV parser helper can be used to visualize and experiment with how integrator.io parses CSV files (or any other delimited text files) into the JSON records/rows that then get processed by your flow. See CSV parser helper.

EDI X12

EDI x12 format (required): Select the EDI X12 file format that matches the files you are exporting from Amazon S3.

EDIFACT

EDIFACT format (required): Select the EDIFACT file format that matches the files you are exporting from Amazon S3.

Fixed Width

Format (required): Select the Fixed width file format that matches the files you are exporting from Amazon S3.

JSON

Sample file (that would be parsed) (required): This field only displays if you choose CSV, JSON, XLSX, or XML as a file type. Choose a sample file to define the record structure. Click Choose file and navigate to a sample version of the files you will be exporting. 

Resource path: You can use this field optionally to define the JSON path to the resources you are exporting from the JSON file with handlebars syntax.

XLSX

Sample file (that would be parsed) (required): This field only displays if you choose CSV, JSON, XLSX, or XML as a file type. Choose a sample file to define the record structure. Click Choose file and navigate to a sample version of the files you will be exporting. 

File has header: Check this box if the files you are exporting contain a top level header row. If the first row of your XLSX file is reserved for column names (and not actual data), then check this box.

XML

Sample file (that would be parsed) (required): This field only displays if you choose CSV, JSON, XLSX, or XML as a file type. Choose a sample file to define the record structure. Click Choose file and navigate to a sample version of the files you will be exporting. 

XML parser helper: The XML parser will give you immediate feedback on how your parse options are applied against raw XML data. See XML parser helper.

Important: While creating an FTP XML import, remember that an unencrypted file size is limited to 250 Mb and encrypted file size is limited to 150 Mb. During the import process, the file size can increase upto 500mb.

Where would you like to transfer from?

Directory path (required): Specify the path of the FTP folder that contains the files to be transferred. integrator.io will transfer all files from the folder path specified, and delete them from the folder once the transfer completes. You can (optionally) configure integrator.io to leave files in the folder, or to transfer files that match a certain starts with or ends with file name pattern.

File name starts with: Use this field to specify a file name prefix to filter which files in the FTP server Directory path will be transferred. For example, if you set this value to test then only files where the name starts with test will be transferred (like test-myFile.csv).

File name ends with: Use this field to specify a file name postfix to filter which files in the FTP server Directory path will be transferred. For example, if you set this value to test.csv then only files where the name ends with test.csv will be transferred (like myFile-test.csv). You must specify the file extension for this filter to work correctly.

How would you like to group and sort records?

The Sorting and Grouping options allow you to manage your flow’s files by sorting and grouping records by field. This feature allows you to aggregate your data based on your chosen field(s), and sort the records in ascending or descending order (ASC or DESC).

Take the following JSON record:

{
  "page_of_records": [
    {
      "record": {
        "NAME": "Sarah",
        "AGE": 97,
        "PURCHASE": "Beach towel",
        "CATEGORY": "Summer"
      }
    },
    {
      "record": {
        "NAME": "John",
        "AGE": 98,
        "PURCHASE": "Ski gloves",
        "CATEGORY": "Winter"
      }
    },
    {
      "record": {
        "NAME": "Ana",
        "AGE": 99,
        "PURCHASE": "Beach ball",
        "CATEGORY": "Summer"
      }
    },
    {
      "record": {
        "NAME": "James",
        "AGE": 100,
        "PURCHASE": "Snowboard",
        "CATEGORY": "Winter"      
       }
    }
  ]
}

Using the example JSON record, group by the Category field. Your data is grouped into Summer and Winter rows. Of course, integrator.io can handle more detailed groupings, say if you want to group records by Category and Purchase, or by Category, Age, and Purchase.

{
  "page_of_records": [
    {
      "rows": [
        {
          "CATEGORY": "Summer",
          "NAME": "Sarah",
          "AGE": 97,
          "PURCHASE": "Beach towel"
        },
        {
          "CATEGORY": "Summer",
          "NAME": "Ana",
          "AGE": 98,
          "PURCHASE": "Beach ball"
        }
      ]
    },
    {
      "rows": [
        {
          "CATEGORY": "Winter",
          "NAME": "John",
          "AGE": 99,
          "PURCHASE": "Ski gloves"
        },
        {
          "CATEGORY": "Winter",
          "NAME": "James",
          "AGE": 100,
          "PURCHASE": "Snowboard"
        }
      ]
    }
  ]
}

In some cases, you might have CSV files:

In this case, you’d group by column (Category), and each row would be grouped into the Summer or Winter category. Logically, it would look something like this:

Tip: Remember that the Page size property under the Advanced section determines how many records/groups can go into a page. If a single group exceeds the page size, that group is discarded from the export, and an error indicating this issue is reported.

Mock Output

You can add mock output data to exports and lookups in the Mock output section of the Edit export panel as simulated output data when configuring a flow.

Advanced

Decrypt files: This option is enabled if your connection is configured to use PGP encryption/decryption.

Decompress files: Set this field to True if you're exporting compressed files.

Leave file on server: If you set this field to True integrator.io will NOT delete files from your FTP server after your export completes. The files will stay on the FTP server, and if the export process runs again, the same files will be exported again.

Backup files path: Enter a value relative to the root directory, if you want to archive a copy of the exported files. For example, to have integrator.io store a copy of the exported file at ftps://ftp.my-site.com/integrations/backup, enter /integrations/backup. This path must already exist on the server.

File encoding: The file encoding indicates how the individual characters in your data are represented on the file system. Depending on the source system of the data, the encoding can take on different formats. If left blank, the default encoding is UTF-8, but you can also choose Windows-1252 or UTF-16LE formats.

Page size: Specify how many records you want in each page of data. The default Page size value (when you leave this field blank) is 20.

Data URI template: This field ensures that all the errors in your job dashboard have a link to the original data in your FTP server. This field uses handlebars templates to generate dynamic links based on the exported data. For example, if you are exporting a CSV file, this field could include one or more columns from the file, such as {{{internal_id}} or {{{email}}}.

Do not store retry data: Check this box if you do NOT want integrator.io to store retry data for records that fail in your flow. Storing retry data can slow down your flow's overall performance if you are processing very large numbers of records that are failing. Storing retry data also means that anyone with access to your flow's dashboard can see the retry data in plain text.

Override trace key template: (optional): Define a trace key that integrator.io will use to identify a unique record. You can use a single field such as {{{field1}}} or use a handlebar expression. For example, this syntax {{join “_” field1 field2}} will generate a trace key template field1_field2. When this field is set, you will override the platform default trace key field.

Batch size: (optional): Enter the maximum number of concurrent files (from 1 to 1000) that can be processed in a single batch request. This setting doesn’t limit the total number of files that you can process in a flow; instead, it allows you to optimize for really big files where bigger batches might cause network timeout errors. If you are importing very small files, increasing the batch size to include several files at a time will improve flow performance.

Save your export

Click Save to save your current settings.

Click Save & close to save your changes and exit the Create export page.

Click Close to exit the Create export page without saving.

Download your data

For FTP exports, integrator.io will store a copy of the data that was consumed by a given job unless you've specifically configured your export to not store a copy of the data. You can access these files by clicking on the Actions ... icon. These can be re-uploaded to the appropriate directory of your FTP server in case you need to process the files again. It will also process any records that you've filtered out again.