Articles in this section

Transfer Celigo Storage files using the Celigo APIs connector

Michelle Welch
Updated

This article describes the configuration options for exporting files from Celigo Storage and importing files into Celigo Storage using the Celigo APIs connector.

Before you begin: Set up a connection to Celigo APIs. The connection scope must be Full access, Custom, or Integration access. To use PGP encryption or decryption, follow the instructions to configure PGP on the connection before setting up the transfer step.

Export Celigo Storage files

General

  • Name your transfer — Name your resource so that you can easily reference it from other parts of the application.
  • Description — Describe your resource so that other users can quickly understand what it is doing without having to read through all the fields and settings. Be sure to highlight any nuances that a user should be aware of before using your resource in their flows. Also, keep this field up to date as you make changes to the resource.
  • Connection — Select an existing Celigo APIs connection, or select Create connection to define a new one. The connection provides the authentication and scope that determines which Celigo Storage folders this export can access.
  • Parse files being transferred — Do you need to parse CSV, XML, JSON, etc. files into records before sending them to other applications, or should the files simply be transferred raw as-is (for example, PDFs or binary files)?

How would you like to parse files?

File type (required): Choose the type of file to be exported from Celigo Storage. The file type you select changes the fields available on the Create export panel. Acceptable file types include:

  • CSV (or any delimited text file)
  • EDI X12
  • EDIFACT
  • Fixed Width
  • JSON
  • XLSX
  • XML

CSV (or any delimited text file)

Sample file (that would be parsed) (required): This field only displays if you choose CSV, JSON, XLSX, or XML as a file type. Choose a sample file to define the record structure. Select Choose file and navigate to a sample version of the files you will be exporting.

CSV parser helper: The CSV parser helper can be used to visualize and experiment with how integrator.io parses CSV files (or any other delimited text files) into the JSON records/rows that then get processed by your flow.

EDI X12

EDI x12 format (required): Select the EDI X12 file format that matches the files you are exporting from Celigo Storage.

EDIFACT

EDIFACT format (required): Select the EDIFACT file format that matches the files you are exporting from Celigo Storage.

Fixed Width

Format (required): Select the Fixed width file format that matches the files you are exporting from Celigo Storage.

JSON

Sample file (that would be parsed) (required): Choose a sample file to define the record structure. Select Choose file and navigate to a sample version of the files you will be exporting.

Resource path: You can use this field optionally to define the JSON path to the resources you are exporting from the JSON file with Handlebars syntax.

XLSX

Sample file (that would be parsed) (required): Choose a sample file to define the record structure. Select Choose file and navigate to a sample version of the files you will be exporting.

File has header: Check this box if the files you are exporting contain a top-level header row. If the first row of your XLSX file is reserved for column names (and not actual data), then check this box.

XML

Sample file (that would be parsed) (required): Choose a sample file to define the record structure. Select Choose file and navigate to a sample version of the files you will be exporting.

XML parser helper: The XML parser will give you immediate feedback on how your parse options are applied against raw XML data.

Where would you like to transfer from?

  • Directory — Choose how to specify the source folder. The Directory field supports three input modes — Browse, Relative path, and Folder ID. Browse and Folder ID use the same underlying reference type; Browse is a visual way to find and select a folder ID. Select Browse to open a visual folder picker loaded from Celigo Storage. The selected folder's ID is written back to the field automatically. Use the search bar in the folder picker to find folders by name within the folder currently being viewed — search is scoped to the current folder, not your entire account. Folders load in batches; if your target folder isn't visible, scroll or search to find it. Folder creation is not available in the folder picker — create folders in Celigo Storage (Resources > File storage) before browsing.
  • Relative path — Enter a path relative to your Celigo Storage root directory. Resolved at runtime. Supports Handlebars expressions so you can build the path dynamically from record data — for example, /inbound/{{record.orderType}}. If the source path doesn't exist when the flow runs, the export step fails.
  • Folder ID — Enter the Celigo Storage folder ID directly. Supports Handlebars expressions for dynamic folder targeting. Folder IDs are stable across renames and moves — renaming or moving a folder in Celigo Storage doesn't break a step that references it by ID. Select Browse to open a visual folder picker; the selected folder's ID is written back to the field automatically, without needing to look it up separately.
  • File filtering conditions — Add one or more conditions (or groups of conditions) to define the criteria used to determine which files are retrieved when the export runs. If no conditions are set, the export retrieves all files in the specified folder. See Create file filtering conditions for file transfer exports.

Sort and group records

The Sorting and Grouping option allows you to manage your flow's files by sorting and grouping records by field. This feature allows you to aggregate your data based on your chosen field(s), and sort the records in ascending or descending order (ASC or DESC).

Sort by

This feature allows you to sort by multiple fields (ASC or DESC) or in the order that the fields were initially entered. You can enter the "Sort records by fields" by a field ID or by using dot notation.

Note:

  • This feature does not sort numeric fields, including dates, in descending or ascending values.
  • Sorting only applies to file provider exports. You can't sort records for HTTP, database, and data warehouse exports or lookups at this time.

Group by

You can group records when parsing files from any file source, or when exporting records from HTTP, databases, or data warehouses. Select the fields you want to group, based on your sample response data. Fields other than basic data types (such as string and number) are unavailable. If the source application does not sort the exported data, then grouping may not work as expected.

Tip: The Page size property under the Advanced section determines how many records or groups can go into a page. If a single group exceeds the page size, that group is discarded from the export, and an error is reported.

Advanced

  • Decrypt files — Use this option to decrypt files using PGP. This option is enabled only when you have configured PGP on the Celigo APIs connection (see Set up a connection to Celigo APIs for information on configuring PGP) and selected the parse files option. When disabled, decryption is not performed when files are parsed.
  • Decompress files — If you're exporting compressed files, set this field to True, and then select a compression format.
  • Leave file on server — Check this box to override the default behavior, in which integrator.io deletes the files from Celigo Storage after transferring them. When selected, the files remain in the folder, and the same files are transferred on the next flow run.

    Note: When running a flow in test mode, integrator.io will not delete files from Celigo Storage, regardless of this setting.
  • File encoding — The file encoding indicates how the individual characters in your data are represented on the file system. The default encoding is UTF-8. Supported formats are UTF-8, Windows-1254, and UTF-16LE. If you do not know the encoding of your data, it is most likely UTF-8.
  • Backup directory — Store backup copies of processed files in a Celigo Storage folder. Leave empty to skip backups. The Backup directory field supports the same three input modes as Directory — Browse, Relative path, and Folder ID. Select Browse to select a folder; the selected folder's ID is written back to the field automatically.
  • Page size — Specifies how many records to include in each page of data when an export runs in the context of a flow. The default is 20. A page is automatically capped when it exceeds 5 MB.
  • Data URI template — Generates dynamic links to the source data in the export application for records that fail, making it easier to trace errors in your job dashboard. Uses a Handlebars template based on the data being exported.
  • Do not store retry data — Check this box if you do not want integrator.io to store retry data for records that fail in your flow.
  • Override trace key template — Define a trace key that integrator.io will use to identify a unique record. Any value you provide overrides the default trace key. You can specify a single field such as {{record.field1}}, or use a Handlebars expression such as {{join "_" record.field1 record.field2}}.

 

Import Celigo Storage files

General

  • Name your transfer — Name your resource so that you can easily reference it from other parts of the application.
  • Description — Describe your resource so that other users can quickly understand what it is doing without having to read through all the fields and settings. Be sure to highlight any nuances that a user should be aware of before using your resource in their flows. Also, keep this field up to date as you make changes to the resource.
  • Connection — Select an existing Celigo APIs connection, or select Create connection to define a new one. The connection provides the authentication and scope that determines which Celigo Storage folders this import can write to.
  • One to many — There are advanced use cases where a parent record is being passed around in a flow, but you actually need to process child records contained within the parent record context. One to many is used in cases where you have a single record that internally needs to create multiple records. This field cannot be used when importing a CSV file.
  • Generate files from records — Select Yes to build a new CSV, XML, JSON, or other file from records in the flow. Select No to transfer raw files as-is (such as PDFs or binary files).

How would you like to generate files?

  • File type — Select the type of file that you want to generate. For example, select CSV if you are importing a flat, delimited text file, or XLSX for a binary Microsoft Excel file. The file name you specify can include a file extension, although it does not affect this file type setting. See Import files to an FTP server for more information.
  • How many files would you like to generate? — One file per flow run or one file per page of records.

Where would you like the files transferred?

  • Directory — Choose how to specify the destination folder. The Directory field supports three input modes — Browse, Relative path, and Folder ID. Browse and Folder ID use the same underlying reference type; Browse is a visual way to find and select a folder ID. Select Browse to open a visual folder picker loaded from Celigo Storage. The selected folder's ID is written back to the field automatically. Use the search bar in the folder picker to find folders by name within the folder currently being viewed — search is scoped to the current folder, not your entire account. Folders load in batches; if your target folder isn't visible, scroll or search to find it. Folder creation is not available in the folder picker — create folders in Celigo Storage (Resources > File storage) before browsing.
  • Relative path — Enter a path relative to your Celigo Storage root directory. Resolved at runtime. Supports Handlebars expressions — for example, /outbound/{{record.orderType}}. If the destination path doesn't exist when the flow runs, the import step creates it automatically.
  • Folder ID — Enter the Celigo Storage folder ID directly. Supports Handlebars expressions for dynamic folder targeting. Folder IDs are stable across renames and moves — renaming or moving a folder in Celigo Storage doesn't break a step that references it by ID. Select Browse to open a visual folder picker; the selected folder's ID is written back to the field automatically.
  • File name — Specify how files written to Celigo Storage should be named. You can type {{{ to include a predefined timestamp template in your file name — for example, FileXYZ-{{{timestamp "YY-MM-DD" "America/Los_Angeles"}}}.txt. You can include any file extension; it does not affect the type of data generated.
  • Overwrite existing file — When checked, if a file with the same name already exists in the destination folder, it is replaced. When unchecked (default), a new file is created alongside the existing one — the new file's name has a number appended (for example, report (1).csv). Consider your intended behavior carefully: leaving this unchecked can accumulate duplicate files over multiple flow runs.

Advanced

  • Encrypt files — Use this option to encrypt files using PGP as they are written to Celigo Storage. This option is enabled only when you have configured PGP on the Celigo APIs connection. When disabled, encryption is not performed.
  • File encoding — The file encoding indicates how the individual characters in your data are represented on the file system. Leave this field blank if you do not know the encoding format. The default encoding is UTF-8. Other supported formats are Windows-1252 and UTF-16LE.
  • Compress files — Check this box to compress files before they are written to Celigo Storage. Once checked, a new setting appears for you to choose the compression format.
  • Backup directory — Store backup copies of processed files in a Celigo Storage folder. Leave empty to skip backups. The Backup directory field supports the same three input modes as Directory — Browse, Relative path, and Folder ID. Select Browse to select a folder; the selected folder's ID is written back to the field automatically.