Articles in this section

Import or transfer files into Amazon S3

An Amazon S3 import generates files from the records in your flow and transfers them to a bucket in Amazon S3. Use it to send data such as orders, customers, or inventory to S3 as JSON files, either one file per flow run or one file per record, so that downstream systems, data lakes, and analytics tools can pick them up.

This article describes how to create an Amazon S3 import or transfer step in your flow, and explains every field on the Create import and Create transfer forms, which have almost the same settings.

Before you begin

  • Create an Amazon S3 connection and make sure it is Online. If the connection shows Offline, a warning banner appears at the top of the form and configuration is limited until you fix it.
  • Have the name of the destination bucket ready, including any folder path inside it.
  • Confirm that the credentials used by the connection are allowed to write objects to that bucket. If you plan to use a backup bucket or server-side encryption, confirm they are allowed to do so as well.
  • Decide which file type you need and how many files should be created for each flow run.

Create an Amazon S3 import or transfer

  1. From the Tools menu, click Flow builder > Add destination/lookup. Or, from the Resources menu, click Imports > + Create import.
  2. From the Application list, select Amazon S3.
  3. Create or select an Amazon S3 connection.
  4. If asked, click Transfer files into destination application and create a transfer or use an existing transfer. Use a transfer step when you want to move files to Amazon S3, either files that your flow generates from records or files that already exist, such as PDFs and images, which are transferred as they are.
  5. Complete the sections described below. The Create import and Create transfer forms have the same fields, except that Generate files from record appears only on the Create transfer form: General, How would you like to generate files?, Where would you like the files transferred?, and Advanced.
  6. Click Save or Save & close.

General

Field Description
Name your import (required) Enter a clear, distinguishable name, for example Orders to S3. A unique name makes the import easy to find when you choose it from a list of imports later.
Description Optional. Describe the purpose of the import and note anything another user should know before using it in a flow. Keep the description up to date as you change the import.
Connection (required) The Amazon S3 connection to use. The connection holds the credentials and other information needed to access your bucket. Click Edit to change the connection's settings. A red Offline label means the connection is not working; click Fix your connection in the banner to bring it back online.
One to many (required) Select Yes (advanced) if a single record creates multiple records, for example when a parent record contains child records that you need to process. Otherwise, keep No. This setting cannot be used when importing a CSV file.
Generate files from record (transfer only) Select Yes to build a new CSV, XML, JSON, or other file from your records. Select No to transfer raw files as they are (PDF, JPG, and so on). This setting appears only on the Create transfer form.

How would you like to generate files?

File type (required): Select the type of file that you want to generate. For example, choose CSV if you are importing a flat, delimited text file, or XLSX for a binary Microsoft Excel file. The file name you specify can include a file extension, but it does not affect this setting. You can then choose a sample file to define the record structure. Each file type includes additional settings to customize your import. Acceptable file types include:

  • CSV (or any delimited text file)
  • EDI X12
  • EDIFACT
  • Fixed width
  • JSON
  • XLSX
  • XML

The settings you see depend on the file type. See the group that matches your file type below.

CSV (or any delimited text file), XLSX

The CSV and XLSX file types have similar settings in this section. The CSV file type has more settings that you can configure.

Field Description
Sample file (that would be generated) If you have a sample file that represents the record structure of the files to transfer, you can upload it here. Celigo uses the sample file to auto-set various fields (where possible) and to help you map data in a subsequent step. The sample file does not need to be large (5 records at most), but it should contain all the fields that you want to work with and be in the same format that the import needs to generate in production.
Include header Select the checkbox to include a header. For CSV, this sets the first row of the file to column names so that anyone reading the file can quickly see what each column represents. It is selected by default. For XLSX, this setting is optional.
How many files would you like to generate? (optional) Select how the flow generates output files. Select One file per flow run to create a single file that contains all of the records processed during the flow run. If the flow includes more than one export, one file is generated for each export. For real-time flows, a file is created using all of the records received in the current request. Select One file per page of records to generate a separate file for each page of records processed during the flow run.
CSV generator helper (optional, CSV only) Use the CSV generator helper to visualize and experiment with how Celigo builds CSV files (or any other delimited text files) from the JSON records or rows that your flow processes.
Column delimiter (optional, CSV only) Select the delimiter that separates fields (or values) in a record. Comma is the default, but you can choose any of the other options in the list.
Row delimiter (optional, CSV only) Select the delimiter that marks the end of a row (or record). The default is line feed (new line), but you can choose any of the other options in the list.
Replace new line with space (optional, CSV only) Select this checkbox only if you want to replace new lines in the content of the data (except row delimiters) with a space.
Replace tab with space (optional, CSV only) Select this checkbox only if you want to replace tabs in the content of the data (except column delimiters) with a space.
Truncate last row delimiter (optional, CSV only) Select this checkbox only if you want to prevent errors that may occur if a string value in the CSV file exceeds the maximum allowed length.
Wrap with quotes (optional, CSV only) Select this checkbox only if you want to wrap all values in the file with quotes.
Custom header rows (optional, CSV only) In some rare cases, you need to include one or more custom header rows before the actual CSV data. Use this field to add those prefix rows.

EDI X12, EDIFACT

The EDI X12 and EDIFACT file types have the same settings in this section. The settings apply if you have a B2B Manager subscription.

Field Description
EDI profile Select an existing EDI profile, or click the plus sign (+) to create one. An EDI profile in Celigo is a centralized configuration that maps and enforces the rules, guidelines, and document structures for a specific trading partner (such as a retailer, distributor, or 3PL). It acts as a single source of truth to translate and validate business documents (like 850s, 856s, and 810s) between your ERP and your trading partner.
Generic definition Select a document type to convert the canonical data to an EDI document to use in your flow's import.
EDI generator helper Launch the EDI generator helper to use file definition rules (v2) and check the EDI files generated. The helper lets you modify the templated rules to solve more specific file generation requirements. In the helper, the rules panel specifies and generates EDI files from the JSON records or rows that your flow processes. You can use the sample flow data and generated file panels to test your rules.
How many files would you like to generate? (optional) Select how the flow generates output files. Select One file per page of records to generate a separate file for each page of records processed during the flow run. Select One file per key to generate a separate file for each document type defined in the ST segment (Transaction Set Header) during the flow run.

Fixed width

Field Description
Format Select the document type for a trading partner.
File generator helper Launch the file generator helper to use file definition rules and check the files generated. The helper lets you modify the templated rules to solve more specific file generation requirements. In the helper, the rules panel specifies and generates files from the JSON records or rows that your flow processes. You can use the sample flow data and generated file panels to test your rules.
How many files would you like to generate? (optional) Select how the flow generates output files. Select One file per page of records to generate a separate file for each page of records processed during the flow run. Select One file per key to generate a separate file for each document type defined in the ST segment (Transaction Set Header) during the flow run.

JSON, XML

The JSON and XML file types have the same settings in this section.

Field Description
Sample file (that would be generated) If you have a sample file that represents the record structure of the files to transfer, you can upload it here. Celigo uses the sample file to auto-set various fields (where possible) and to help you map data in a subsequent step. The sample file does not need to be large (5 records at most), but it should contain all the fields that you want to work with and be in the same format that the import needs to generate in production.
How many files would you like to generate? (optional) Select how the flow generates output files. Select One file per flow run to create a single file that contains all of the records processed during the flow run. If the flow includes more than one export, one file is generated for each export. For real-time flows, a file is created using all of the records received in the current request. Select One file per page of records to generate a separate file for each page of records processed during the flow run.

Where would you like the files transferred?

Field Description
Region (required) The AWS region where your bucket lives, for example US East (N. Virginia) [us-east-1]. It must match the bucket's region.
Bucket name (required) The name of the destination bucket. To place files in a folder, add the path, such as MySite/Orders. Files are transferred into the last folder in the path. Click { } to insert dynamic values.
File key (required) How the files are named in S3. The default is file-{{timestamp "x"}}.json, which adds a timestamp so each name is unique. You can include a predefined timestamp or a unique file identifier (UUID) in the name, and any extension you like. The extension never changes the type of data generated.
JSON document (optional) or XML document (required) A JSON or XML template that defines the content of each file. Select { } to open the handlebars editor, then enter the template to define the structure of the JSON or XML file. This setting appears only if you select JSON or XML as the File type in the How would you like to generate files? section. The template is optional for JSON and required for XML. For template patterns and examples, see Generate JSON or XML files with handlebars for transfers.
Use server-side encryption (SSE-S3) Select this checkbox to have Amazon S3 encrypt the files at rest.

Tip: If two files are written with the same file key, the later file overwrites the earlier one. Keep a timestamp or record ID in the file key to avoid this.

Advanced

Field Description
Encrypt files Optional. Encrypts the generated files using a cryptographic system. This option is available only when at least one cryptographic system is configured in the connection. If it is cleared, the files are not encrypted.
Backup bucket name Optional. The bucket (and folder path, if needed) where files are backed up after a successful transfer. Click { } to use dynamic values.
File encoding Optional. How the characters in your data are represented. Leave it blank if you are not sure. The default is UTF-8; the other supported formats are Windows-1252 and UTF-16LE.
Compress files Optional. Compresses the imported data into a .zip file.

Save and test

Click Save to keep the import and continue editing, Save & close to save and return to your flow, or Close to discard unsaved changes. Run the flow with a small set of records, then open your bucket and confirm the file appears with the expected name and content.

Troubleshooting

  • Connection shows Offline: Open the connection, re-enter your credentials, and test it again. Required fields on this form stay limited until the connection is online.
  • Access denied errors: Check that the credentials have permission to write to the bucket and folder path, and that the region matches the bucket.
  • Files overwritten: Add a timestamp or record ID to the File key so each name is unique.