This article describes how to export files from Amazon S3.
Create Amazon S3 exports
Start creating an export from Amazon S3 in either of the following ways:
- From the Tools menu, select Flow builder. Then, click Add source.
– or – - From the Resources menu, select Exports. Then, click + Create export.
From the Application drop-down list, select Amazon S3 and your available Amazon S3 connection.
Edit Amazon S3 export details
Name (required): Provide a clear and distinguishable name. You will have the option to choose this export throughout integrator.io, and a unique identifier will prove helpful later when selecting among a list of imports that you’ve created.
Description (optional): Describe your export so that you and others can quickly understand its purpose. Be sure to highlight any nuances that a user should be aware of before using this import in a flow. As you make changes to the resource, be sure to keep this description up to date.
How would you like to parse files?
File type (required): Choose the type of file to be exported from Amazon S3. For example, choose CSV if you are exporting a flat, delimited text file, or XLSX for a binary Microsoft Excel file. The file type you select changes the fields available on the Create export panel. Acceptable file types include:
CSV (or any delimited text file)
Sample file (that would be parsed) (required): This field only displays if you choose CSV, JSON, XLSX, or XML as a file type. Choose a sample file to define the record structure. Click Choose file and navigate to a sample version of the files you will be exporting.
CSV parser helper: The CSV parser helper can be used to visualize and experiment with how integrator.io parses CSV files (or any other delimited text files) into the JSON records/rows that then get processed by your flow. See CSV parser helper.
EDI X12
EDI x12 format (required): Select the EDI X12 file format that matches the files you are exporting from Amazon S3.
EDIFACT
EDIFACT format (required): Select the EDIFACT file format that matches the files you are exporting from Amazon S3.
Fixed Width
Format (required): Select the Fixed width file format that matches the files you are exporting from Amazon S3.
JSON
Sample file (that would be parsed) (required): This field only displays if you choose CSV, JSON, XLSX, or XML as a file type. Choose a sample file to define the record structure. Click Choose file and navigate to a sample version of the files you will be exporting.
Resource path: You can use this field optionally to define the JSON path to the resources you are exporting from the JSON file with handlebars syntax.
XLSX
Sample file (that would be parsed) (required): This field only displays if you choose CSV, JSON, XLSX, or XML as a file type. Choose a sample file to define the record structure. Click Choose file and navigate to a sample version of the files you will be exporting.
File has header: Check this box if the files you are exporting contain a top level header row. If the first row of your XLSX file is reserved for column names (and not actual data), then check this box.
XML
Sample file (that would be parsed) (required): This field only displays if you choose CSV, JSON, XLSX, or XML as a file type. Choose a sample file to define the record structure. Click Choose file and navigate to a sample version of the files you will be exporting.
XML parser helper: The XML parser will give you immediate feedback on how your parse options are applied against raw XML data. See XML parser helper.
Where would you like to transfer files from?
Region (required): Select the name of the Amazon S3 location where the request is being made. If not set, this field defaults to US East (N. Virginia) [us-east-1].
Bucket name (required): Specify the top level Amazon S3 folder. Amazon S3 bucket names cannot contain forward slashes or nested folder structures. If you have a nested folder structure in the bucket (such as folder1/subfolderA/topic1), the bucket name is folder1. Enter the remaining subfolderA/topic1 in the Prefix field. Click the pencil icon to the right of this field to create a handlebars template for bucket names.
Prefix: Enter the prefix (similar to a folder structure within the Amazon S3 bucket) to transfer only the files contained in that location. For example, if the full path is "BucketName/Project/WordFiles/", then a prefix is "Project/WordFiles/". Partitions within the prefix support up to 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second. If left blank, all files will be picked for transfer. Refer to Amazon S3 documentation for more details.
File filtering conditions: See Create file filtering conditions for FTP exports.
How would you like to group and sort records?
The Sorting and Grouping drop-down menu allows you to manage your flow’s files by sorting and grouping records by field. This feature gives you the ability to group records by field (column1, column2, column3…), and sort the records in ascending or descending order (ASC or DESC).
Advanced
Decrypt files: This option is enabled if your connection is configured to use PGP encryption/decryption.
Decompress files: Set this field to True if you're exporting compressed files.
Leave file on server: If you set this field to True integrator.io will NOT delete files from your FTP server after your export completes. The files will stay on the FTP server, and if the export process runs again, the same files will be exported again.
Backup bucket name: Provide the top-level Amazon S3 folder. Celigo recommends providing the Amazon S3 bucket name. It will use the source bucket’s folder path within the backup bucket to create the backup. For example, if a file is exported from bucket A at the path x/y/z and the backup bucket is B, it will be backed up at the x/y/z path in B.
File encoding: The file encoding indicates how the individual characters in your data are represented on the file system. Depending on the source system of the data, the encoding can take on different formats. If left blank, the default encoding is UTF-8, but you can also choose Windows-1252 or UTF-16LE formats.
Page size: Specify how many records you want in each page of data. The default Page size value (when you leave this field blank) is 20.
Data URI template: This field ensures that all the errors in your job dashboard have a link to the original data in your FTP server. This field uses handlebars templates to generate dynamic links based on the exported data. For example, if you are exporting a CSV file, this field could include one or more columns from the file, such as {{{internal_id}} or {{{email}}}.
Override trace key template: (optional): Define a trace key that integrator.io will use to identify a unique record. You can use a single field such as {{{field1}}} or use a handlebar expression. For example, this syntax {{join “_” field1 field2}} will generate a trace key template field1_field2. When this field is set, you will override the platform default trace key field.
Save your export
Click Save to save your current settings
Click Save & close to save your changes and exit the Create export page.
Click Close to exit the Create export page without saving.
Comments
Please sign in to leave a comment.