Articles in this section

Import files to Google Cloud Storage

Michelle Welch
Updated

This article describes how to import files and records to Google Cloud Storage (GCS).

Create a Google Cloud Storage import

Start creating an import to Google Cloud Storage in either of the following ways:

  • From the Build menu, select Flows > Create flow. Then, click Add destination/lookup.

  • From the Resources menu, select Imports. Then, click + Create import.

From the Application drop-down list, select Google Cloud Storage and your available Google Cloud Storage connection.

When prompted, select the type of import you want to create:

Transfer files into Google Cloud Storage

General

Name (required): Provide a clear and distinguishable name. You will have the option to choose this import throughout integrator.io, and a unique identifier will prove helpful later when selecting among a list of imports that you've created.

Description (optional): Describe your import so that you and others can quickly understand its purpose. Be sure to highlight any nuances that a user should be aware of before using this import in a flow. As you make changes to the resource, be sure to keep this description up to date.

Connection (required): The Google Cloud Storage connection to use for your import.

One to many: Set to Yes (advanced) to send each record to multiple destinations. Defaults to No.

Generate files from record: Choose Yes to build a new file (CSV, JSON, XML, and so on) from record data. Choose No to transfer raw files as-is (PDF, images, and so forth).

How would you like to generate files?

File type (required): Select the type of file that you want to generate. For example, choose CSV if you are importing a flat, delimited text file, or XLSX for a binary Microsoft Excel file. The file name you specify can include a file extension, but it does not affect this setting. You can then choose a sample file to define the record structure. For example, if you are importing CSV files (i.e. files containing Comma Separated Value data), choose CSV. Each file type includes additional settings to customize your import. Acceptable file types include:

  • CSV (or any delimited text file)

  • EDI X12 

  • EDIFACT 

  • Fixed width 

  • JSON 

  • XLSX 

  • XML 

Based on the file type, the settings you see will differ. See the settings in a file type from the grouped sections below.

CSV (or any delimited text file), XLSX

The CSV and XLSX file types have similar settings for this section. The CSV file type has more settings that you can configure.

generate-files-csv-FTP.png

Sample file (that would be generated): If you have a sample file that represents the record structure of the files to transfer, you can upload the sample file here. We will use the sample file to auto-set various fields (where possible), and also help you map data in a subsequent step. The sample file that you provide does not need to be overly large (at max 5 records), but it should contain all the fields that you want to work with, and also be in the same format that the transfer will need to generate when running in a production capacity.

Include header: Check the box to

CSV: set the very first row in your CSV file as column names so that anyone reading the CSV file can quickly understand what each column represents). It's selected by default.

XLSX: (optional) set the header in your XML file that specifies the XML version number, and optionally the character encodings, as the XML declaration on the first line of the document.

How many files would you like to generate?: (optional) Select how the flow should generate output files. Select One file per flow run to create a single file that contains all of the records processed during the flow run. If the flow includes more than one export, one file will be generated for each export. For real-time flows, a file is created using all of the records received in the current request. Select One file per page of records to generate a separate file for each page of records processed during the flow run. 

CSV generator helper: (optional – CSV only) The CSV generator helper can be used to visualize and experiment with how integrator.io builds CSV files (or any other delimited text files) from the JSON records/rows being processed by your flow.

Column delimiter: (optional – CSV only) Select the delimiter that separates fields (or values) in a record. While comma is the default separator, it could also be any of the other options provided in the list.

Row delimiter: (optional – CSV only) Select the delimiter that separates the end of a row (or record). While the default separator is line feed or new line, it could also be any of the other options provided in the list.

Replace new line with space: (optional for CSV only) Check this box only if you want to replace new lines in the content of the data (except rowDelimiters) with a space.

Replace tab with space: (optional – CSV only) Check this box oly if you want to replace tabs in the content of the data (except columnDelimiters) with a space.

Truncate last row delimiter (optional – CSV only) Check this box only if you want to prevent errors that may occur if a string value in the CSV file exceeds the maximum allowed length.

Wrap with quotes: (optional – CSV only) Check this box only if you want to wrap all the values in the file with quotes.

Custom header rows:(optional – CSV only) In some rare cases, it is necessary to include one or more custom header rows before the actual CSV data. Use this field to add any such prefix rows.

See also File generation for structured data in CSV and XLSX.

JSON, XML

The E and XML file types have the same settings for this section.

Import1.png

Sample file (that would be generated): If you have a sample file that represents the record structure of the files to transfer, you can upload the sample file here. We will use the sample file to auto-set various fields (where possible), and also help you map data in a subsequent step. The sample file that you provide does not need to be overly large (at max 5 records), but it should contain all the fields that you want to work with, and also be in the same format that the transfer will need to generate when running in a production capacity.

How many files would you like to generate?: (optional) Select how the flow should generate output files. Select One file per flow run to create a single file that contains all of the records processed during the flow run. If the flow includes more than one export, one file will be generated for each export. For real-time flows, a file is created using all of the records received in the current request. Select One file per page of records to generate a separate file for each page of records processed during the flow run. 

Fixed Width

Fixed width file type has the same settings for this section.

generate-files-fixed-with-FTP.png

Format: Select the document type for a trading partner.

File generator helper: Launch the file generator helper to use file definition rules and check the files generated. The file generator helper allows you to modify the templated rules to solve more specific file generation requirements. Within the generator helper, the rules panel specifies and generates EDI files from the JSON records/rows being processed by your flow. You can use the sample flow data and generated file panels to test your rules.

Where would you like files transferred?

Bucket name (required): Enter the name of the Google Cloud Storage bucket where you want to upload files. You can also select a bucket from the drop-down list, which is populated from your connection's project. If the list cannot be retrieved, type the bucket name directly. You can use handlebars syntax to set the bucket name dynamically.

File name (required): Enter the full object name (path) for the uploaded file, including any folder path — for example, raw_json/orders_{{timestamp}}.json. You can use handlebars syntax to generate dynamic file names.

Tip

You can type '{{{' to use handlebars to include a predefined timestamp or random unique file identifier (UUID) in your file name.

Timestamp handlebar example: If you type FileXYZ-{{{timestamp "YY-MM-DD" "America/Los_Angeles"}}}.txt , the files will be named with the pattern: FileXYZ-16-06-30.txt.

Verify that your timestamp format is filesystem compatible. Incompatible ASCII characters include:

  • < (less than)

  • > (greater than)

  • : (colon)

  • " (double quote)

  • / (forward slash)

  • \ (backslash)

  • | (vertical bar or pipe)

  • ? (question mark)

  • * (asterisk)

Random handlebar example: If you type FileXYZ-{{{random "UUID"}}}.txt , the file will be named with the following pattern: FileXYZ-69368e91d9a440f79165b73afd46859d.txt generating a unique ID (UUID) of the file. The length of the UUID defaults to 32 if left unspecified, so typing FileXYZ-{{{random "UUID" 16}}}.txt would limit the UUID to 16 characters.

You can also include whatever file extension you like, and the file extension will never change the type of data generated.

Name collision handling: Defaults to Overwrite existing file. GCS does not support duplicate file names within the same bucket and path, so overwrite is the only available mode. To ensure each upload is unique, add a dynamic value such as {{timestamp}} to your File name.

Advanced

Setting

Description

Compress files

Compresses your import data as a .zip file before uploading to GCS.

Encrypt files

Use this cryptographic system to encrypt generated files before uploading. This option is enabled only when your connection is configured with at least one PGP cryptographic system.

File encoding

Select the character encoding for generated files. Defaults to UTF-8. Other supported values include Windows-1252 and UTF-16LE.

Mime type

Select the MIME type for the uploaded file. Defaults to Do not override, which causes integrator.io to send the MIME type matching the selected File type automatically — for example, JSON files are uploaded as application/json and CSV files as text/csv. If no file type is selected, GCS uses application/octet-stream. Select a specific MIME type or enter a custom value only if you need to override this behavior. Supports handlebars syntax for dynamic values.

Custom metadata

Add name/value pairs to attach as custom metadata to the uploaded GCS object. Each pair is stored as an x-goog-meta-* header on the object and can be used for downstream categorization, compliance tagging, or lifecycle policies. You can use handlebars syntax in the value field to reference dynamic data — for example, name Environment, value {{{environment}}}. Handlebars is supported on values only, not on names.

Blob key path

Enter the path to the blob key in the record payload when transferring raw files without file generation.

Import records into Google Cloud Storage

General

Name (required): Provide a clear and distinguishable name.

Description (optional): Describe your import so that you and others can quickly understand its purpose.

Connection (required): The Google Cloud Storage connection to use for your import.

What would you like to import?

Resource (required): Select the GCS resource to act on. The only accepted value at launch is Objects.

API endpoint (required): Select the operation to perform. Available endpoints for Objects:

  • Delete object — deletes a single object from a bucket. Requires Bucket and Object name. Both fields support handlebars syntax.

  • Copy object — copies an object to a new location. Requires Source bucket, Source object name, Destination bucket, and Destination object name. Leave Destination bucket blank to copy within the source bucket. All fields support handlebars syntax.

Save your import

  • Click Save to save your current settings.

  • Click Save & close to save your changes and exit the page.

  • Click Close to exit without saving.

Related articles