Transformations allow you to change your source data before it is processed by a flow. You can simplify and modify the data so it's easier for your import to understand.
Note
Transformation 2.0 is not available for an import step.
Level 2: Advanced Flow Builder & Managing Flows > Advanced Flow Builder: Advanced Configurations > Advanced Configuration: Transformation 2.0 & Hooks
Transformations may not always be necessary. For instance, when you have a flow with one source and one destination, it might be easier to define a mapping when sending data, rather than using a transformation. Transformations become necessary when:
-
There is one source and many destination steps: In this case, transformations allow you to simplify the data before it is processed by multiple imports. For example, if you create an export with many fields, objects, arrays within arrays, or other complicated JSON structures, you can use a transformation to flatten or simplify the record so only the needed information is sent to the next steps.
-
There are many sources and one destination: In this scenario, transformations help you modify the data from all the sources so it's only processed once by the destination app. For example, if you are syncing a MongoDB database to Snowflake, you can use transformations to map those source records over to a canonical representation of data that your flow wants to process.
There are four distinct ways to do transformations in integrator.io. Here are the pros and cons of each:
Rules toggle
-
Rules 1.0 Engine (Transformation 1.0): This is the original transformation method in Celigo.
-
Pros: It is a familiar method for those who have been using integrator.io for a long time.
-
Cons: It lacks the advanced functionality and ease of use of the newer Rules 2.0 Engine.
-
-
Rules 2.0 Engine (Transformation 2.0): This is the updated transformation method in integrator.io. It allows for complex transformations and is more user-friendly.
-
Pros: It offers much deeper functionality than the Rules 1.0 Engine, including the ability to process any level of nested arrays. It supports JSON Path for selecting source record fields. It supports handlebar expressions for referencing custom settings fields and building custom field values using formulas. It also supports three different modes based on your use case:
-
make targeted changes to the input record
-
create an output record from the input
-
create output rows from the input
-
-
Cons: It may require some learning for those who are used to the Rules 1.0 Engine.
-
-
JavaScript: If you're proficient with JavaScript, this method allows you to take over the transformation process completely in your code.
-
Pros: It provides full control over the transformation process and is ideal for developers who prefer coding.
-
Cons: It requires knowledge of JavaScript and may not be suitable if you're not familiar with coding.
-
-
JavaScript preSavePage hooks: This method (
-
Pros: It allows you to work with larger data structures and perform operations such as merging records, deleting records, creating more records within the page, and sorting data.
-
Cons: It requires knowledge of JavaScript and may be more complex to use than the other methods. It also requires comfort with working with larger data structures.
-
There are three modes you can use in Transformation 2.0. The modes are available in a dropdown near the top, left-hand side of the screen.
Three modes
This mode is useful when you want to make simple targeted changes to source records but keep everything else in the source records as-is.
This mode is helpful when creating a new output record from the source record. It allows you to define the output record from scratch, so you can pick and choose only the fields you need from the source record. It's also useful when you want to transform the data from multiple different sources into a canonical format that the flow can then process, making it easier to add new sources in the future.
There are four transformation field types:
-
Standard: This is the default transformation type. Use it when you want to map a field directly from the source record to the output record.
-
Hard-coded: This is useful when you want to set a specific, static value for a field in the output record.
-
Static lookup: This is used when defining a table containing input-to-output value mappings. For example, every time you see "apple" change it to "banana".
-
Handlebars expression: This is useful for creating a custom value for a field in the output record using a handlebars template. This option allows you to build any type of output and is particularly useful when you want to concatenate fields, multiply numbers, source from custom settings in the flow, or perform any other complex processing on input field values.
Field types
The supported output and input data types include string, number, boolean, object, string array, number array, and object array. Recursive definitions are also supported.
The transformation 2.0 engine automatically converts data types where possible. For example, if a number is mapped to a string, the platform will automatically convert the number to a string. However, some data types are not compatible with each other, and this will cause the mapping to fail. See how all the different input and output data types are mapped.
The Action to take if the input field has no value field allows you to configure a custom value, null, or an empty string.
The Settings description field allows you to document why you're making certain settings for a field mapping. Here, you can add comments or notes about the transformation.
JSON path is used in the transformation 2.0 engine to select fields from the source record. It is a standard framework that includes documentation and examples. The expressions can be simple or complex and they support a wide range of querying options including array slices and recursive descent.
Handlebar expressions can be used whenever you need to generate an output field that is not just a single field from the source record. It's useful if you need to concatenate fields, multiply numbers, encode or decode strings, format multiple fields uniquely, or source values from custom settings in the flow.
Click the refresh () button to display recent changes to the input sample data when transforming fields.
This displays all fields with all subfields expanded () or top-level fields with all subfields collapsed.
Click the magnifying glass to open search mode to find output fields by name. Transformations 2.0 displays and highlights output fields with names that match the search criteria. Adding a new row while in search mode closes search mode.
Automatically generate the output JSON structure with the output expected field names. Users can then transform each JSON element into its corresponding input field name.
Use Preview and Auto-preview to ensure that the transformations and mappings are working as expected. Testing and previewing allows you to verify your results and mappings as they're built. For example, when creating a new field, use Preview to see the exact output based on the input record you're working with. This allows you to make sure that the field is working exactly as you want it to.
Transformation 2.0 and Mapper 2.0 use the same engine and have almost identical interfaces. They're both used to define mappings and transformations for data, which are foundational capabilities for building successful flows.
With Transformation 2.0, you can change the source data before it gets processed by the flow. On the other hand, Mapper 2.0 takes the flow's data and transforms it into the final form that import applications accept.
These two features are the most commonly used in flows. Mastering one feature makes it easier to understand and use the other since they are very similar.
Create string array sourcing from multiple different places in the input.
Create object array sourcing from multiple different places in the input.
Use a static lookup to translate values between input and output records.
When you configure the output fields, you need not provide prompts. In Transformation 2.0, after adding the fields, you can select one or more fields and click to auto-transform the output fields.
You can auto-transform all fields, a specific field, or multiple specific fields. When you select a parent-level field, all the fields below that parent are automatically selected. Auto-transform doesn’t overwrite the input field values that already exist or are entered manually. After you select the fields, click Auto-transform output fields. It auto-transforms all selected un-mapped input fields, that is the ones that do not have the input field added.
When auto-transform is clicked, all options that modify the existing output structure will be disabled. Close auto-map to go back to mappings.
Note
The option is enabled only when you add a new output field with a value. After the fields are mapped, the Auto-transform destination fields option is disabled.
For more information, see Auto transform output fields using Celigo AI.
Comments
Please sign in to leave a comment.