Articles in this section

Configure a schema drift policy

Stephen Brandt
Updated

The schema drift policy defines how the sync will handle changes to the source data structure. With an active metadata approach, the sync automatically detects changes and adjusts the data ingestion processes and target schema, ensuring continuous and accurate operation.

SchemaDriftPolicy.png

Celigo syncs offer you granular control over schema drift in the following priority:

  1. A field’s Schema structure setting overrides its parent object’s or export’s Schema policy.

  2. An object’s or export’s Schema policy, if any, overrides the source application’s Schema drift policy.

  3. The source application’s global Schema drift policy applies to all of its objects and exports, unless they have more specific settings.

When a table/object is

Added

  • Choose Automate changes to allow the Celigo platform to automatically add the table to the destination application.

  • Choose Ignore if you don’t want to change your sync and have it take any actions when a table is added.

This setting applies only at the source application-level Schema drift policy when you are extracting objects (Ignore is unavailable for endpoints that have only exports to select).

When a column/field is

Added

  • Choose Automate changes to allow the Celigo platform to automatically add the column or field to the destination application.

  • Choose Ignore if you don’t want this sync to take any action when a column or field is added to the source data.

Removed

The required selection Automate changes instructs the sync to mark soft and hard deletes as removed in a new column, leaving your archived data intact.

Reduced

The required selection Automate changes handles data type or length changes, which reduce the field size, detected in the source schema.

Expanded

The required selection Automate changes handles data type or length changes, which increase the field size, detected in the source schema. For example, when identifying a Salesforce text area, the schema’s data type is set to VARCHAR(MAX) in order to account for longer historical data and avoid any errors in the sync.

Schema structure

When the new field is an object

  • Choose Preserve to import the JSON object ({"key1": "value", "key2"...}) as a field value.

  • Choose Flatten to add the nested fields and values to the parent’s table as additional rows at the same level.

When the new field is an array

  • Choose Preserve to import the JSON array (["value1", "value2"...]) as a field value.

  • Choose Normalize to add the nested fields and values as additional tables in the destination application.