Articles in this section

Create a sync from an application’s exports

Michelle Welch
Updated

When object-level selection isn't supported or doesn't offer enough coverage, syncing via Exports gives you maximum flexibility for custom and complex data extraction in Celigo sync. You can create new exports or reuse existing ones from any connected third-party application. The following instructions walk you through connecting a source application, configuring schema drift policies, selecting exports, and defining field-level settings for sync ingestion.

Set up an export-based source sync

An export is a reusable Celigo resource that defines how data is pulled out of an application — the endpoint or query to call, any filters, and the shape of the results. Because exports work with any source you can connect to, they're how you sync data from applications that don't support object-based selection, and how you get precise control over the records returned when the available objects are too broad, too narrow, or don't exist. If you already have exports built for your flows, you can reuse them here.

Note

Webhooks and AS2 exports aren't supported as sync sources and won't appear in the list.

To create a sync, you need to configure your source data, select your target destination, and set a run schedule.

  1. Go to Home home-icon.png, select Create, and then select Sync.
    create-sync.png

  2. Choose the integration that will hold the sync. Choose one with the same destination as your new sync, or select Create integration to create a new one (see Create or open an integration).

    Note

    An integration can only contain syncs that share the same destination application.

  3. Select Next to open the Create sync wizard.
  4. Follow the steps to choose a source and connect to it (see Create a data ingestion sync).
  5. Set the schema drift policy (see Configure a schema drift policy).
    schema-drift-policy.png
  6. If your connection has options other than exports, select Exports.
    netsuite-exports.png

    • If source application metadata isn't available for an export, Celigo sync provides intelligent defaults: it inspects sample request data to infer each column's data type (such as mapping created_at to DATETIMENTZ).

      Tip

      For data ingestion exports, retrieve as much context as possible by using a List or Get all API endpoint or writing an inclusive SQL statement (for example, pulling all Zendesk tickets without extra parameters). Choose Export type: All for complete datasets or Delta for incremental syncs.

  7. Optionally, add an field-level schema policy override (see Configure a schema drift policy).
  8. Select each export you want to sync, or select the checkbox in the header row to select all.
  9. Select an export's name to configure its modes and field mappings. From here, you can do the following (some options may differ based on your connection type):

    • Select Edit next to the Export to adjust the export's configuration.

      edit-export.png
    • Enter a Destination table name. Celigo sync provides intelligent defaults: it derives initial table names directly from the export name (for example, get_shopify_customers from "Get Shopify customers").
    • Select an Ingestion mode (Replace, Append, or Merge). Note that there is no Extract mode setting—to change how data is pulled, edit the export and select All or Delta as the Export type.
    • Choose which fields or sub-fields you want to include with each sync.
    • Select a Schema structure based on your target data type. Use Preserve for both nested objects and arrays to keep the existing parent-child hierarchy. To convert child records into individual rows alongside parent data, select Flatten (for nested objects) or Normalize (for arrays). Learn more about schema structures.

    • Select Primary key key-icon.png to set the primary key, or mark additional fields for a composite key.

      Note

      When merging tables, every source export requires at least one primary key. If you normalize a child field (preserving the array), you must set a primary key at each normalized array level for its destination table. If the child field is flattened, a primary key can be assigned at any level of the hierarchy.

    • Select Mask mask.png in a field to prevent sending sensitive values, such as PII, to the destination. See Mask sensitive fields during a sync.
    • Data types are read-only, they are based on the schema Celigo sync infers.
  10. Once you've configured your source settings and mapped your fields, select Next to save your progress and set up your destination. See the following:

Mask sensitive fields during a sync

Masking lets you protect sensitive source data, such as social security numbers or email addresses, before it's loaded into your destination. Celigo sync masks a field by hashing its value, so the data remains consistent and joinable across your destination tables, but is no longer human-readable.

How masking works

  • Celigo sync hashes masked values using SHA-256 and encodes them in hexadecimal format. Hashing produces the same output for the same input, so masked values can still be used for joins across destination tables, or to identify when the same source value appears in multiple rows, without exposing the original value.
  • Masking happens within Celigo, before data is loaded. Values are never sent to the destination unmasked and then hashed there.
  • When you mask a field, its destination data type changes to string, even if the source field's original data type was numeric.

What can't be masked

  • Fields marked as a primary key can't be masked.
  • Array and object fields can't be masked directly.

Learn more