Celigo sync is a data ingestion pipeline that moves operational data from source applications into an analytical destination, such as Snowflake, Google BigQuery, or NetSuite Analytics Warehouse (NSAW), at scale. It extends Celigo's intelligent automation platform with an app-driven data ingestion experience that reduces the configuration typically required to bring data into a data warehouse.
Prerequisites:
All Celigo subscriptions include one active sync. Contact your account manager for details on Data Ingestion add-ons and volume packs.
What is data ingestion?
Data ingestion is the process of extracting data from source systems (apps, databases, files) and loading it into an analytics target (a data warehouse or data lake) so it can be queried and analyzed. Celigo sync is Celigo's data ingestion pipeline; this section explains the broader concept of data ingestion before showing how Celigo sync fits into it.
Data ingestion vs. data integration
Traditional data integration requires writing custom transformation logic for every table, taking weeks to build. Ingestion tools standardize the extract and load steps, allowing you to set up multi-table pipelines in minutes.
- Data integration - Moves, combines, and transforms data across systems (harmonizing fields, creating aggregates, and restructuring objects).
- Data ingestion - A specific subset of data integration focused exclusively on the extract and load phases. It does not transform, harmonize, or restructure data; it moves it, as close to its original form as possible, into an analytics destination where other tools can do that work.
Data integration patterns
Data integration typically happens in one of two orders, depending on when transformation occurs relative to loading.
- Extract transform load (ETL): Data is extracted from a source system, transformed within the data integration technology to support analytical workloads, and then loaded into the target warehouse.
- Extract load transform (ELT - what Celigo Sync uses): Data is extracted from a source system and loaded in its native, or close to native, form. Once loaded, the data is transformed into the structure needed for analytical workloads.
Celigo sync follows the ELT pattern - rapidly extracting and loading data so your warehouse can handle the transformations.
How Celigo sync differs from flows
Flows designed in Flow builder, and syncs built with Celigo sync, share a common runtime, connector set, metadata repository, and user registry. Both can extract data from a source application, transform it, and load it to a data warehouse. Celigo sync automates work that Flow builder requires you to configure manually: it discovers source metadata, creates and maintains destination tables, and applies schema drift handling automatically, without requiring you to build an individual flow per object.
Celigo sync features
Celigo sync combines flexible source extraction, automated warehouse loading, schema drift management, lightweight transformations, full operational visibility, and enterprise governance into a single data ingestion pipeline.
Source applications
- Built-in support - Offers metadata-aware support for over 1000+ applications.
- Flexible data export - Pulls data from HTTP-based apps, databases, and universal connections (such as REST, FTP, and GraphQL). See Exports (source) overview.
- Extract options - Ingests full historical data during your initial setup (backfill) and runs scheduled incremental updates going forward (polling).
Destination applications
- Supported warehouses - Currently loads data into Snowflake, Google BigQuery, and NetSuite Analytics Warehouse (NSAW).
- Automated setup - Automatically creates and maintains target tables based on the source objects you select, mapping data types to match the target.
- Flexible loading - Lets you update (merge deltas), append, or replace target data depending on your use case (see Create a data ingestion sync).
Metadata-driven automation
- Smart schema management - Uses source metadata when available or relies on built-in schema inference.
- Drift detection - Automatically flags schema changes—like added or removed objects and columns, or shifted data types.
- Policy-driven drift management- Applies specific automated actions whenever schema drift occurs, based on rules you set at the sync or object level.
Transformations and modeling
- Lightweight prep - Performs lightweight "little-t" changes right inside the platform to ensure source-to-target compatibility, leaving heavy transformation work for downstream tools.
- Built-in options - Easily mask sensitive data, manage primary keys, and choose between normalizing or flattening hierarchical data.
Operations and visibility
- Clear dashboards - Displays loaded row counts per object alongside overall sync status.
- Instant alerts - Logs events (like schema drift), highlights errors in-product, and sends email notifications.
- Full backfill - Allows you to trigger a complete resync whenever needed.
Security and governance
- Enterprise security - Built directly on Celigo’s platform security, inheriting role-based access control (RBAC) and enterprise governance standards.
Next up: Create a data ingestion sync.