Safeguard data pipeline integrity by proactively monitoring data ingestion syncs and diagnosing failed or partially failed runs. By learning to quickly locate sync errors, pinpoint their root causes across the extraction, transformation, and loading (ETL) stages, and apply targeted resolutions, you can minimize pipeline downtime and ensure your downstream systems always run on reliable, up-to-date data.
Unlike error management for application integrations, data ingestion errors are tied to the sync run, not to individual records. You don't hold, edit, or retry records one at a time. Instead, you fix the underlying cause and let the next run clear the errors.
Go to the Dashboard to review sync run errors for the latest sync, along with the job run history. Error details are consolidated at the run level, and you can click through to view the errors that apply to individual objects, such as a specific destination table.
If the most recent run of a sync contains errors, the count appears next to the sync in the homepage sync list and on the integration tile.
In the dashboard, errors appear in the graph and in the verbose sync run table:
-
To see error details, select the error number link.
-
To see the errors thrown when ingesting a specific table, find the sync run row, select Actions (...), and then select View tables.
Errors are categorized by the specific sync stage in which they occur: overall (initialization), extract, transform, or load. Identifying the active stage is key, as it tells you whether the issue can be resolved on your end or if it requires internal platform support. Additionally, sync errors are temporary; because error counts do not persist across lifecycles, completing a successful run or manually re-running the sync will automatically clear them.
Table 1.
|
Stage |
Typical causes |
Resolution |
|---|---|---|
|
Overall (initialization) |
Initialization flow failure, metadata extraction failure. |
Re-run the sync, or wait for the next scheduled sync. |
|
Extract |
Connection down, insufficient permissions, API rate limits on the source. |
Fix the issue on the source system, and then re-run or wait for the next scheduled sync. |
|
Transform |
Mapping failures, data type conversion failures. |
Internal Celigo issue. Contact Celigo support. Re-running is likely to keep failing until Celigo resolves the issue. |
|
Load |
Bulk load failure, file generation failure, storage (S3) failure, or destination data warehouse issues (for example, data quality violations, warehouse running out of credits). |
Re-run the sync. The re-run includes a fresh extract. |
Extract errors mean the sync couldn't pull data from the source. Common causes are a connection that's down, insufficient permissions on the source account, or API rate limits on the source system. Restore connectivity, adjust the source permissions, or wait for the rate limit to reset, and then re-run the sync or wait for the next scheduled sync.
Transform errors, such as mapping failures or data type conversion failures, are internal Celigo issues that you can't resolve from integrator.io. Contact Celigo support. Re-running the sync before the issue is fixed is likely to produce the same error.
After you make a change, confirm the fix by starting a manual run or waiting for the next scheduled sync. A run that completes without errors confirms the issue is resolved.