Data flows/integrations

Anitha Abraham
  • Updated

Contents

Data exports

When querying an endpoint, in many cases you can further refine the data you’re retrieving:

  • All – always export all data
  • Delta – export only data created/updated since the last time the flow ran, based on a given date field
  • Once – export records only once by setting a flag in the source system as each record is processed
  • Limit – export between 1 and 100 records

You can also retrieve additional source data in a “lookup” flow step, following an export or import.

Tip: Each “page” of exported data cannot exceed 5 MB. (When integrator.io gets data from a source, it splits the data into pages to make the records more manageable for queueing and further processing. You can modify the default page size of 20 records in the Page size advanced setting of an export.) To reduce the size of the records within a page, you can modify the data in a preSavePage hook. That 5 MB also applies as the page moves “downstream” in a flow; be aware of how you might be increasing the size, for example, by adding fields in a lookup.

Known limitation: If an API does not offer functionality to support a delta export, you can achieve similar results by leveraging other integrator.io features, such as sending a value in the query string, creating and updating a custom field, or filtering records based on a known export date.

Data preview

You can configure an export or lookup and run a preview to see sample JSON structure before running it in a flow. You can choose to preview up to 100 records.

HTTP connectors also display full response and request details, including any error messages.

For more information:

Data imports

The most common and fundamental way to sync data is to retrieve a record from one system and insert or update it in another. The following field “mapping” methods are readily available for most apps:

  • Simple mapping
  • Hard-coded mapping
  • Multifield/handlebars expressions
  • Lookups – Dynamic and static queries of the destination app
  • Transformation mapping
  • Data type – Including overrides
  • Conditional mapping – Set a criteria, such as only when “updating a record”
  • Request body mapping
  • Advanced mapping assistants – Integrated for NetSuite and Salesforce destinations
  • Complex JSON mapping – Switch to Mapper 2.0 for complete visual control over data types and record structure in HTTP-based, EDI, and file transfer imports

For commonly used applications, the Celigo platform can automatically suggest destination field mappings that you have not yet added. Auto-map will add field mappings based on common configurations between two applications and how frequently these field mappings are successfully configured by integrator.io customers.

Tip: Each page of data processed in an import cannot exceed 5 MB. See Maximum record size for further implications of increasing data within a flow.

For more information:

Flow branching

Within a flow’s destinations, you can split the exported data into different branches for conditional processing. You can then import records of each branch into different destination applications or perform lookups. You can also add, modify, delete, and merge branches as required – filtered either with logical rules or JavaScript.

For more information:

Test mode

You can debug a flow in test mode to ensure the flow processes your records as expected before enabling the flow. Enter mock data manually or make calls to your source and destination apps to retrieve live data to use for testing purposes.

When you run a flow in test mode, you can review the processing behavior of each flow step option (transformation, mapping, filter, or hook) and fine-tune your flow configuration to handle all variations of results or responses from each application endpoint.

For more information:

Flow scheduling

You can run any flow on demand or set it to initiate with a call from a webhook listener (as a “real-time” flow). You can also schedule flows initiated from an export or file download with the following options:

  • Preset times
  • cron expressions
  • Daisy-chained – Initiated after a selected flow
  • Export delta override – For NetSuite, Salesforce, HTTP, and REST exports

For more information:

Known limitation: Export delta overrides are not available for file providers such as Amazon S3 and Google Drive.

Flow packaging

The Celigo platform allows you to clone a flow, distribute integrations as templates, package flows, and download to another account.

  • Make your integrations more portable by referencing resources with aliases
  • Test in a sandbox environment before deploying into production

For more information: 

Integration Lifecycle Management

Integration Lifecycle Management simplifies and accelerates the process of implementing, modifying, testing and deploying integrations. It supports revisions and enables you to:

  • Create snapshots as backups
  • Compare diffs between two integrations when merging changes
  • Pull changes between a remote and current integration
  • Revert an integration to a selected revision
  • Review changes in a revision

For more information:

integrator.io API

Access integration resources via the RESTful API:

  • Get, update, test, delete, and debug connections
  • Get, create, update, delete, and clone flows, exports, and imports
  • Run an export, import, or flow on demand from your external application
  • Get, create, update, delete, and clone integrations
  • Get, resolve, and retry all errors for an import, as well as get all errors for an export

Permissions

When invoking the integrator.io API, you must supply a secure, valid Bearer token retrieved from your account. Your account’s access allows for the creation and revocation of fine-grained API tokens, where a specific set of exports, imports, My APIs, or connections can be included in its scope. An audit log monitors the use and modification of tokens for recordkeeping.

For more information:

Related articles

Comments

0 comments

Please sign in to leave a comment.