Listen for real-time changes from MongoDB using CDC

In Application, search and select the database connector MongoDB from the list.

In Step type, select Listen for real-time data from source application.

In Name your step, enter a unique name. You can reuse this listener across flows, so a specific name makes it easier to identify in a list.

Optional: In Describe your step, enter a clear and concise description of the flow step to help anyone using this listener in their flows understand its purpose. Keep this description updated as you make changes.

In Connection, select a MongoDB connection from the list. If you don't have a connection, select Create connection at the end of the list and set up a connection to MongoDB.

In Collections, enter a comma-separated list of values to specify which MongoDB collections to listen to.

For example, enter users, database2.products where users represents a collection on the connection database and database2.products represents the products collection on a different database.

Optional: In Aggregation pipeline, enter a specific flow of operations on databases, collections, or fields based on how you would like to process, transform, and return results. Each successive operation is based on the previous result. You must always specify the full path for collections and fields here.

In Additional properties, enter key-value pairs for databases, collections, or fields that you would like to listen to or not listen to. You must always specify the full path for collections and fields here. Based on your requirements, you can set optional properties, such as:

For example, if you want to listen to databases other than the connection database, you can set database.include.list.

Default properties: capture.mode defaults to change_streams_update_full and snapshot.mode defaults to no_data. You can change these default values if required.

In Fields to include, select which parts of the change event to include in the listener payload. By default, only the after field is selected, but you can reduce payload size and simplify downstream processing by choosing only the fields you need (for example, just after and op to capture the updated record and operation type). Nested fields can also be included or excluded for more control.

In Max wait time, set the maximum time (in seconds) for the listener to wait before sending a batch of events. Typically, events are sent when the page size limit is reached. If the events are fewer, they are sent after the wait time. The default is 300 seconds. The minimum wait time is 30 seconds.

In Page size, specify how many records you want in each page of data. The Celigo platform splits exported data into pages of records. The default is 250 records per page; pages are capped automatically at 5 MB. The application you are importing data into will most often be the bottleneck on page size.

In Data URI template, when your flow runs but has data errors, use this field to ensure that all the errors in your job dashboard have a link to the original data from the export application. This field uses a handlebars template to generate the dynamic links based on the data being exported.

Turn on Do not store retry data if you don't want the Celigo platform to store retry data for records that fail in your flow. Storing retry data can slow down your flow's overall performance if you are processing very large numbers of failing records.

In Override trace key template, define a trace key that the Celigo platform will use to identify a unique record. Any value you provide overrides the default trace key for your app. You can specify a single field, such as {{record.field1}}, or use a handlebars expression. For example, {{join "_" record.field1 record.field2}} generates a trace key such as 123_456. If you have applied a transformation to exported data, reference its fields in the trace key template without the path record. — for example, {{field1}}.