Celigo Topics lets one flow publish a message and any number of other flows react to it. A topic is a named channel that stores the messages published to it for a period you set, so every subscribing flow reads them independently — and a flow that was disabled or failing picks up where it left off instead of missing what it didn't see. Use Topics when you need integration processes to react to each other without wiring them together directly, and when losing a message would matter.
Topics is available when the Topics license is enabled for your account.
Why publish to a topic
Without Topics, connecting one flow to another means the producing flow has to know its consumers: you build the consuming flow first, create a listener on it, then point the producing flow at that listener's URL. Every new consumer means changing the producer, and nothing that arrives while a consumer is off is recoverable.
A topic inverts that. The producing flow publishes to a named channel and knows nothing about who reads it. You add, change, or remove consuming flows without touching — or interrupting — the producing side.
Key concepts
Topic
A topic is a named, durable channel that holds messages. It has a name, an optional description, and a retention period.
Topics are account-level resources, scoped to a single environment. Each environment — production and every non-production environment — has its own separate topics. Topics deliberately don't live inside an integration, because decoupling integrations from each other is the point of publishing to a topic.
When you clone an integration, export it as a template, promote it between environments, or pull changes with integration lifecycle management, the topics' configuration carries into the target environment. Retained messages, counts, and cursor positions don't transfer.
A topic name is a label, not an identity. Names don't need to be unique, and you can rename a topic at any time: every publish step, listener, and API call references a stable topic ID that is assigned when the topic is created, never the display name.
Message
A message is a timestamp and a payload. That is all.
Payloads are freeform JSON up to 1 MB. A topic doesn't define or validate a schema, so any JSON within the size limit is accepted. There is no envelope of IDs, types, or attributes for you to populate or manage.
Retention
Retention is how long a topic keeps each message — configurable from 1 hour to 90 days, defaulting to 7 days. When a message reaches the end of the retention window it is deleted, whether or not anything read it.
Retention is what makes a subscribing flow recoverable. A flow that is disabled, broken, or added later can still read anything the topic is holding, and you can replay retained messages to a flow after you fix a problem.
Note: Messages a flow hadn't read before they aged out can't be recovered or replayed, so set retention longer than your worst-case flow downtime. See Choose a retention window.
Subscription
A subscription is a flow listening to a topic. You create one by adding a Listen for messages from a topic step to a flow and choosing a topic. The listener is the subscription — there is no separate subscription object to create or manage.
Every subscription on a topic receives every message. This is fan-out: 10 flows subscribed to one topic each get their own copy of each message, and reading a message never removes it for anyone else.
Cursor
Each subscription tracks its own position in the topic — its cursor — which advances as the flow processes messages. A cursor is runtime state that you don't set or edit; for where a subscription resumes after downtime or a problem, see How a subscription decides where to resume.
Replay
A replay redelivers messages a flow has already processed, starting from the earliest retained message or a time you choose, with an optional end time. Start a replay from the flow builder after you fix an outage, a bug, or a mapping mistake. A replay runs as its own job and delivers only to the flow that requested it. Live delivery continues in parallel, and the subscription's cursor doesn't move, so other flows that share the listener aren't affected. You can only replay messages the topic still retains. For steps, see Subscribe a flow to a topic: Replay messages.
How messages move
- A producer publishes a message to a topic — from a flow step, or from an external system through the Topics API.
- The topic stores the message for its retention period.
- Every active subscription on that topic receives the message.
- Each subscribing flow processes it as a record. Failures land in that flow's error management, where you can retry them.
Messages are delivered to subscribing flows in pages rather than one at a time. A page is delivered when it fills (Page size) or when the wait interval elapses with a partial page (Max wait time), whichever comes first.
Delivery guarantees
Celigo Topics delivers messages at least once. A message is delivered to every subscription, and under some conditions — a retry, a recovery, a replay, or a subscription resuming after an interruption or from a stored position that is no longer valid (see How a subscription decides where to resume) — a message can be delivered more than once.
Note: Build subscribing flows to tolerate duplicates. Design downstream steps so that processing the same message twice produces the same result as processing it once — for example, by using an upsert instead of an insert. For more approaches, see Duplicate records. Exactly-once delivery isn't offered, and no message broker offers it across arbitrary endpoints.
Messages published to a topic are stored in the order they arrive. Ordering across different topics isn't coordinated, and ordering within a topic is best-effort when several producers publish at the same time.
Keep messages in order
To enforce strict ordering, use two settings:
- Publish to the topic from a single producer — one flow or one external system, not several.
- On the imports in your subscribing flow, set a Concurrency ID lock template so that records sharing a key are processed one at a time.
Limits
| Limit | Value |
|---|---|
| Maximum message size | 1 MB |
| Retention range | 1 hour to 90 days |
| Default retention | 7 days |
| Topics per account | 20 by default |
| Publish batch, flow step | Sent at 5 MB, at the configured batch size (default 100 messages), or when the maximum wait time elapses, whichever comes first |
The topic limit is set per account and can be raised — contact your Celigo account team.
Celigo topics versus broker topics
Celigo also offers connectors for external message brokers such as Kafka and JMS. Those brokers have topics too, and the word means something similar — but they aren't the same resource:
- A Celigo topic is created and managed in integrator.io under Resources > Topics. Celigo stores the messages. You don't run any infrastructure.
- A topic on your own broker lives on a Kafka, JMS, or RabbitMQ cluster that you operate. Celigo reaches it through a connector.
Use Celigo Topics for integration-layer messaging between flows. Use the broker connectors when you already run a broker and need to integrate with it.
When to use topics
Use a topic when:
- Several flows need to react to the same data, and you don't want the producing flow to know about any of them.
- An external system needs to send data into Celigo durably, without you managing a webhook URL for every consuming flow.
- Losing data during a flow outage would matter, and you need a consumer to catch up after it recovers.
- You want to add a consumer to something that's already happening, without changing the producer.
Use something else when:
- One flow needs to trigger exactly one other flow, with no durability requirement. A webhook listener is simpler.
- You need a request and a response. Publishing to a topic is one-way — the publisher gets an acknowledgment that the message was stored, never a result from a subscriber. Use an API for request-response work.
- You need to filter which messages reach a flow before they enter it. Every subscription receives every message, and filtering happens inside the flow — see Filter which messages your flow acts on.
- You already run a broker and want to integrate with it. Use the Kafka, JMS, or RabbitMQ connectors.
What you can do with a topic
| Task | Article |
|---|---|
| Create a topic and set its retention | Create and manage topics |
| See what has been published | View and purge a topic's message history |
| Control who can use a topic | Grant access to topics |
| Publish from a flow | Publish messages from a flow |
| Publish from an external system | Topics API reference |
| Make a flow react to messages | Subscribe a flow to a topic |
| Replay messages after an outage or a fix | Subscribe a flow to a topic: Replay messages |
| Trace what a flow published or received | View flow run logs |