Celigo Topics lets one flow publish a message and any number of other flows react to it. A topic is a named channel that stores messages published to it for a period you set, so every subscribing flow reads them independently — and a flow that was disabled or failing picks up where it left off instead of missing what it didn't see. Use Topics when you need integration processes to react to each other without wiring them together directly, and when losing a message would matter.
Topics is available when the Topics license is enabled for your account.
Why publish to a topic
Without Topics, connecting one flow to another means the producing flow has to know its consumers: you build the consuming flow first, create a listener on it, then point the producing flow at that listener's URL. Every new consumer means changing the producer, and nothing that arrives while a consumer is off is recoverable.
A topic inverts that. The producing flow publishes to a named channel and knows nothing about who reads it. You add, change, or remove consuming flows without touching — or interrupting — the producing side.
Key concepts
Topic
A topic is a named, durable channel that holds messages. It has a name, an optional description, and a retention period.
Topics are account-level resources, scoped to a single environment. Each environment — production and every non-production environment — has its own separate topics. Topics deliberately don't live inside an integration, because decoupling integrations is the point of publishing to a topic.
A topic name is a label, not an identity. Names don't need to be unique, and you can rename a topic at any time: every publish step, listener, and API call references a stable topic ID that is assigned when the topic is created, never the display name.
Message
A message is a timestamp and a payload. Payloads are freeform JSON up to 1 MB. A topic doesn't define or validate a schema, so it accepts any JSON within the size limit. You don't need to populate or manage an envelope of IDs, types, or attributes.
Retention
Retention is how long a topic keeps each message — configurable from 1 hour to 90 days, defaulting to 7 days. When a message reaches the end of the retention window, it is deleted, whether or not anything read it.
Retention is what makes a subscribing flow recoverable. A flow that is disabled, broken, or added later can still read anything the topic is holding.
Warning: Choose a retention window longer than your worst-case flow downtime. If a subscribing flow is off or failing longer than the retention period, it deletes the messages it hadn't read and can't recover them. The short end of the menu — 1 hour and 6 hours — is for genuinely transient, fast-moving data only.
Subscription
A subscription is a flow listening to a topic. You create one by adding a Listen for messages from a topic step to a flow and choosing a topic. The listener is the subscription — there is no separate subscription object to create or manage.
Every subscription on a topic receives every message. This is fan-out: 10 flows subscribed to one topic each get their own copy of each message, and reading a message never removes it for anyone else.
Cursor
Each subscription tracks its own position in the topic — its cursor. As a flow processes messages, its cursor advances. If the flow is disabled and re-enabled, it resumes from its cursor rather than starting over or skipping ahead.
Cursors are runtime state, not configuration. You don't set or edit a cursor. The one choice you make is where a brand-new subscription starts, using the listener's Start from setting. See Subscribe a flow to a topic.
How messages move
- A producer publishes a message to a topic — from a flow step, or from an external system through the Topics API.
- The topic stores the message for its retention period.
- Every active subscription on that topic receives the message.
- Each subscribing flow processes it as a record. Failures land in that flow's error management, where you can retry them.
Messages are delivered to subscribing flows in pages rather than one at a time. A page is delivered when it fills (Page size) or when the wait interval elapses with a partial page (Max wait time), whichever comes first.
Delivery guarantees
Celigo Topics delivers messages at least once. A message is delivered to every subscription, and under some conditions — a retry, a recovery, or a subscription resuming after an interruption — a message can be delivered more than once.
Warning: Build subscribing flows to tolerate duplicates. Design downstream steps so that processing the same message twice produces the same result as processing it once. Exactly-once delivery isn't offered, and no message broker offers it across arbitrary endpoints.
Messages published to a topic are stored in the order they arrive. Ordering across different topics isn't coordinated, and ordering within a topic is best-effort when several producers publish at the same time.
Keep messages in order
Until ordering keys are available, you can enforce strict ordering with two settings:
- Publish to the topic from a single producer — one flow or one external system, not several.
- On the imports in your subscribing flow, set a Concurrency ID lock template so that records sharing a key are processed one at a time.
Limits
| Limit | Value |
|---|---|
| Maximum message size | 1 MB |
| Retention range | 1 hour to 90 days |
| Default retention | 7 days |
| Topics per account | 20 by default |
| Publish batch, flow step | Flushes at 5 MB or the configured batch size (default 100 messages), whichever comes first |
The topic limit is set per account and can be raised — contact your Celigo account team.
Celigo topics versus broker topics
Celigo also offers connectors for external message brokers such as Kafka and JMS. Those brokers have topics too, and the word means something similar — but they aren't the same resource:
- A Celigo topic is created and managed in integrator.io under Resources > Topics. Celigo stores the messages. You don't run any infrastructure.
- A topic on your own broker lives on a Kafka, JMS, or RabbitMQ cluster that you operate. Celigo reaches it through a connector.
Use Celigo Topics for integration-layer messaging between flows. Use the broker connectors when you already run a broker and need to integrate with it.
When to use topics
Use a topic when:
- Several flows need to react to the same data, and you don't want the producing flow to know about any of them.
- An external system needs to send data into Celigo durably, without you managing a webhook URL for every consuming flow.
- Losing data during a flow outage would matter, and you need a consumer to catch up after it recovers.
- You want to add a consumer to something that's already happening, without changing the producer.
Use something else when:
- One flow needs to trigger exactly one other flow, with no durability requirement. A webhook listener is simpler.
- You need a request and a response. Publishing to a topic is one-way — the publisher gets an acknowledgment that the message was stored, never a result from a subscriber. Use an API for request-response work.
- You need to filter which messages reach a flow before they enter it. Every subscription receives every message on a topic. You can filter inside the subscribing flow, but the messages still enter it.
- You already run a broker and want to integrate with it. Use the Kafka, JMS, or RabbitMQ connectors.
What you can do with a topic
| Task | Article |
|---|---|
| Create a topic and set its retention | Create and manage topics |
| See what has been published | View and purge a topic's message history |
| Publish from a flow | Publish messages from a flow |
| Make a flow react to messages | Subscribe a flow to a topic |