In addition to securely managing credentials to external applications, connections are an important resource for optimizing the performance of your flows and, if necessary, for preventing flows from exceeding data governance limits set by the external applications being integrated. This article explains how connections work behind the scenes, along with how connections can be configured to optimize performance and adhere to API governance limits.
- Every connection backed by a FIFO queue
- Setting concurrency level on a connection
- Borrowing concurrency between connections
- Examples visualized
A fundamental principle of integrator.io is that every connection resource is backed by its own dedicated FIFO queue, through which all records and tasks are first persisted and then processed by first-in, first-out (FIFO) order. For example, if you are importing orders into an ERP application, the orders are first added to the ERP connection’s queue, and then the orders are processed on a FIFO basis.
The number of messages in the FIFO queue that can be processed in parallel is controlled by the Concurrency level setting of each connection. Wherever you see a list of connections – such as in the Connections tab of Flow Builder or in the Resources menu > Connections page – the total number of messages in each connection’s queue is displayed in the Queue size column.
Connections as queues allow you to process data at a massive scale, in lots of creative ways, without violating governance limits set by external applications. For example, during peak holiday season, you might have thousands of orders coming in every hour from your web store, but your ERP application might allow only 25 API requests to be submitted in parallel at once. You can precisely match the governance dictated by the ERP, and import orders at the maximum possible rate, by simply setting the concurrency level on the connection to 25.
It is also common to establish multiple independent connections to the same application for the purpose of partitioning data flows based on business priorities. The following articles explain in detail how/when to do this activity for your Integration Apps, and then of course the same activity can be done for any flow in your account.
- Leverage multiple connections in your Integration Apps
- Optimize multiple connections in your Integration App flows to improve throughput
Continuing the example above, if your ERP application only allows 25 API requests at a time, you could create two separate connections for the application to keep business-critical information flowing quickly without having to compete for throughput with less time-sensitive flows:
- A high-volume connection dedicated to your order and customer-facing flows.
- A low-volume connection dedicated to less important back office-type flows that sync transactions or item updates.
During peak holiday season, you might set the concurrency level to 20 on the connection supporting your order and customer flows, and to 5 on the connection supporting your back office flows. Then, once peak season is over, the numbers could be changed.
This example uses a very simplistic API governance model where the external ERP application limits the number of parallel API requests. You will likely run into lots of different API governance models depending on the external applications you are working with. For example, some APIs limit the total number API requests per day, per hour, or per minute. Some APIs have different governance models for different endpoints within the same API. Regardless of what you encounter, integrator.io gives you the fundamental building blocks to support any API governance model – the ability to create multiple connections, allocate those connections to flows steps in different ways, and control the concurrency level at which messages are consumed from each connection queue.
“Waiting in line at a store” analogy
If you are not familiar with FIFO queues, it might help to read this analogy. Hint: FIFO queues are used everywhere in our daily lives!
Imagine you are at a store packed with shoppers who want to pay for the items in their carts, but there is only a single cashier, and there is no official queue/line for shoppers to wait in. Total chaos, right?! How can the store fix this? The most basic thing the store can do is to establish a single queue for all shoppers to wait in. The same is true for integration flows between different applications. The number one most important thing you can do to prevent API governance errors, data getting processed out of order, and race conditions is to route all data through a FIFO queue.
Now imagine the store is really, really busy, and the single queue for the single cashier is enormous. The customers will still be very frustrated. How can the store fix this? The next best solution is to hire multiple cashiers, so that multiple customers can be processed in parallel. This is similar to increasing the concurrency level on a connection to allow more than one message to be processed in parallel.
Now imagine that most shoppers in the store are buying a single item, but in the same queue are shoppers with hundreds of items. The cashiers will quickly be occupied with the large orders, and the shoppers with only a single item will remain frustrated waiting in line. To fix this, the store could establish different queues for different numbers of items being purchased. This way, shoppers with only one item can be processed separately from the shoppers with lots of items. This workflow is similar to creating multiple connections and then allocating them to different flow steps based on business priorities. Each connection can have its own concurrency level to match the expected load.
All of these examples are related to increasing throughput and allowing more shoppers to check out their items, but what if checking out should instead slow down due to something the store cannot control? For example, maybe the only way to leave the store is via two elevators, one of which is broken, and the waiting area for the elevators is small. It doesn’t make sense to process customers faster and faster, because then customers will bottleneck at the elevator where there is no space to wait. To fix this, the store could shrink the number of cashiers. This scenario is similar to an external application imposing API governance that forces you to lower the throughput of data, which you can accomplish by lowering the concurrency level per connection or having multiple connections borrow concurrency from each other.
To set or modify Auto-recover rate limit errors/Target concurrency level or Concurrency level connection settings, you must have administer or manage permissions. If you have monitor permissions, you can only view the connection settings. Consider these scenarios for concurrency:
- When the Auto recover rate limit errors setting is enabled
- When the Auto recover rate limit errors setting is not enabled
When the Auto-recover rate limit errors setting is enabled
In the Advanced section on any new connection resource, Auto-recover rate limit errors is enabled with a default Target concurrency level based on your endpoint for efficient processing. This setting applies to all flows that will use the connection.
Target concurrency level defines the maximum number of messages in the connection’s FIFO queue that can be processed in parallel. (Concurrency level is not editable and indicates what the Celigo platform is currently using at any time.)
For example, if the Target concurrency level is set to 6 and a long-running export is running in one of your flows, then you will still have 5 messages that can be processed in parallel. Those additional 5 messages could also be long-running exports, or they could be very quick record imports. It does not matter.
For details on how concurrency is managed to automatically recover from rate limit errors, see Connection setting – Auto-recover rate limit errors.
When the Auto-recover rate limit errors setting is not enabled
When Auto-recover rate limit errors is not enabled, the Target concurrency level is not shown.
In this case, the Concurrency level will define the maximum number of messages in the connection’s FIFO queue and the processing will be exactly the same as explained in the example above. This setting applies to all flows using the connection.
Tip: The maximum concurrency level allowed per connection is 25, but you can always create multiple connections for the same application to achieve higher concurrency levels when needed.
Note: If you do not select any Concurrency level value for an HTTP-based connection (that is, the setting is left blank), then the concurrency defaults to “burst mode” behind the scenes. In burst mode, the connection will call the endpoint API as fast as possible. For example, if a single page of records being processed by your flow contains 100 records, then a connection running in burst mode could send 100 individual HTTP requests in parallel to your endpoint. If you have Auto-recover rate limit errors enabled and select burst mode, the target concurrency value will be automatically set to 25.
Tip: It is always advisable to check the external application’s documentation to understand how it enforces API governance, and be sure to check the limit for the specific account level tier that you purchased. If you are allowed only 1 API request per minute and your Concurrency level is set to 3, then you will quickly see governance errors in your flows when they run at scale.
Managing concurrency and how it relates to flow performance is also covered in Fine-tune integrator.io for optimal data throughput.
As mentioned at the very beginning of this article, connections are responsible for securely managing credentials to the external applications being integrated. Sometimes you will want to create multiple independent connections solely for the purpose of giving different teams different permissions to those applications.
For example, your core IT team might create a connection to your ERP application using administrator credentials, but the ecommerce team might not be allowed to use the same administrator credentials due to data security rules at the company. You could set up a separate connection with a different set of credentials limiting the ecommerce team to a different API role.
Using separate connections for different teams works great until you run into API governance limits due to the multiple connections each using their own independent concurrency level settings. The solution for this is to have connections “borrow” concurrency from others, and then all of the messages across borrowed connections will be stored in the same FIFO queue, and the concurrency level can be set in just one place to comply with the external API’s governance policy. See Borrow concurrency from in the connection settings, shown above.
In the simplest terms, Borrow concurrency from allows multiple independent connections to share the same concurrency level, and then all data flowing through all the related connections gets submitted to the endpoint application via the same shared concurrency model.
Restricting multiple connections to the same shared concurrency level
Picture an integration with three separate connections to the same endpoint: Connection A, Connection B, and Connection C.
In this instance, it is important to stay within a maximum of 2 parallel requests to the endpoint from all 3 connections. That is, they must use a shared concurrency level of 2 for all requests, which can be accomplished with the following steps:
- Modify Connection A to set its Concurrency level to 2.
- In Connection B still, select Borrow concurrency from > Connection A.
- Save Connection B.
- Modify Connection C, by selecting Borrow concurrency from > Connection A.
- Save Connection C.
When the flows run, all three connections will share the same concurrency level of 2.
Using multiple connections to partition flow processing
Another scenario in which you might want to create two or more connections and set separate concurrency levels is as follows:
- One connection serves flows that generate high-volume, high-priority, time-sensitive data, such as Order and Fulfilment.
- Another connection is intended for flows that generate high-volume, low-priority, time-insensitive data, such as syncing product information
Such a configuration enables you to have greater levels of concurrency available for your high-priority data, which should get synced faster.