In addition to securely managing credentials to external applications, connections are an important resource for optimizing the performance of your flows and, if necessary, for preventing flows from exceeding data governance limits set by the external applications being integrated. This article explains how connections work behind the scenes, along with how connections can be configured to optimize performance and adhere to API governance limits.
Contents
- Every connection backed by a FIFO queue
- Setting concurrency level on a connection
- Borrowing concurrency between connections
- Examples visualized
Every connection backed by a FIFO queue
A fundamental principle of integrator.io is that every connection resource is backed by its own dedicated FIFO queue, through which all records and tasks are first persisted and then processed by first-in, first-out (FIFO) order. For example, if you are importing orders into an ERP application, the orders are first added to the ERP connection’s queue, and then the orders are processed on a FIFO basis.
The number of messages in the FIFO queue that can be processed in parallel is controlled by the Concurrency level setting of each connection. The maximum concurrency level is based on your subscription plan. Wherever you see a list of connections – such as in the Connections tab of Flow Builder or in the Resources menu > Connections page – the total number of messages in each connection’s queue is displayed in the Queue size column.
Connections as queues allow you to process data at a massive scale, in lots of creative ways, without violating governance limits set by external applications. For example, during peak holiday season, you might have thousands of orders coming in every hour from your web store, but your ERP application might allow only 25 API requests to be submitted in parallel at once. You can precisely match the governance dictated by the ERP, and import orders at the maximum possible rate, by simply setting the concurrency level on the connection to 25.
It is also common to establish multiple independent connections to the same application for the purpose of partitioning data flows based on business priorities. The following articles explain in detail how/when to do this activity for your Integration Apps, and then of course the same activity can be done for any flow in your account.
- Leverage multiple connections in your Integration Apps
- Optimize multiple connections in your Integration App flows to improve throughput
Continuing the example above, if your ERP application only allows 25 API requests at a time, you could create two separate connections for the application to keep business-critical information flowing quickly without having to compete for throughput with less time-sensitive flows:
- A high-volume connection dedicated to your order and customer-facing flows.
- A low-volume connection dedicated to less important back office-type flows that sync transactions or item updates.
During peak holiday season, you might set the concurrency level to 20 on the connection supporting your order and customer flows, and to 5 on the connection supporting your back office flows. Then, once peak season is over, the numbers could be changed.
This example uses a very simplistic API governance model where the external ERP application limits the number of parallel API requests. You will likely run into lots of different API governance models depending on the external applications you are working with. For example, some APIs limit the total number API requests per day, per hour, or per minute. Some APIs have different governance models for different endpoints within the same API. Regardless of what you encounter, integrator.io gives you the fundamental building blocks to support any API governance model – the ability to create multiple connections, allocate those connections to flows steps in different ways, and control the concurrency level at which messages are consumed from each connection queue.
“Waiting in line at a store” analogy
If you are not familiar with FIFO queues, it might help to read this analogy. Hint: FIFO queues are used everywhere in our daily lives!
Imagine you are at a store packed with shoppers who want to pay for the items in their carts, but there is only a single cashier, and there is no official queue/line for shoppers to wait in. Total chaos, right?! How can the store fix this? The most basic thing the store can do is to establish a single queue for all shoppers to wait in. The same is true for integration flows between different applications. The number one most important thing you can do to prevent API governance errors, data getting processed out of order, and race conditions is to route all data through a FIFO queue.
Now imagine the store is really, really busy, and the single queue for the single cashier is enormous. The customers will still be very frustrated. How can the store fix this? The next best solution is to hire multiple cashiers, so that multiple customers can be processed in parallel. This is similar to increasing the concurrency level on a connection to allow more than one message to be processed in parallel.
Now imagine that most shoppers in the store are buying a single item, but in the same queue are shoppers with hundreds of items. The cashiers will quickly be occupied with the large orders, and the shoppers with only a single item will remain frustrated waiting in line. To fix this, the store could establish different queues for different numbers of items being purchased. This way, shoppers with only one item can be processed separately from the shoppers with lots of items. This workflow is similar to creating multiple connections and then allocating them to different flow steps based on business priorities. Each connection can have its own concurrency level to match the expected load.
All of these examples are related to increasing throughput and allowing more shoppers to check out their items, but what if checking out should instead slow down due to something the store cannot control? For example, maybe the only way to leave the store is via two elevators, one of which is broken, and the waiting area for the elevators is small. It doesn’t make sense to process customers faster and faster, because then customers will bottleneck at the elevator where there is no space to wait. To fix this, the store could shrink the number of cashiers. This scenario is similar to an external application imposing API governance that forces you to lower the throughput of data, which you can accomplish by lowering the concurrency level per connection or having multiple connections borrow concurrency from each other.
Setting concurrency level on a connection
To set or modify Auto-recover rate limit errors/Target concurrency level or Concurrency level connection settings, you must have administer or manage permissions. If you have monitor permissions, you can only view the connection settings. Consider these scenarios for concurrency:
- When the Auto recover rate limit errors setting is enabled
- When the Auto recover rate limit errors setting is not enabled
When the Auto-recover rate limit errors setting is enabled
In the Advanced section on any new connection resource, Auto-recover rate limit errors is enabled with a default Target concurrency level based on your endpoint for efficient processing. This setting applies to all flows that will use the connection.
Target concurrency level defines the maximum number of messages in the connection’s FIFO queue that can be processed in parallel. (Concurrency level is not editable and indicates what the Celigo platform is currently using at any time.)
For example, if the Target concurrency level is set to 6 and a long-running export is running in one of your flows, then you will still have 5 messages that can be processed in parallel. Those additional 5 messages could also be long-running exports, or they could be very quick record imports. It does not matter.
For details on how concurrency is managed to automatically recover from rate limit errors, see Connection setting – Auto-recover rate limit errors.
When the Auto-recover rate limit errors setting is not enabled
When Auto-recover rate limit errors is not enabled, the Target concurrency level is not shown.
In this case, the Concurrency level will define the maximum number of messages in the connection’s FIFO queue and the processing will be exactly the same as explained in the example above. This setting applies to all flows using the connection.
Tip: The maximum concurrency level allowed per connection is 25, but you can always create multiple connections for the same application to achieve higher concurrency levels when needed.
Note: If you do not select any Concurrency level value for an HTTP-based connection (that is, the setting is left blank), then the concurrency defaults to “burst mode” behind the scenes. In burst mode, the connection will call the endpoint API as fast as possible. For example, if a single page of records being processed by your flow contains 100 records, then a connection running in burst mode could send 100 individual HTTP requests in parallel to your endpoint. If you have Auto-recover rate limit errors enabled and select burst mode, the target concurrency value will be automatically set to 25.
Tip: It is always advisable to check the external application’s documentation to understand how it enforces API governance, and be sure to check the limit for the specific account level tier that you purchased. If you are allowed only 1 API request per minute and your Concurrency level is set to 3, then you will quickly see governance errors in your flows when they run at scale.
Managing concurrency and how it relates to flow performance is also covered in Fine-tune integrator.io for optimal data throughput.
Borrowing concurrency between connections
As mentioned at the very beginning of this article, connections are responsible for securely managing credentials to the external applications being integrated. Sometimes you will want to create multiple independent connections solely for the purpose of giving different teams different permissions to those applications.
For example, your core IT team might create a connection to your ERP application using administrator credentials, but the ecommerce team might not be allowed to use the same administrator credentials due to data security rules at the company. You could set up a separate connection with a different set of credentials limiting the ecommerce team to a different API role.
Using separate connections for different teams works great until you run into API governance limits due to the multiple connections each using their own independent concurrency level settings. The solution for this is to have connections “borrow” concurrency from others, and then all of the messages across borrowed connections will be stored in the same FIFO queue, and the concurrency level can be set in just one place to comply with the external API’s governance policy. See Borrow concurrency from in the connection settings, shown above.
In the simplest terms, Borrow concurrency from allows multiple independent connections to share the same concurrency level, and then all data flowing through all the related connections gets submitted to the endpoint application via the same shared concurrency model.
Examples visualized
Restricting multiple connections to the same shared concurrency level
Picture an integration with three separate connections to the same endpoint: Connection A, Connection B, and Connection C.
In this instance, it is important to stay within a maximum of 2 parallel requests to the endpoint from all 3 connections. That is, they must use a shared concurrency level of 2 for all requests, which can be accomplished with the following steps:
- Modify Connection A to set its Concurrency level to 2.
- In Connection B still, select Borrow concurrency from > Connection A.
- Save Connection B.
- Modify Connection C, by selecting Borrow concurrency from > Connection A.
- Save Connection C.
When the flows run, all three connections will share the same concurrency level of 2.
Using multiple connections to partition flow processing
Another scenario in which you might want to create two or more connections and set separate concurrency levels is as follows:
- One connection serves flows that generate high-volume, high-priority, time-sensitive data, such as Order and Fulfilment.
- Another connection is intended for flows that generate high-volume, low-priority, time-insensitive data, such as syncing product information
Such a configuration enables you to have greater levels of concurrency available for your high-priority data, which should get synced faster.
Comments
6 comments
In this example, could those additional 5 messages be from the same long-running export? Or does it need to be from a different export?
If we are running only one export with a concurrency level of 6, does that mean up to 6 records from that same export can be processed concurrently? If so, what happens when another export using the same connection suddenly starts? How do those two exports split the concurrency levels?
A long running export (if on the source side of the flow) will typically only use 1 concurrency because it's going out and making 1 api request for a page of data. The concurrency level is for how many api requests we are making to the endpoint, which can be different than the number of actual records when getting a list of records back from an api or making a batch import request to an api.
In the case of a long export, we would make one request to get the first page of results, then make subsequent requests for more pages of data. This inherently uses 1 concurrency because I have to get a response back before making a request for a new page of data.
Where an export could take up more concurrency is if the export is a lookup step within your flow. For example, if you need to make an api request per record in order to get additional information about the original record. In this case, the lookup step would use additional concurrency because it can ping the api in parallel for each specific record.
Hopefully that helps!
Thanks Tyler. Do you have any insight on how we can improve processing time for single long running imports? We are using the Amazon <> Netsuite IA and we occasionally have long processing times for the Netsuite import step. It sounds like adding concurrency does not improve the processing time for a single flow but it would allow more than one flow to process at a time. Is there a way to reduce the processing time for a single long running import?
Hi Anna - what NetSuite imports are you seeing long processing times for?
Also, maybe I need to reword my response here. Increasing your concurrency definitely could have benefits for your import steps (an import step is where we are importing data into the final destination; export step is where we are fetching data from a source and bringing it into Celigo). When the import step is creating or updating on a per record basis, increasing the concurrency allows multiple records to be created or updated in parallel. For example, if you are pulling orders from Amazon and importing them to NetSuite, if your NetSuite concurrency is set to 1, then we will create/update 1 order in NetSuite at a time. So we will make 1 request to NetSuite, get a response, then make an additional request for the next order. If you were to set your NetSuite concurrency to 2 on the NetSuite connector, we would send 2 requests to NetSuite at the same time in order to create/update 2 orders at one time in parallel.
In your initial question, you asked about exports and how concurrency was used there. For example, if I have an export to get Amazon orders, I make a single api request to get 50 orders at a time. Subsequently, once I get the response for the first 50 orders, I then make another request for the next 50 orders and so on and so on until I've gotten all my orders. For exports, we have to get one page of 50 records at a time before we make another call for the next page of 50. Therefore we typically only use a concurrency of 1 since we can only make 1 request at a time.
To bring it all together, your concurrency level on the connection level is what is used across all your flows that use that connection. For example, if I have flow 1 that imports Amazon orders to NetSuite, flow 2 that is pulling inventory from NetSuite to send to Amazon, and both of those flows use the same NetSuite connection, each flow shares the concurrency of that connection.
Does that help clarify it a bit further?
Hi Tyler,
That helps to clarify things a lot, thank you so much.
In this example, say that this Netsuite connection has a concurrency of 2. Flow #1 is importing Amazon orders to Netsuite and it is the only flow running on that Netsuite connection, so it is importing 2 records in parallel by using up both concurrencies. Then say that Flow #2 starts running, does it queue up after Flow #1 or does Flow #1 lower the concurrency usage so that Flow #2 can run in parallel?
In your example, flow 2 will queue up after the messages that flow 1 has already put in the FIFO queue to process. If flow 1 is still exporting new source records after flow 2 starts queuing messages at the same time, then you will have both flow 1 and flow 2 messages being processed at the same time in FIFO order.
Please sign in to leave a comment.