Delta Export laging
Hello,
I'm using the Delta Export Type option to export orders by {{lastExportDateTime}}. In the logs I've noticed that there is a lag of 10 or more seconds when exporting the data. Here are the logs:
As you can see the flow was run at 15:05:27 and then in the next run the time set under updated_at_min parameter is 15:05:12 this is 15 seconds back from the previous run. Also as you can see in the other logs the lag is different for each run. I need the time for the updated_at_min parameter to match the time when the flow was run last so I can avoid pulling orders that were already pulled or skipping orders. Can this be done? Is there a way to avoid the lag between the runs?
Also another thing that I have noticed is that the flow is setup to run every 5 minutes but it runs every 6.
Comments
The time on the log is not meant to match the {{lastExportDateTime}} value that you see in the query parameter. The time on the log represents when the HTTP request was made, but the {{lastExportDateTime}} value sources from the flow job record that you see in the run history/console tab in flow builder. To state the obvious, it is very important for delta flows to never miss or export data twice, and we have robust logic to ensure our {{lastExportDateTime}} values are exact and transactional across flow runs.
Regarding scheduling every 5 vs 6 minutes, our scheduler is robust in that it will never miss a run, but if there are other flows trying to run in the same connection queues in your account, or if you are running the flow at a very popular time, then it is possible to see slight delays; but ultimately it will run as close as possible to your desired frequency and scheduled times.
A delta flow using only {{lastExportDateTime}} will always run into race/overlap conditions because of the lag between the time the export is queued and the time it actually runs.
To fix this for flows that cant handle overlap there is an undocumented variable {{currentExportDateTime}} that you can add to your query if the target API supports a date range, that way it will be like this:
url?from={{lastExportDateTime}}&to={{currentExportDateTime}}
url?from=10:00&to=10:02
url?from=10:02&to=10:04
url?from=10:04&to=10:06
(for flows that need 100% one-to-one reliability I would recommend not using delta at all but use Once paradigm)
Yes, nice addition Bas!
Both {{currentExportDateTime}} and {{lastExportDateTime}} are needed to truly guarantee records are not exported twice across different flow runs, and the target API needs to support receiving a date/time range vs just a single date/time.
All the delta flows that I use in my own integrator.io account are idempotent, and it is harmless to occasionally export records twice, so I often forget about {{currentExportDateTime}}.
It is unfortunate that {{currentExportDateTime}} is not documented though. I will pass a note to our docs team.
Thanks!
Please sign in to leave a comment.