Skip to main content

Delta Export laging

Comments

3 comments

  • Scott Henderson CTO
    Answer Pro
    Top Contributor

    The time on the log is not meant to match the {{lastExportDateTime}} value that you see in the query parameter.  The time on the log represents when the HTTP request was made, but the {{lastExportDateTime}} value sources from the flow job record that you see in the run history/console tab in flow builder.  To state the obvious, it is very important for delta flows to never miss or export data twice, and we have robust logic to ensure our {{lastExportDateTime}} values are  exact and transactional across flow runs.

    Regarding scheduling every 5 vs 6 minutes, our scheduler is robust in that it will never miss a run, but if there are other flows trying to run in the same connection queues in your account, or if you are running the flow at a very popular time, then it is possible to see slight delays; but ultimately it will run as close as possible to your desired frequency and scheduled times.

    1
  • Bas van Ditzhuijzen Strategic Partner NetSuite Integration Whiz Salesforce Integration Whiz
    Answer Pro
    Engaged
    Celigo University Level 4: Legendary

    A delta flow using only {{lastExportDateTime}} will always run into race/overlap conditions because of the lag between the time the export is queued and the time it actually runs.

    To fix this for flows that cant handle overlap there is an undocumented variable {{currentExportDateTime}} that you can add to your query if the target API supports a date range, that way it will be like this:

    url?from={{lastExportDateTime}}&to={{currentExportDateTime}}

    url?from=10:00&to=10:02

    url?from=10:02&to=10:04

    url?from=10:04&to=10:06

     

    (for flows that need 100% one-to-one reliability I would recommend not using delta at all but use Once paradigm)

     

    2
  • Scott Henderson CTO
    Answer Pro
    Top Contributor

    Yes, nice addition Bas!

    Both {{currentExportDateTime}} and {{lastExportDateTime}} are needed to truly guarantee records are not exported twice across different flow runs, and the target API needs to support receiving a date/time range vs just a single date/time. 

    All the delta flows that I use in my own integrator.io account are idempotent, and it is harmless to occasionally export records twice, so I often forget about {{currentExportDateTime}}.

    It is unfortunate that {{currentExportDateTime}} is not documented though.  I will pass a note to our docs team.

    Thanks!

    1

Please sign in to leave a comment.