Skip to main content

Fine-tune integrator.io for optimal data throughput

Comments

4 comments

  • David Gollom Referral Partner
    Great Answer
    Celigo University Level 2: Seasoned
    Engaged

    I appreciate this information - thank you. I have a follow up on the FTP batch size parameter.  If we're processing small files, then we should set the batch number higher?  We have a use case where the might be 100 small files of about 2KB in size with a few large ones around 200KB mixed in.  We're parsing these into JSON data.  I set the batch size to be 20.  Do you think there will be any issues?? I'm not sure where these timeouts mentioned about would be happening.  Can I set it to 100??  The page size is set to 1.  Does the batch size impact the page size?  

    0
  • Courtney Jordan Experience Strategy & Design Director Community moderator
    Celigo University Level 4: Legendary
    Awesome Follow-up
    Top Contributor

    Hi David Gollom!
    We're checking into your question and will get back to you asap.

    0
  • viliandy leonardo Product Management Director

    Hi David Gollom,

    • The FTP export Page size defines the number of records that you want IO to group for processing and sending downstream to be imported into their destination.
    • The  FTP export Batch size specifies the number of files you want IO to export from the FTP site in the same session without having to spend time logging out and logging back in to the server for each file.
    • FTP batch size does not impact page size but they both work together to control the integration overall performance.

    If you expect many large-sized files to export, then setting a really high batch size might experience network timeout errors. However, if you expect many small-sized files, then it is advisable to set a higher batch size value so that IO does not process one small file at a time, leading to a more performant integration. You can start with batch size 20. The new error management (to be released) has a chart that plots average processing time/success event over time, and you can use this chart to analyze your flow performance and tweak the batch size as you see fit.  

    The Concurrency in the FTP connection provides another control to tune your FTP export performance. Setting this field allows you to open multiple connections to the same FTP site and export multiple files from the FTP site. To be sure, please note that that the FTP connection concurrency setting only works for an export step and it is not applicable for import step.

    Thanks 

    0
  • David Gollom Referral Partner
    Great Answer
    Celigo University Level 2: Seasoned
    Engaged

    Thank you so much for clarifying that.  It makes sense now.  I made the change to 20 earlier for the batch size and the performance improved greatly. 

    0

Please sign in to leave a comment.