Skip to main content

Fine-tune integrator.io for optimal data throughput

Follow

Comments

4 comments

  • David Gollom

    I appreciate this information - thank you. I have a follow up on the FTP batch size parameter.  If we're processing small files, then we should set the batch number higher?  We have a use case where the might be 100 small files of about 2KB in size with a few large ones around 200KB mixed in.  We're parsing these into JSON data.  I set the batch size to be 20.  Do you think there will be any issues?? I'm not sure where these timeouts mentioned about would be happening.  Can I set it to 100??  The page size is set to 1.  Does the batch size impact the page size?  

    Comment actions Permalink
  • viliandy leonardo

    Hi David Gollom,

    • The FTP export Page size defines the number of records that you want IO to group for processing and sending downstream to be imported into their destination.
    • The  FTP export Batch size specifies the number of files you want IO to export from the FTP site in the same session without having to spend time logging out and logging back in to the server for each file.
    • FTP batch size does not impact page size but they both work together to control the integration overall performance.

    If you expect many large-sized files to export, then setting a really high batch size might experience network timeout errors. However, if you expect many small-sized files, then it is advisable to set a higher batch size value so that IO does not process one small file at a time, leading to a more performant integration. You can start with batch size 20. The new error management (to be released) has a chart that plots average processing time/success event over time, and you can use this chart to analyze your flow performance and tweak the batch size as you see fit.  

    The Concurrency in the FTP connection provides another control to tune your FTP export performance. Setting this field allows you to open multiple connections to the same FTP site and export multiple files from the FTP site. To be sure, please note that that the FTP connection concurrency setting only works for an export step and it is not applicable for import step.

    Thanks 

    Comment actions Permalink
  • Courtney Jordan

    Hi David Gollom!
    We're checking into your question and will get back to you asap.

    Comment actions Permalink
  • David Gollom

    Thank you so much for clarifying that.  It makes sense now.  I made the change to 20 earlier for the batch size and the performance improved greatly. 

    Comment actions Permalink

Please sign in to leave a comment.