New flow FTP to API runs way to slowly
Answered
I'm trying to speed up a new FTP to API transfer.
For my test which I ran several time I made 459 files available and by looking at the Run History I see that the FTP pickup took (34 - 37) minutes and the API export took (2 -3) minutes. This fileset had four files that were significant in size 3.3, 3.0, 1.1 and 0.5 M all of the remaining were 17Kb or smaller.
The FTPS server is running on our LAN, we have gigabit plus internet speed, we have checked our firewall settings, disabled deep packet inspection etc and we have run out of options. This FTPS server is dedicated to working with Celigo.
We ran an independent test to connect to this server from another SAS service and via the same protocol FTPS we were able to pick up 1700 files in 3 minutes so the bottleneck does not seem to be our server, network or internal security.
I have concurrency turned up to 4 but this made no noticeable improvement, setting a batch size triggers an error message about Blob transfers and wont let me save the change.
I was expecting speeds about 8 to 10 times faster than I'm seeing. What happens if I have this flow scheduled and the previous occurrence is still running, will it not start until the previous finishes or does it know which files are already being worked and will skip them or will it try to retransfer the same files?
-
Hi, what error message do you get when you set the batch size?
Thanks
0 -
Could you confirm if this is the error that you are noticing while trying to save the FTP export after setting the batch size.
If yes, let us know what file type you are trying to export from FTP, it will help us to guide you further. Because, while I tried to replicate the error I noticed this error occurs only when the option "Parse files being transferred" is set to "No" (This must be set to "No" if the customer want to send the raw files as-is). Currently we are not supporting batching in Blob export.
Regards,
Pridhvi
0 -
Yes this Blob export message is the one preventing us from testing batching.
0 -
I missed part of your question. In the integration flow we are struggling with we are transferring a number of files with .csv and .txt file extensions from an inhouse integration gateway to our ecommerce site where we are dropping the files in an Azure Blob. We are not parsing or transforming these files just delivering them.
0 -
Batch exporting raw files should be supported as of this morning's product release. Please use the below field on your source FTP export bubble. I would try 100 first to see the performance increase, and then if that is not fast enough you can go all the way to 1000. If the field is left blank, then we are only processing 1 file at a time by default, which is going to be super slow for thousands of smaller files.
0
Please sign in to leave a comment.
Comments
5 comments