File Storage Connections
Hello,
Recently, we have been running into an issue of transferring files to services that host files on a separate server such as Google Cloud Storage or AWS. Frequently, these APIs require users to connect directly with the could storage instance. For example, to upload a file to Shopify, you must first make a request to the Shopify API's upload endpoint, which will then return a URL and API key for its Google Cloud Instance to do the actual file upload. Freshsales does something very similar with its Amazon S3 instance.
In and of itself, this pattern is not an issue, as it simply requires making a second request to the next endpoint. However, given that Integrator.io measures subscription volume by endpoints, these endpoints can quickly consume a substantial portion of a subscription by effectively doubling the number of required connections.
Is there any workaround to this issue? Perhaps a way to indicate to Integrator.io that this file connection is really just a part of the initial connection?
Thank you for any insights!
-
Ezriah there isn't currently a workaround or way to handle this other than what you mentioned. Thanks for the feedback, though. We're currently gathering file upload requirements from various sources to identify any gaps and I'll include this in that.
1 -
Tyler Lamparter Thank you for your response.
I do understand how this would be tricky to implement given how the platform is set up, as you would want to avoid provisioning an additional connection without limiting the functionality of the existing connection.
A couple of thoughts on how this could be done:
1. Named connectors should be relatively straightforward; these connections could simply include the file upload endpoints along with the other endpoints, which could then be linked in Celigo's backend to the storage service. For example, in the Shopify connection, you could include an endpoint for "Upload File" that actually points towards the appropriate Google Cloud instance. This would actually have three benefits: a) reduce the duplicate subscription cost; b) make it much easier for novice users to complete these flows as they wouldn't have to worry about managing an additional service (indeed, they wouldn't even need to know it existed); and c) would make it very easy from your end to prevent abuse of this additional connection as you could direct exactly where this additional endpoint is directed.
2. Universal connectors would be admittedly harder because you would have no idea where the upload connection should point. Thus, you would have a difficult time controlling any additional endpoints given for file upload and ensuring they are used solely within the scope of this connection. Perhaps there could be a way to designate a file upload endpoint in the HTTP connection screen. Then, there might be an option on the import block (alongside the usual hook, filter, mapping, etc. options) for a file upload, which would go to the previously designated endpoint. This way, the scope of the additional endpoint would be limited to that particular HTTP connection.
0 -
Ezriah thanks for the ideas. If I was a betting man, I would assume we would build this in a universal way within the HTTP framework so that it could live under the same connector you need it for. We built this for the opposite pattern of needing to download public files that are returned from a particular endpoint: https://docs.celigo.com/hc/en-us/articles/15447841934619-Celigo-platform-2023-5-1-release-notes.
0
Please sign in to leave a comment.
Comments
3 comments