FAQ: Can integrator.io handle a lookup response greater than 5MB?

As a standard, integrator.io does not support lookup responses greater than 5 MB. However, there is an alternative solution for large single-record or unchanged datasets returned by a JSON lookup. This workaround is valid for use cases like retrieving all Project Ids from a source and then retrieving all the project details for a specific Project Id. In this case, you won’t be able to perform a normal lookup to retrieve the project details because the response is over 5 MB.

Important: You will need access to your company’s FTP to send and get the datasets.

1. For the initial source step, export the data from your JSON source application as normal. Remember, in this use case you’re exporting every Project Id from a source.
2. Using either your source application connector or an HTTP connector, create a JSON Destination & Lookup step and use Lookup additional files (per record) to get your specific project file.

   

3. After you create an export to retrieve your file, create an FTP Destination & Lookup step and use Transfer files into destination application to import the raw data into your FTP.

   

   Warning: Do not let this transfer generate files from the records.

At this point, you have successfully uploaded your project file to your FTP. You can create a second flow to retrieve the file from the FTP. integrator.io can parse this file into JSON and you can use a transformation to pare down the data to 5 MB. This will work for any JSON lookup less than 25 MB.