As a standard, integrator.io does not support lookup responses greater than 5 MB. However, there is an alternative solution for large single-record or unchanged datasets returned by a JSON lookup. This workaround is valid for use cases like retrieving all Project Ids from a source and then retrieving all the project details for a specific Project Id. In this case, you won’t be able to perform a normal lookup to retrieve the project details because the response is over 5 MB.
Important: You will need access to your company’s FTP to send and get the datasets.
- For the initial source step, export the data from your JSON source application as normal. Remember, in this use case you’re exporting every Project Id from a source.
- Using either your source application connector or an HTTP connector, create a JSON Destination & Lookup step and use Lookup additional files (per record) to get your specific project file.
- After you create an export to retrieve your file, create an FTP Destination & Lookup step and use Transfer files into destination application to import the raw data into your FTP.
Warning: Do not let this transfer generate files from the records.
At this point, you have successfully uploaded your project file to your FTP. You can create a second flow to retrieve the file from the FTP. integrator.io can parse this file into JSON and you can use a transformation to pare down the data to 5 MB. This will work for any JSON lookup less than 25 MB.