As a standard, integrator.io does not support lookup responses greater than 5 MB. However, there is an alternative solution for large single-record or unchanged datasets returned by a JSON lookup. This workaround is valid for use cases like retrieving all Project Ids from a source and then retrieving all the project details for a specific Project Id. In this case, you won’t be able to perform a normal lookup to retrieve the project details because the response is over 5 MB.
Important: You will need access to your company’s FTP to send and get the datasets.
- For the initial source step, export the data from your JSON source application as normal. Remember, in this use case you’re exporting every Project Id from a source.
-
Using either your source application connector or an HTTP connector, create a JSON Destination & Lookup step and use Lookup additional files (per record) to get your specific project file.
-
After you create an export to retrieve your file, create an FTP Destination & Lookup step and use Transfer files into destination application to import the raw data into your FTP.
Warning: Do not let this transfer generate files from the records.
At this point, you have successfully uploaded your project file to your FTP. You can create a second flow to retrieve the file from the FTP. integrator.io can parse this file into JSON and you can use a transformation to pare down the data to 5 MB. This will work for any JSON lookup less than 25 MB.
Why a lookup exceeds 5 MB
A lookup step returns its entire response for each incoming record in one stream, and that stream is capped at 5 MB whatever page size you set. A lookup that asks the source for everything that changed since the last run against a large list crosses the limit quickly. For example, a Mailchimp lookup filtering ~96,000 members by “changed since” fails with Response stream exceeded limit of 5242880 bytes.
Resolution
Return less data per record. Use one or more of the following:
- Use the source API’s narrowest filter. For example, in Mailchimp, filter on “unsubscribed since” with
status=unsubscribedinstead of “changed since”. - Request only the fields you need.
- Run the flow more often so each run picks up fewer changes.
- Move the query to a standalone export step, which pages its results and isn’t subject to the 5 MB lookup limit.
Note: Retrying the failed record doesn’t help. The retry repeats the same request and fails with the same error.
Known issue: retrying a lookup-step error
If a lookup step shows the error !flowUtil.findPageGeneratorConfigByExportId(flow, _pageGeneratorId) as an internal error, it is the failed retry of an earlier lookup-step error, and the error that was retried is marked resolved. This isn’t an outage. Don’t retry lookup-step errors. Instead, open the step’s resolved errors, find the one that was retried, fix the underlying cause, and run the flow again.