FAQ: How can I process large sets of data?

I’ve observed records being imported immediately after the data is paged. A limit of 20 records per page is too small when I need to aggregate (group, sum, or RBAR) data in larger sets. How can we queue and process larger sets of data? 

The default page size contains 20 exported records. You can modify the Page size advanced setting of an export, up to 5 MB.

Tip: In some cases, the application that you are importing data into will practically restrict the page size value. For example, NetSuite and Salesforce both specify a maximum number of records that can be submitted in any single request, according to their API guides.

Aside from exports, Celigo integrator.io does support the ability to aggregate arbitrarily large data for files retrieved from FTP or Amazon S3.