API pagination – Celigo Help Center

When configuring exports with a universal connector (like the HTTP connector or the REST connector), you will need to configure the pagination strategy used by the service you are exporting data from. For dedicated connectors that Celigo has created for the service, these settings are configured automatically, but you may need to review the API documentation for services that don't use a connector that Celigo has prebuilt for the service. The terminology associated with these paging methods isn't universal, so it may sometimes be difficult to determine which paging method to choose even after consulting the API documentation provided by the service. This article describes each paging method supported by integrator.io and also provides examples of how third-party API documentation describes each strategy.

Note: For all path values related to pagination, verify that you are using a path that matches the record structure displayed in the HTTP Response preview section. For example, in this case the path would be data.customers.pageInfo.endCursor.

Does this API support paging?

The Does this API support paging? section of the edit export page allows you to configure the paging methods for any service.

Some APIs offer paging functionality in order to limit their HTTP response to a manageable size if the total set of resources is large. If the API supports pagination, then you can configure the Does this API support paging? section.

integrator.io supports the following paging methods:

Custom relative URI
Custom request body
Link Header
Next page token
Next Page URL
Page number parameter
Skip number parameter

Link Header

Choose this method if the API points to subsequent pages in the link header of the API response. When you select this paging method, by default integrator.io uses HTTP conventions to look for the next page URL within a dedicated link header value. You can include multiple URLs for page navigation (forward, back, first, last, etc.).

Link header relation: If the API uses multiple values, you can define those values here. The conventions for these relation values include prev, next, last, and first (integrator.io defaults to next). Since some APIs may not comply with the default "next" relation, you can override it by entering the value expected by your HTTP app. For more information on link headers, see the IETF standards for web linking.

Link Header pagination documentation examples

The following services use Link Header pagination. The following links to their documentation illustrate how this pagination strategy is referenced in other documentation sets:

Next Page URL

Use this paging method when the API provides the complete URL to which integrator.io must make the request for getting the next page.

Path: Enter the path used to retrieve the response.

Next Page URL pagination documentation examples

The following services use Next Page URL pagination. The following links to their documentation illustrate how this pagination strategy is referenced in other documentation sets:

Page

Choose this paging method if the API uses a page number parameter to paginate the results.

Page: Set an initial Page value if the export should start at a custom page offset. This value automatically increments with each subsequent page request. The value can be referenced as a placeholder in the body, headers, or relative URI of the export request. If left blank, the page value is set to 0.

Max page path: Some APIs return the total number of pages available in each of their page responses. Use this optional field as a trigger to stop making page requests. This field tells integrator.io where to find the maximum page value within each page response. If omitted, integrator.io continues to make requests until no resources return or a 404 (not found) response occurs.

Max count path: Some APIs return the total number of resources available in each of their page responses. Use this optional field as a trigger to stop making page requests. This field tells integrator.io where to find the total resource count within each page response. If omitted, integrator.io continues to make requests until no resources return or a 404 (not found) response occurs.

Page pagination documentation examples

The following services use Page pagination. The following links to their documentation illustrate how this pagination strategy is referenced in other documentation sets:

Post Body

Choose this paging method if paging is tracked by a value included in the POST request body.

Build paging post body: Click Launch to open the Build paging post body editor. Here you can enter handlebars expressions to reference a value in the body of the POST request.

Post Body pagination documentation examples

The following services use Post Body pagination. The following links to their documentation illustrate how this pagination strategy is referenced in other documentation sets:

{  
  "DateField": "Modified",
  "FromDate": "0001-01-01T00:00:00.0000000Z",
  "ToDate": "0001-01-01T00:00:00.0000000Z",
  "PageSize": 1000,
  "PageNumber": 0,
  "TenantToken": "String",
  "UserToken": "String"
 }

Relative URI

Use this paging method if the API provides an ID or token as part of a page response. This ID or token is used as part of the URL in the following request in order to retrieve the next page response.

Relative URI: Enter the relative URI that retrieves the next page response.

Relative URI pagination documentation examples

The following services use Relative URI pagination. The following links to their documentation illustrate how this pagination strategy is referenced in other documentation sets:

Skip

Use this paging method if the API uses a skip-take paging mechanism (also known as offset or limit).

Skip: Use this field if the export should start at an offset from the first record. This value is automatically set on each subsequent page request. The value can be referenced as a placeholder in the body, headers, or relativeURI of the export request. If left blank, the skip value is set to 0.

Note: If the records returned from an API are nested within an array inside the main record, you must define the location of the records in the Path to records in HTTP response body field in the Nonstandard API response patterns section of the export.

See Export data from an HTTP source app for more information.

Max page path: Some APIs return the total number of pages available in each of their page responses. This field tells integrator.io where to find the maximum page value within each page response, and when this page count is met, no more page requests will be made. If left blank, integrator.io will continue to make requests until no resources are returned, or a 404 (not found) response returns.

Max count path: Some APIs return the total number of resources available in each of their page responses. This field tells integrator.io where to find the total resource count within each page response, and when this resource count is met, no more page requests will be made. If left blank, integrator.io continues to make requests until no resources are returned, or a 404 (not found) response returns.

Skip pagination documentation examples

The following services use Skip pagination. The following links to their documentation illustrate how this pagination strategy is referenced in other documentation sets:

Openair.com

Acumatica.com

Adp.com

Token

Choose this paging method if the API returns a token to the next page within the response body.

Token: If the initial export should contain a token, this value is automatically set on each subsequent page request. You can reference the value as a placeholder in the body, headers, or relative URI of the export request.

For example:

/products?token={{{export.http.paging.token}}}

Path: If the paging method is set to token, integrator.io needs to be able to find the token in the HTTP response of each page in order to construct the next page request. This field lets integrator.io know where to look for the next page token. If no value is found at this path, then paging is terminated (final page reached). When the paging method is set to Next page URL, integrator.io uses this Path field to find where in the HTTP response to look to find a URL to use when requesting the next page of data. The format of the path is dependent on the media type (content type) of the original (first) page response.

Example JSON response path

result.paging.nextPageUrl

Example XPATH for XML path

/result/paging/nextPageUrl

Paging terminates when no URL is found at the specified path.

Relative URI: Use this optional field in combination with any paging method.

Note: In previous versions of integrator.io, when subsequent page requests needed to use a different relative URI from the first page request, you had to use block handlebars helpers to configure your relative URI to handle subsequent requests. For example:

"/users/{{settings.integration.seller_id}}/items/search?search_type=scan{{#if export.http.paging.token}}&scroll_id={{export.http.paging.token}}{{/if}}&limit=50&status=active"

This practice is not longer necessary because integrator.io now automatically performs that logic. The same result that previously required the #if expression would result with the following example:

"/users/{{settings.integration.seller_id}}/items/search?search_type=scan&scroll_id={{export.http.paging.token}}&limit=50"

Token path after first request: Use this optional field in combination with any paging method. In some cases, subsequent page requests need to use a different relative URI than the one used for the first page request.

Resource path: This is an optional override to the resource path if subsequent page responses have a different response structure. Note that the format of this path is specific to the media type.

Example JSON resource path

results.customers

Example XPATH for XML resource path

/SearchResults/Customers/Customer

Last page status code: Some APIs return an error status code (less than 200 or greater than 300) as the last page response. Enter the response code integrator.io should expect in these situations so the system doesn't interpret the response as an error. You can use this field in conjunction with the Last page path and Last page values fields. integrator.io assumes error 404 is the last page response, so if the API you are connecting to also uses that convention, you can leave this field blank.

Last page path: If you have entered a Last page status code value, you use Last page path to search for a specific field in the response object to determine if paging is complete. For example, the specified field can be "error.message". If Last page path is empty, then paging will be considered complete; otherwise, the response will be considered an error.

Last page values: If you have entered values for the Last page status code and Last page path fields, then use this field to check a specific field's value in the response object to determine if paging is complete. If the value in the response object matches the value in the Last page values field, then paging will be considered complete; otherwise, this response will be considered an error.

Token pagination documentation examples

The following services use Token pagination. The following links to their documentation illustrate how this pagination strategy is referenced in other documentation sets: