When configuring exports with a universal connector (like the HTTP connector or the REST connector), you will need to configure the pagination strategy used by the service you are exporting data from. For dedicated connectors that Celigo has created for the service, these settings are configured automatically, but you may need to review the API documentation for services that don't use a connector that Celigo has prebuilt for the service. The terminology associated with these paging methods isn't universal, so it may sometimes be difficult to determine which paging method to choose even after consulting the API documentation provided by the service. This article describes each paging method supported by integrator.io and also provides examples of how third-party API documentation describes each strategy.
Does this API support paging?
The Does this API support paging? section of the edit export page allows you to configure the paging methods for any service.
Some APIs offer paging functionality in order to limit their HTTP response to a manageable size if the total set of resources is large. If the API supports pagination, then you can configure the Does this API support paging? section.
integrator.io supports the following paging methods:
- Custom relative URI
- Custom request body
- Link Header
- Next page token
- Next Page URL
- Page number parameter
- Skip number parameter
Custom relative URI
Use this paging method if the API provides an ID or token as part of a page response. This ID or token is used as part of the URL in the following request in order to retrieve the next page response.
Override relative URI for subsequent page requests: This field only needs to be set if subsequent page requests require a different relative URI than what is configured in the primary relative URI field.
Override HTTP status code for last page: This field only needs to be set if the HTTP status code for the last page is not 404. For example, an API could return a generic 400 status code instead, and then use a field in the HTTP response body to indicate no more pages are available.
Path to paging complete field in HTTP respose body: This field only needs to be set if the API returns a field in the HTTP response body to indicate paging is complete. For example, if an API returns the field 'errorMessage' with the value 'No more pages' to indicate paging is done, then you would set this field to 'errorMessage' to tell integrator.io to check this field on each page response.
Paging complete values: Use this field to limit the exact values in the HTTP response body field that should be used to determine if paging is complete. For example, if an API returns the field 'errorMessage' with the value 'No more pages' to indicate paging is done, then you would set this field to 'No more pages'. To provide multiple values, use a comma-separated list.
The following services use Custom relative URI pagination. The following links to their documentation illustrate how this pagination strategy is referenced in other documentation sets:
Custom request body
Choose this paging method if paging is tracked by a value included in the POST request body.
Override HTTP request body for subsequent page requests: This field only needs to be set if subsequent page requests require a different HTTP request body than what is configured in the primary HTTP request body field.
Override HTTP status code for last page: This field only needs to be set if the HTTP status code for the last page is not 404. For example, an API could return a generic 400 status code instead, and then use a field in the HTTP response body to indicate no more pages are available.
Path to paging complete field in HTTP respose body: This field only needs to be set if the API returns a field in the HTTP response body to indicate paging is complete. For example, if an API returns the field 'errorMessage' with the value 'No more pages' to indicate paging is done, then you would set this field to 'errorMessage' to tell integrator.io to check this field on each page response.
Paging complete values: Use this field to limit the exact values in the HTTP response body field that should be used to determine if paging is complete. For example, if an API returns the field 'errorMessage' with the value 'No more pages' to indicate paging is done, then you would set this field to 'No more pages'. To provide multiple values, use a comma-separated list.
Custom request body example
When using custom request body pagination, field values from the previous page of records inform integrator.io how to retrieve the next page of records. In such cases, the "last_record" object contains the identifier. However, if the “last_record” object doesn’t contain the required identifier, you can use the "full_response" object to retrieve the identifier to access the next page of records.
Sample Request
{
"requestDetails": "sampleRequestDetails",
"page": {{add previous_page.last_record.page 1}},
"pageSize": 2
}
Sample Input Data
{
"previous_page":
{
"last_record":
{
"data": [
{
"sampleField": "sampleRecord1"
},
{
"sampleField": "sampleRecord2"
}],
"page": 1,
"pageSize": 2,
"total": 100
},
"full_response":
{}
},
"connection":
{
"name": "test NCI",
"http":
{
"unencrypted":
{
"field": "value"
},
"encrypted": "********"
}
},
"export":
{
"name": "test NCI"
},
"settings":
{
"integration":
{},
"flow":
{},
"flowGrouping":
{},
"connection":
{},
"export":
{}
}
}
The following services use Custom request body pagination. The following links to their documentation illustrate how this pagination strategy is referenced in other documentation sets:
{
"DateField": "Modified",
"FromDate": "0001-01-01T00:00:00.0000000Z",
"ToDate": "0001-01-01T00:00:00.0000000Z",
"PageSize": 1000,
"PageNumber": 0,
"TenantToken": "String",
"UserToken": "String"
}
Link Header
Choose this method if the API points to subsequent pages in the link header of the API response. When you select this paging method, by default integrator.io uses HTTP conventions to look for the next page URL within a dedicated link header value. You can include multiple URLs for page navigation (forward, back, first, last, etc.).
Override link header relation name: When the paging method is set to "Link Header", by default IO uses HTTP conventions to look for the next page url within a dedicated "link" header value. It is possible, within this link header, to include multiple URLs facilitating page navigation forward, back, or even first or last. In cases where multiple values are found, integrator.io needs to know which to use. If the API uses multiple values, you can define those values here. The conventions for these relation values include prev, next, last, and first (integrator.io defaults to next). Since some APIs may not comply with the default "next" relation, you can override it by entering the value expected by your HTTP app. For more information on link headers, see the IETF standards for web linking.
Override HTTP status code for last page: This field only needs to be set if the HTTP status code for the last page is not 404. For example, an API could return a generic 400 status code instead, and then use a field in the HTTP response body to indicate no more pages are available.
Path to paging complete field in HTTP response body: This field only needs to be set if the API returns a field in the HTTP response body to indicate paging is complete. For example, if an API returns the field 'errorMessage' with the value 'No more pages' to indicate paging is done, then you would set this field to 'errorMessage' to tell integrator.io to check this field on each page response.
Paging complete values: Use this field to limit the exact values in the HTTP response body field that should be used to determine if paging is complete. For example, if an API returns the field 'errorMessage' with the value 'No more pages' to indicate paging is done, then you would set this field to 'No more pages'. To provide multiple values, use a comma-separated list.
The following services use Link Header pagination. The following links to their documentation illustrate how this pagination strategy is referenced in other documentation sets:
Next page token
Choose this paging method if the API returns a token to the next page within the response body.
Path to next page token field in HTTP response body: Use this field to tell integrator.io where in the HTTP response body to find the token to use when requesting the next page of data. If the paging method is set to Next page token, integrator.io needs to be able to find the token in the HTTP response of each page in order to construct the next page request. This field lets integrator.io know where to look for the next page token. If no value is found at this path, then paging is terminated (final page reached). The format of the path is dependent on the media type (content type) of the original (first) page response.
Example JSON response path
result.paging.nextPageUrl
Example XPATH for XML path
/result/paging/nextPageUrl
Paging terminates when no URL is found at the specified path.
Override initial token value: This field only needs to be set if the initial export request should contain a specific token value.
Override relative URI for subsequent page requests: This field only needs to be set if subsequent page requests require a different relative URI than what is configured in the primary relative URI field.
Override path to next page token field for subsequent page requests: This field only needs to be set if subsequent page requests return a different response structure, and the next page token field is located in a different place than the original request.
Override path to records for subsequent page requests: This field only needs to be set if subsequent page requests return a different response structure, and the records are located in a different place than the original request.
Override HTTP status code for last page: This field only needs to be set if the HTTP status code for the last page is not 404. For example, an API could return a generic 400 status code instead, and then use a field in the HTTP response body to indicate no more pages are available.
Path to paging complete field in HTTP response body: This field only needs to be set if the API returns a field in the HTTP response body to indicate paging is complete. For example, if an API returns the field 'errorMessage' with the value 'No more pages' to indicate paging is done, then you would set this field to 'errorMessage' to tell integrator.io to check this field on each page response.
Paging complete values: Use this field to limit the exact values in the HTTP response body field that should be used to determine if paging is complete. For example, if an API returns the field 'errorMessage' with the value 'No more pages' to indicate paging is done, then you would set this field to 'No more pages'. To provide multiple values, use a comma-separated list.
The following services use Next page token pagination. The following links to their documentation illustrate how this pagination strategy is referenced in other documentation sets:
Next Page URL
Use this paging method when the API provides the complete URL to which integrator.io must make the request for getting the next page.
Path to next page URL field in HTTP response body: Use this field to tell integrator.io where in the HTTP response body to find the URL to use when requesting the next page of data.
Override HTTP status code for last page: This field only needs to be set if the HTTP status code for the last page is not 404. For example, an API could return a generic 400 status code instead, and then use a field in the HTTP response body to indicate no more pages are available.
Path to paging complete field in HTTP response body: This field only needs to be set if the API returns a field in the HTTP response body to indicate paging is complete. For example, if an API returns the field 'errorMessage' with the value 'No more pages' to indicate paging is done, then you would set this field to 'errorMessage' to tell integrator.io to check this field on each page response.
Paging complete values: Use this field to limit the exact values in the HTTP response body field that should be used to determine if paging is complete. For example, if an API returns the field 'errorMessage' with the value 'No more pages' to indicate paging is done, then you would set this field to 'No more pages'. To provide multiple values, use a comma-separated list.
The following services use Next page URL pagination. The following links to their documentation illustrate how this pagination strategy is referenced in other documentation sets:
Page number parameter
Choose this paging method if the API uses a page number parameter to paginate the results.
Override page number start index: This field only needs to be set if the export should start at a specific page number other than 0.
Path to total number of pages field in HTTP response body: Some APIs return the total number of pages available in each of their page responses. This optional field can be used as a trigger to stop making page requests. This field tells integrator.io where to find the maximum page value within each page response, and when this page count is met, no more page requests will be made. If omitted, integrator.io will continue to make requests until no resources are returned, or a 404 (not found) response is encountered.
Path to total number of results field in HTTP response body: Some APIs return the total number of resources available in each of their page responses. This optional field can be used as a trigger to stop making page requests. This field tells integrator.io where to find the total resource count within each page response, and when this resource count is met, no more page requests will be made. If omitted, integrator.io will continue to make requests until no resources are returned, or a 404 (not found) response is encountered.
Override HTTP status code for last page: This field only needs to be set if the HTTP status code for the last page is not 404. For example, an API could return a generic 400 status code instead, and then use a field in the HTTP response body to indicate no more pages are available.
Path to paging complete field in HTTP response body: This field only needs to be set if the API returns a field in the HTTP response body to indicate paging is complete. For example, if an API returns the field 'errorMessage' with the value 'No more pages' to indicate paging is done, then you would set this field to 'errorMessage' to tell integrator.io to check this field on each page response.
Paging complete values: Use this field to limit the exact values in the HTTP response body field that should be used to determine if paging is complete. For example, if an API returns the field 'errorMessage' with the value 'No more pages' to indicate paging is done, then you would set this field to 'No more pages'. To provide multiple values, use a comma-separated list.
The following services use Page number parameter pagination. The following links to their documentation illustrate how this pagination strategy is referenced in other documentation sets:
Skip number parameter
Use this paging method if the API uses a skip-take paging mechanism (also known as offset or limit).
Override skip number start index: This field only needs to be set if the export should start at a specific skip number index other than 0.
Path to total number of pages field in HTTP response body: Some APIs return the total number of pages available in each of their page responses. This optional field can be used as a trigger to stop making page requests. This field tells integrator.io where to find the maximum page value within each page response, and when this page count is met, no more page requests will be made. If omitted, integrator.io will continue to make requests until no resources are returned, or a 404 (not found) response is encountered.
Path to total number of results field in HTTP response body: Some APIs return the total number of resources available in each of their page responses. This optional field can be used as a trigger to stop making page requests. This field tells integrator.io where to find the total resource count within each page response, and when this resource count is met, no more page requests will be made. If omitted, integrator.io will continue to make requests until no resources are returned, or a 404 (not found) response is encountered.
Override HTTP status code for last page: This field only needs to be set if the HTTP status code for the last page is not 404. For example, an API could return a generic 400 status code instead, and then use a field in the HTTP response body to indicate no more pages are available.
Path to paging complete field in HTTP response body: This field only needs to be set if the API returns a field in the HTTP response body to indicate paging is complete. For example, if an API returns the field 'errorMessage' with the value 'No more pages' to indicate paging is done, then you would set this field to 'errorMessage' to tell integrator.io to check this field on each page response.
Paging complete values: Use this field to limit the exact values in the HTTP response body field that should be used to determine if paging is complete. For example, if an API returns the field 'errorMessage' with the value 'No more pages' to indicate paging is done, then you would set this field to 'No more pages'. To provide multiple values, use a comma-separated list.
The following services use Skip number parameter pagination. The following links to their documentation illustrate how this pagination strategy is referenced in other documentation sets:
Comments
2 comments
There doesn't seem to be much difference in between custom relative uri and next page token pagination methods. Both list Stripe as an example?
Hi Joshua Jackson,
Thanks for your feedback. We're working on improving the API pagination examples to be more thorough in terms of their various use cases. If you need help setting up pagination for a specific platform, please submit a support ticket.
Tom
Please sign in to leave a comment.