Paginated data not being passed through remainder of flow
Really hoping someone here can help me out with this, as I can't figure out why this is happening, and the Celigo support represent I've been working with via a support ticket can't figure it out either.
Background:
I have a flow step that performs a Shopify GraphQL query to get a variety of product data. The amount of data returned requires me to perform pagination due to rate limits (I query 100 products per call as you'll see below). The queries work great, and in the preview pane, I see all records, which make me think my pagination is working. Below are some screenshots on how I have this flow step configured with the pagination:
And these are the queries I have set in the above screenshots:
{
"query": "query ($numProducts: Int!, $sku_query: String, $cursor: String) { productVariants (first: $numProducts, query: $sku_query, after: $cursor) { pageInfo { hasNextPage endCursor } edges { node { id sku legacyResourceId inventoryItem { id legacyResourceId inventoryLevels(first: 1) { edges { node { id available location { id } } } } } } } } }",
"variables" : {
"numProducts": 100,
"cursor": null,
"sku_query": "{{#each record.inventory_data as |inv_record|}}{{#compare @index ">" 0}} OR {{/compare}}sku:{{inv_record.sku}}{{/each}}"
}
}
{{#if previous_page.full_response.data.productVariants.pageInfo.hasNextPage}}
{
"query": "query ($numProducts: Int!, $sku_query: String, $cursor: String) { productVariants (first: $numProducts, query: $sku_query, after: $cursor) { pageInfo { hasNextPage endCursor } edges { node { id sku legacyResourceId inventoryItem { id legacyResourceId inventoryLevels(first: 1) { edges { node { id available location { id } } } } } } } } }",
"variables" : {
"numProducts": 100,
"cursor": "{{{previous_page.full_response.data.productVariants.pageInfo.endCursor}}}",
"sku_query": "{{#each record.inventory_data as |inv_record|}}{{#compare @index ">" 0}} OR {{/compare}}sku:{{inv_record.sku}}{{/each}}"
}
}
{{/if}}
When I preview this data, this is how the records are displayed in the "Parsed output" tab on the preview pane:
{ "page_of_records": [
{ "record":
{ "node": { [Shopify product data here] } }
},
{ "record":
{ "node": { [Shopify product data here] } }
},
{ "record":
{ "node": { [Shopify product data here] } }
},
{ "record":
{ "node": { [Shopify product data here] } }
},
[...rest of product records]
]}
In addition, when I set the debug logs on this flow step, I am seeing 8 different logs appear when I run the flow, which also makes me think that my pagination is working.
Issue:
On the Results Mapping associated with this flow step, the Preview pane is only showing me the first record. Thinking this may be a bug with the Preview, I created a Javascript hook that intentionally breaks so I can review the actual data that appears in the "Edit retry data" window of the error display, but that also only shows me the first record. Through the remainder of my flow (which is only one other flow step), I am also only seeing the first record being passed through.
That said, how can I get all the records from this paginated flow step to pass through the remainder of my flow?
Comments
Hi Josh, a nice way to test if you pagination is working correctly is to set the pagesize to a low number, so in this case set "numProducts" to 2 on both query body's. Then click the preview button on the export. If the pagination is working correctly Celigo will do 5 calls, show 10 records on the Parsed Output tab, and show the last call (2 records) on the HTTP Response tab.
Even if pagination is working incorrectly I would not expect only the first record to be passed but the whole page, do you have any Result Mappings on the export?
Hi Bas, thank you for that. Yes I have tried that previously, and I believe it is working. When I set both queries to have numProducts of 2, in the Preview pane it shows 1 Page with 10 Records (since that is what I have set in the Preview area), the Parsed Output tab shows the 10 records, and the HTTP Response tab shows the last two records. Below is a screenshot of the Parsed Output tab and the HTTP Response tab:
Parsed Output:
HTTP Response:
I can't show the two records in a full screenshot, but here are the two records output from that display:
{ "data": { "productVariants": { "pageInfo": { "hasNextPage": true, "endCursor": "eyJsYXN0X2lkIjo0NDQ4MDg0NjIwMTE0OSwibGFzdF92YWx1ZSI6IjQ0NDgwODQ2MjAxMTQ5In0=" }, "edges": [ { "node": { "id": "gid://shopify/ProductVariant/44480846102845", "sku": "AC-220-WTRF-NA-130", "legacyResourceId": "44480846102845", "inventoryItem": { "id": "gid://shopify/InventoryItem/46528478511421", "legacyResourceId": "46528478511421", "inventoryLevels": { "edges": [ { "node": { "id": "gid://shopify/InventoryLevel/110539079997?inventory_item_id=46528478511421", "available": 72, "location": { "id": "gid://shopify/Location/74161586493" } } } ] } } } }, { "node": { "id": "gid://shopify/ProductVariant/44480846201149", "sku": "AC-220-GYRF-NA-125", "legacyResourceId": "44480846201149", "inventoryItem": { "id": "gid://shopify/InventoryItem/46528478609725", "legacyResourceId": "46528478609725", "inventoryLevels": { "edges": [ { "node": { "id": "gid://shopify/InventoryLevel/110539079997?inventory_item_id=46528478609725", "available": 41, "location": { "id": "gid://shopify/Location/74161586493" } } } ] } } } } ] } }, "extensions": { "cost": { "requestedQueryCost": 14, "actualQueryCost": 14, "throttleStatus": { "maximumAvailable": 10000, "currentlyAvailable": 9986, "restoreRate": 500 } } } }The only thing I'm not sure how to tell from the Preview section is that Celigo did 5 calls - I am only assuming that is the case. But when I set up the debug logger and actually run the flow, I see all the log files appear with the relevant data.
Any other ideas what the problem could be, or is this a bug in the platform?
I also tried using the "Next page token" pagination option, but that either gives me errors or I don't get any data beyond the first page. Per your documentation at https://docs.celigo.com/hc/en-us/articles/360059620151-API-pagination#Tokenpagingmethod, it says "For all path values related to pagination, verify that you are using a path that matches the record structure displayed in the HTTP Response preview section".
My HTTP Response for the first page looks like the following:
{"data": {
"productVariants": {
"pageInfo": {
"hasNextPage": true,
"endCursor": "eyJsYXN0X2lkIjo0NDQ4MDg0NTY3Njg2MSwibGFzdF92YWx1ZSI6IjQ0NDgwODQ1Njc2ODYxIn0="
},
Which would make me think the following "Next page token" configuration would be correct, but it is throwing an error as seen in the preview section:
Any ideas on what this issue is?
Can you change the following?
"Path to next page token field in HTTP response body *" -> data.productVariants.pageInfo.endCursor
"Path to paging complete field in HTTP response body" -> data.productVariants.pageInfo.hasNextPage
"Paging complete values" -> false
Thanks for that info, I tried that but still get the same error:
This makes some sense to me though since my initial query passes in the "cursor" parameter of null, so I would assume it is trying to use null for both paginated calls. That said, I tried removing all cursor references from my initial graphQL query, but I still get the same "page_response_error" code shown in the Preview above.
Quick update - I believe I got the pagination to work via the "Next Page Token" Paging method, but I am still running into the original issue I mentioned on this ticket where the paginated results are not being passed down the flow.
For anyone else running into this pagination issue using the "Next Page Token" paging method, I made the updates Scott mentioned in the previous comment, and then updated my original graphQL query above to add some conditional logic to the "cursor" variable (which I discovered via some documentation). Instead of having "cursor": null, I changed it to:
"cursor": {{#if export.http.paging.token}}"{{export.http.paging.token}}"{{else}}null{{/if}}Back to my original problem, I only have one other flow step that runs after this paginated data flow step. For reference, this is how the flow is structured (the flow step on the left is where I am paginating all the data, and the hook just performs some calculations and adds new data to be passed down to the final flow step):
Evidence of the pagination appearing to work is in the following. These are the logs that are generated from the "Shopify (GraphQL) Get Product and Inventory Location Data by SKU" step:
And then there is only one log generated from the "Shopify (GraphQL) Update Inventory on Product Variant" step, which only seems to be processing a single record (even though the query loops through an array of objects, and only a single object is included in the array that is being processed in this step):
Any idea why all the paginated data is not being passed over to my final flow step?
Shoot, after I pasted my screenshot of the response mapping, I noticed I was using data[0] instead of data[*]. Changing that allowed all the results to be passed through the remainder of the flow.
Everything is now working as expected. Thank you all for your help with this.
Please sign in to leave a comment.