Paginated data not being passed through remainder of flow

Comments

7 comments

  • Bas van Ditzhuijzen Strategic Partner NetSuite Integration Whiz Salesforce Integration Whiz
    Partner Implementation Certification
    Great Answer
    Answer Pro
    Engaged
    Celigo University Level 4: Legendary

    Hi Josh, a nice way to test if you pagination is working correctly is to set the pagesize to a low number, so in this case set "numProducts" to 2 on both query body's. Then click the preview button on the export. If the pagination is working correctly Celigo will do 5 calls, show 10 records on the Parsed Output tab, and show the last call (2 records) on the HTTP Response tab.

    Even if pagination is working incorrectly I would not expect only the first record to be passed but the whole page, do you have any Result Mappings on the export?

    0
  • Josh G

    Hi Bas, thank you for that.  Yes I have tried that previously, and I believe it is working.  When I set both queries to have numProducts of 2, in the Preview pane it shows 1 Page with 10 Records (since that is what I have set in the Preview area), the Parsed Output tab shows the 10 records, and the HTTP Response tab shows the last two records.  Below is a screenshot of the Parsed Output tab and the HTTP Response tab:

    Parsed Output:

    HTTP Response:
    I can't show the two records in a full screenshot, but here are the two records output from that display:

    { "data": { "productVariants": { "pageInfo": { "hasNextPage": true, "endCursor": "eyJsYXN0X2lkIjo0NDQ4MDg0NjIwMTE0OSwibGFzdF92YWx1ZSI6IjQ0NDgwODQ2MjAxMTQ5In0=" }, "edges": [ { "node": { "id": "gid://shopify/ProductVariant/44480846102845", "sku": "AC-220-WTRF-NA-130", "legacyResourceId": "44480846102845", "inventoryItem": { "id": "gid://shopify/InventoryItem/46528478511421", "legacyResourceId": "46528478511421", "inventoryLevels": { "edges": [ { "node": { "id": "gid://shopify/InventoryLevel/110539079997?inventory_item_id=46528478511421", "available": 72, "location": { "id": "gid://shopify/Location/74161586493" } } } ] } } } }, { "node": { "id": "gid://shopify/ProductVariant/44480846201149", "sku": "AC-220-GYRF-NA-125", "legacyResourceId": "44480846201149", "inventoryItem": { "id": "gid://shopify/InventoryItem/46528478609725", "legacyResourceId": "46528478609725", "inventoryLevels": { "edges": [ { "node": { "id": "gid://shopify/InventoryLevel/110539079997?inventory_item_id=46528478609725", "available": 41, "location": { "id": "gid://shopify/Location/74161586493" } } } ] } } } } ] } }, "extensions": { "cost": { "requestedQueryCost": 14, "actualQueryCost": 14, "throttleStatus": { "maximumAvailable": 10000, "currentlyAvailable": 9986, "restoreRate": 500 } } } }

    The only thing I'm not sure how to tell from the Preview section is that Celigo did 5 calls - I am only assuming that is the case.  But when I set up the debug logger and actually run the flow, I see all the log files appear with the relevant data.

    0
  • Josh G

    Any other ideas what the problem could be, or is this a bug in the platform? 

    I also tried using the "Next page token" pagination option, but that either gives me errors or I don't get any data beyond the first page.  Per your documentation at https://docs.celigo.com/hc/en-us/articles/360059620151-API-pagination#Tokenpagingmethod, it says "For all path values related to pagination, verify that you are using a path that matches the record structure displayed in the HTTP Response preview section".

    My HTTP Response for the first page looks like the following:

    {
      "data": {
        "productVariants": {
          "pageInfo": {
            "hasNextPage": true,
          "endCursor": "eyJsYXN0X2lkIjo0NDQ4MDg0NTY3Njg2MSwibGFzdF92YWx1ZSI6IjQ0NDgwODQ1Njc2ODYxIn0="
          },

    Which would make me think the following "Next page token" configuration would be correct, but it is throwing an error as seen in the preview section:

    Any ideas on what this issue is?

    0
  • Scott Henderson CTO
    Celigo University Level 1: Skilled
    Answer Pro
    Top Contributor

    Can you change the following?

    "Path to next page token field in HTTP response body" -> data.productVariants.pageInfo.endCursor

    "Path to paging complete field in HTTP response body" -> data.productVariants.pageInfo.hasNextPage

    "Paging complete values" -> false

    0
  • Josh G

    Thanks for that info, I tried that but still get the same error:

    This makes some sense to me though since my initial query passes in the "cursor" parameter of null, so I would assume it is trying to use null for both paginated calls.  That said, I tried removing all cursor references from my initial graphQL query, but I still get the same "page_response_error" code shown in the Preview above. 

    0
  • Josh G

    Quick update - I believe I got the pagination to work via the "Next Page Token" Paging method, but I am still running into the original issue I mentioned on this ticket where the paginated results are not being passed down the flow.  

    For anyone else running into this pagination issue using the "Next Page Token" paging method, I made the updates Scott mentioned in the previous comment, and then updated my original graphQL query above to add some conditional logic to the "cursor" variable (which I discovered via some documentation).  Instead of having "cursor": null, I changed it to:

    "cursor": {{#if export.http.paging.token}}"{{export.http.paging.token}}"{{else}}null{{/if}}

    Back to my original problem, I only have one other flow step that runs after this paginated data flow step.  For reference, this is how the flow is structured (the flow step on the left is where I am paginating all the data, and the hook just performs some calculations and adds new data to be passed down to the final flow step):

    The odd thing is when I look in that response mapping node, it only ever shows the first record when I click Preview.  For reference, this is what the mapping looks like showing only one record being returned:

    Evidence of the pagination appearing to work is in the following.  These are the logs that are generated from the "Shopify (GraphQL) Get Product and Inventory Location Data by SKU" step:

    And then there is only one log generated from the "Shopify  (GraphQL) Update Inventory on Product Variant" step, which only seems to be processing a single record (even though the query loops through an array of objects, and only a single object is included in the array that is being processed in this step):

    Any idea why all the paginated data is not being passed over to my final flow step?

    0
  • Josh G

    Shoot, after I pasted my screenshot of the response mapping, I noticed I was using data[0] instead of data[*].  Changing that allowed all the results to be passed through the remainder of the flow.

    Everything is now working as expected. Thank you all for your help with this.

    0

Please sign in to leave a comment.