Skip to main content

Can't get page size larger than 30?

Comments

5 comments

  • Scott Henderson CTO
    Answer Pro
    Top Contributor

    Yes, each page is limited to 5MB is size.  We will automatically cap the number of records when this size limit is reached, and we will auto route records to the next page of data, etc...

    In general, it is not recommended that you max out the pages in terms of MBs, and the 5MB is meant to be a never needed upper limit (but that helps us size and scale servers in our backend).

    There is no performance advantage to having 10 vs 50 pages of data in a typical flow, and your biggest lever for increasing data throughput in your flow is via the "Concurrency level" field on the connection resources.  i.e. You can set the concurrency on your Snowflake connection to 10 or 20 to drastically speed things up.  For your specific setup outlined above, I highly recommend setting the page size field to 10 max (i.e. because each of your records contains 100 orders, so this means each page will have 1000 orders, and this is a perfect per request size for Snowflake data loading IMO).

    Hope this helps.

    -Scott 

    0
  • Alex Baeza
    Answer Pro
    Celigo University Level 4: Legendary
    Awesome Follow-up

    Thanks for this insight.

    I was wondering, If instead of doing "one to many" in my Snowflake import bubble, I specify the path to the individual records in your Amazon source export, would this reduce my page size? This might be a small tweak that can help with the overall performance.

    0
  • Scott Henderson CTO
    Answer Pro
    Top Contributor

    Yeah I def recommending doing what I suggested here, and then once you make that change, you should set 'Page size' on your Amazon source export to 100 and test the performance, and then try 1000 and test the performance again, and then just pick a final number between 100 - 1000 based on the results of your perf tests.  If you dont have time to run this perf test, then just go with 100 and that should def be fast enough to get started.

    0
  • Kathyana Rule Community moderator Technical Writer

    Hi Alex Baeza

    Here are some articles that touch on Page size: 

    Let me know if you still can't find what you're looking for. Thanks! 

    0
  • Alex Baeza
    Answer Pro
    Celigo University Level 4: Legendary
    Awesome Follow-up

    Thanks for all your help.

    I ended up configuring the source to my records in the source and then changing the page size to 3000 (although I was able to go higher). 

    I then adjusted my Snowflake batch size to 3000, and that seems to work fine. 

    0

Please sign in to leave a comment.