Option to do aggregate on the entire data set

Divya Navakumar

• June 16, 2022 18:22
• Edited

Hello,

It would be great to have an option to perform an aggregate on the entire data set - currently, I understand that it is limited by page size. It would be great if there is some kind of a workaround, like a global cache, which can be accessed during mapping as well.

Thanks,

Divya

1

• Official comment

  

  Bhavik Shah Senior Product Manager

  • November 18, 2022 13:21

  

  

  UPDATE : We recently released the support for Grouping. You can now group your source records for HTTP and Database connectors. Please follow this article to learn more about this : 

   

  https://docs.celigo.com/hc/en-us/articles/4405373029019-Sort-and-group-records#Group
• 

  Amanjot Singh Group Product Manager

  • June 21, 2022 19:09

  

  

  

  

  Hello Divya,
  
  Could you help me understand your use case in more detail? What operations are you planning to perform on aggregate data? It would be great to connect over zoom and understand the use case. Let me know if we can meet sometime over the next couple of weeks.

  Regards,

  1
• 

  Divya Navakumar

  • July 04, 2022 08:58

  

  

  Hello Amanjot Singh,

   

  It would be great to connect to discuss this issue. Please let me know your availability.

   

  Thanks,

  Divya

  0
• 

  Amanjot Singh Group Product Manager

  • July 05, 2022 21:45

  

  

  

  

  Hello Divya,

  I just sent you an email

  0
• 

  Marco Pietersen

  • October 05, 2022 09:45

  Did anything come of this? 
  It's really frustrating that the payload as a whole isn't accessible. And the payload that is accessible has different structure in the different blocks. 

  0
• 

  Bhavik Shah Senior Product Manager

  • October 06, 2022 17:44
  • Edited

  

  

  We are coming up with a feature in our upcoming release. You should be able to group the source data for HTTP and Database connectors. 
  
  You can find the list of upcoming features, vote or comment on them and submit a new idea using the Product Portal. 

   

  

   

   

  0
• 

  Bhavik Shah Senior Product Manager

  • October 06, 2022 17:56

  

  

  Marco Pietersen - Could you please clarify this further ? The payload should be accessible as long as it is not exceeding the defined page size. 
  
  https://docs.celigo.com/hc/en-us/articles/360043927292-Fine-tune-integrator-io-for-optimal-data-throughput#page-size

  0
• 

  Marco Pietersen

  • October 11, 2022 13:53

  yes, but sometimes it's
  page_of_records": [    {       "record": {
  Sometimes it's record, sometimes its parent record, sometimes its data. Sometimes options are availble sometimes not...
  The root of the payload is dynamic, not a fixed template.
  
  I'm not talking about accessibility, but about structure of the elements. 

   

   

  0

Please sign in to leave a comment.