Option to do aggregate on the entire data set
Hello,
It would be great to have an option to perform an aggregate on the entire data set - currently, I understand that it is limited by page size. It would be great if there is some kind of a workaround, like a global cache, which can be accessed during mapping as well.
Thanks,
Divya
1
Comments
UPDATE : We recently released the support for Grouping. You can now group your source records for HTTP and Database connectors. Please follow this article to learn more about this :
https://docs.celigo.com/hc/en-us/articles/4405373029019-Sort-and-group-records#Group
Hello Divya,
Could you help me understand your use case in more detail? What operations are you planning to perform on aggregate data? It would be great to connect over zoom and understand the use case. Let me know if we can meet sometime over the next couple of weeks.
Regards,
Hello Amanjot Singh,
It would be great to connect to discuss this issue. Please let me know your availability.
Thanks,
Divya
Hello Divya,
I just sent you an email
Did anything come of this?
It's really frustrating that the payload as a whole isn't accessible. And the payload that is accessible has different structure in the different blocks.
We are coming up with a feature in our upcoming release. You should be able to group the source data for HTTP and Database connectors.
You can find the list of upcoming features, vote or comment on them and submit a new idea using the Product Portal.
Marco Pietersen - Could you please clarify this further ? The payload should be accessible as long as it is not exceeding the defined page size.
https://docs.celigo.com/hc/en-us/articles/360043927292-Fine-tune-integrator-io-for-optimal-data-throughput#page-size
yes, but sometimes it's
page_of_records": [ { "record": {
Sometimes it's record, sometimes its parent record, sometimes its data. Sometimes options are availble sometimes not...
The root of the payload is dynamic, not a fixed template.
I'm not talking about accessibility, but about structure of the elements.
Please sign in to leave a comment.