Possible to sort across all pages of export?
Completed
Hello,
I have a flow that loads a CSV file from an FTP server. I do not have any control over the file or how it's generated. The sorting of the rows is incorrect and I need to sort it in IO. I added a preSavePage javascript function which works perfect on a page of results. The problem is when my data spans multiple pages, I can't sort across pages. I know you can increase the page size and I've done this, but there's a 5MB limit so this isn't a bulletproof solution.
My question: Is there a way in IO to combine pages or a hook/event to operate on all pages of results?
-Steve
-
Official comment
Shrehith Martin Jahn - Thanks for posting to the community. We have grouping capability available on FTP connector. In our upcoming release, this feature will be extended to Universal HTTP and Database connectors.
https://docs.celigo.com/hc/en-us/articles/360045238292-Export-files-from-an-FTP-server#groupSort
-
Hi Steve Klett,
Great to hear from you! Unfortunately, there's currently no way to handle this scenario, although we do have some concepts in the planning stage. Let me check with our PM on these, Vili, and get back to you. Thanks so much for letting us know about this need!
0 -
There is no alternative or a creative way in IO to sort file records that span multiple pages.
We have a feature backlog to sort file content to address your use case. I am gathering customer's feedback on how they plan or want to use this sort. Would you please explain what your flow scenario is and provide some examples of the file layout along with sorting rule.
Thanks
0 -
Hi,
I have similar ask where I need to sort and then use group by on CSV so I can eliminate the duplicates from CSV. I was thinking of having a parent work flow which dumps the CSV and then next work flow will consume it and do group by. Looks like it won't work as for group by we need to sort the content.
Is there any global variable which has scope within integration and can ne written and called in various flows so it will eliminate the CSV dump and can be sorted out using JS?
Thanks,
Abhinav Kaushal
0 -
Thanks for posting to the community, Abhinav Kaushal. I’m checking with our experts and will get to back to you with their recommendation.
0 -
Abhinav Kaushal - I heard back from our engineers. We don't have any global variable that has scope beyond the current execution. All variables are passed in as arguments for the hooks or other function being called.
0 -
Hello,
I am having the same issue. Do we have any solution developed to handle this scenario now?
0 -
Anyone have an update on this request?
We have source system that provides rows of data that need to be grouped before being sent to the destination system. We have a Pre Save Page script but unfortunately when Integrator IO splits the data set into Pages, it sometimes splits groups of records that should have been grouped together.
Current workaround is to import the raw data rows into a NetSuite custom record, then use a saved search and record grouping functionality from the Integrator IO Export from NetSuite functionality. This adds overhead as we need to write the custom records, then do an update to the custom record to either mark the rows as processed and/or delete the custom record entries.
0 -
UPDATE : We recently released the support for Grouping. You can now group your source records for HTTP and Database connectors. Please follow this article to learn more about this :
https://docs.celigo.com/hc/en-us/articles/4405373029019-Sort-and-group-records#Group
0
Please sign in to leave a comment.
Comments
9 comments