Indexing and Incrementing an Identifer
I am creating an XML file and I need to increment the inboundDocumentidentifer for each record. I have three separate flows that need to do this. On all three I have tried setting my Page Size to 10,000 in hopes that I could do it through the @index function as you see below. On one flow that worked since I didn't have that many records, but on the other 2 flows it is not and is throwing me a 5MB page size error.
Whats the best way to solve for this? I know I will need to reduce my page size (might be able to do 100 records/page but will have to test), but do I need a pre-map step on the import for the indexing before importing into AWS S3 and converting to XML? Again, the inboundDocumentidentifier needs to increment across pages instead of reseting to 0 for each new page. I've included a few screenshots below.
Import Step
Sample of XML header with @index that works if all records are on one page
Comments
Dave Guderian only way would be to use the state api. Here is an example used for an EDI project. Probably would need concurrency of 1 for your connection on your import step.
https://docs.celigo.com/hc/en-us/community/posts/21854609671195/comments/22324964406683
Tyler Lamparter - Is it possible to set the concurrency for AWS S3 to 1? I'm not seeing that option when I go to that connection.
Tyler Lamparter - My engineering manager was able to take the Java Script you wrote and modify it for our purposes and its working great now. One remaining question is if I can reset the counter to 0 for each flow run. Are there any values automatically passed per flow run that we could cache and then key off of for the next flow run? Since they would (hopefully) be different per flow run that would allow us I think to reset the counter to 0 for each flow run.
Alternatively, could I do something in the settings as below? Would that work?
Dave Guderian settings isn't a good place for something like this since it requires constantly updating the IO resource. To reset the counter, add a preSavePage script to the export on your flow and then modify the script above to reset the counter when pageIndex = 0. The input to the preSavePage script has a field called pageIndex.
Dave Guderian you potentially could also add the pageIndex and recordIndex from the preSavePage script into every record and then use those as the sequence. Your sequence number would be (pageIndex + 1) * (recordIndex + 1). Since the index starts at 0, I added 1 to them. Since the S3 connector uses a concurrency of 1 to build the file, the pages should go in order, but I haven't tested/verified that.
Please sign in to leave a comment.