Why was my flow cancelled?
I had created a flow that reads from SP-API and writes to a single table
I had set the batch and page size set to 3000 and have run this flow with multiple pages without any issues.
Since this is a temp table that will be deleted after I merge it into the final table, I wanted to keep a history of all the exports requested from SP-API, because it is sometimes useful to see what the data looked like on a certain day in the past (I plan to run this job daily).
So, I created a branch so that when a page is read, that same page is written to both the temp table and a history table:
When I ran this new branched flow, it gave me the following results:
Both imports show as canceled.
The temp table contained 907 records (event though the run console shows 0 under the success column)
The history table was empty.
After reading https://docs.celigo.com/hc/en-us/community/posts/4720680616219-TROUBLESHOOTING-Why-is-my-job-within-a-flow-that-imports-to-FTP-seemingly-canceled-?input_string=Why%20was%20my%20flow%20cancelled%3F I figured that maybe my page/batch size might be some how affecting this. Maybe the export page is somehow duplicated and causes the 3M size to trigger a cancellation, so I reduced my page/batch size to 1000.
Here are the results:
Once again the run console reports canceled on both tables and that only one page was processed, but when I look at the tables in snowflake both tables contained 1206 rows.
Assuming that I was headed in the right direction, I again modified my page size and batch size from 1000 to 500, and re-ran the flow with the same start date parameters.
Confusingly, this time, the temp table table contained 0 rows and the history table contained 500 rows.
I had previously set the sp-api connection concurrency to 1 in an attempt to throttle my requests to the SP-API, and thought that this might be the problem, so I changed it to 3 thinking that maybe this might be causing a cancellation
This had no effect on the results.
I will continue experimenting, perhaps without branching.
Comments
Hello Alex Baeza,
Can you retry your flow? We had a recent release that might help solve these issues. If this doesn't work, we recommend opening a ticket so Support can review your configuration and work with you on a solution. Thanks!
Kathyana
Thank you, I will retry this.
I retried creating the branched flow below:
The export checks for data in the stage table. The two branches should insert into a history table (using sql) and also merge into a merge table (using sql).
What I want is for those to SQL statements to execute in parallel. My concurrency for the database is set to 10
These two sql statements work fine without the branch if I execute one step after the other.
Hi Alex Baeza,
We are looking into the page index issue reported for snowflake.
Could you please confirm whether the flow cancel issue first reported in this thread has been resolved.
As Kathyana suggested ,Please raise a ticket if you are still facing the flow cancellation issue.
Thanks,
Vikram
Please sign in to leave a comment.