Automatically restart process at job completion

Steven Smith

• April 04, 2024 16:12

I am building a process flow that is going to do a 1 time import of around 3 million records from a SQL server DB into NetSuite. I am planning on doing it in batches of 100,000 as I am not sure how well it would run if my initial SQL statement pulled all 3 million records at once. It currently takes about 1 hour and 12 minutes to import 100,000 records and then mark them as processed.

1. Is there a way within a process flow to automatically restart the job at the end of the process until all the records are processed so that I dont have to worry about timing gaps between schedules?
2. If it can't be automatically restarted and I have to run it on a schedule, what would happen if there is a delay in the process and the next schedule queues up while the job is still running?

0

• 

  David Gollom Strategic Partner

  • April 04, 2024 16:17

  

  

  

  

  

  You can use the Next Integration Flow option in the flow settings and select the same flow. This will create an infinite loop!

  0
• 

  Steven Smith

  • April 04, 2024 16:30

  For some reason that field doesn't show me the current process flows, only other process flows.

  0
• 

  David Gollom Strategic Partner

  • April 04, 2024 16:40

  

  

  

  

  

  They took that loophole away!!  Sorry.  If you schedule it frequently it should be fine. It just won't run if it's alrady running. That's been my experience anyway.  

  0
• 

  Tyler Lamparter Principal Product Manager

  • April 04, 2024 20:20

  

  

  

  

  

  Steven Smith you can use next integration flow option to run a flow that has a dummy export on it (i.e. exporting 1 single record) and then you have an import step that calls the /v1/flows/<flowId>/run api to trigger the first flow to run again. The export on your second flow could be a select count(*) from unprocessed_records. That way it will only loop if there are still records to be processed. 

  1
• 

  Scott Henderson CTO

  • April 04, 2024 23:30

  

  

  

  Regarding "If it can't be automatically restarted and I have to run it on a schedule, what would happen if there is a delay in the process and the next schedule queues up while the job is still running?"

  It is totally harmless to schedule something more frequently then it is able to finish, and it will ignore any new scheduled runs if there is still something in progress.

  0

Please sign in to leave a comment.