hung jobs

[bookvik]

this happens whenever something goes wrong in a problem. some kind of exceptional situation.

can you give advice on how to deal with this?

as a result, the workers end and everything gets up because new tasks are waiting for release and old ones are not released

-client, :supertype :tasklet, :workload-fn #titanoboa.exp.Expression{:value io.titanoboa.tasklet.httpclient/request, :type nil}, :next [], :properties {:url http://root_server:3000/purchase, :request-method :patch, :response-property-name :rest-response, :query-params {:id #titanoboa.exp.Expression{:value (format "eq.%s" (:id (:purchase *properties*))), :type clojure}}, :body #titanoboa.exp.Expression{:value (let [purchase (:purchase *properties*)]
tb_1             |   (let [stream (:stream purchase)]
tb_1             |     (clojure.data.json/write-str
tb_1             |       {:pre_started_at (:pre_started_at stream)
tb_1             |        :started_at     (:started_at stream)
tb_1             |        :ended_at       (:ended_at stream)}))), :type clojure}, :content-type :json}, :revision 2} {:id foobar, :type :custom, :supertype :tasklet, :workload-fn #titanoboa.exp.Expression{:value (fn [p] ), :type clojure}, :next [], :properties {}}]}, :jobdir #object[java.io.File 0x7552a745 job-folders/9bfa777a-4045-4b8f-9abb-0341cb46886e], :start #inst "2021-04-14T17:44:49.431-00:00", :history [], :state :initial, :suspendable? nil, :jobid 9bfa777a-4045-4b8f-9abb-0341cb46886e, :callback-ch nil, :create-folder? true, :next-step nil, :step nil} ] into new jobs channel...
tb_1             |  INFO [Thread-15] - Initializing a new job; First step will be: [ load-purchase ]
tb_1             |  INFO [Thread-15] - Retrieved job [ 9bfa777a-4045-4b8f-9abb-0341cb46886e ] from jobs channel; Starting step [ load-purchase ] with thread stack  nil ; Command is  nil
tb_1             |  INFO [Thread-15] - orchestrate-step called with results: state:  :running  next-step:  convert-to-json  thread-stack:  nil
tb_1             |  INFO [Thread-15] - Next step is  convert-to-json ; Submitting into jobs channel for next step's processing...
tb_1             |  INFO [Thread-15] - Acking main message for step  load-purchase  with thread stack  nil
tb_1             |  INFO [Thread-15] - Initializing a next step; next step [ convert-to-json ] was found among steps as step [ convert-to-json ]
tb_1             |  INFO [Thread-15] - Retrieved job [ 9bfa777a-4045-4b8f-9abb-0341cb46886e ] from jobs channel; Starting step [ convert-to-json ] with thread stack  nil ; Command is  nil
tb_1             |  INFO [Thread-15] - orchestrate-step called with results: state:  :running  next-step:  [update-purchase-dates foobar]  thread-stack:  nil
tb_1             |  INFO [Thread-15] - Next step is  [update-purchase-dates foobar] ; Submitting into jobs channel for next step's processing...
tb_1             |  INFO [Thread-15] - Dispatching multiple parallel next steps:  [{:step-id convert-to-json, :next-step update-purchase-dates, :thread-id :main, :callback-chan nil, :jobid 9bfa777a-4045-4b8f-9abb-0341cb46886e} {:step-id convert-to-json, :next-step foobar, :thread-id :thread-1, :callback-chan #titanoboa.channel.rmq.RMQPollingChannel{:queue amq.gen-8qTFG9TQFCoPaTxVFGKQeQ, :auto-ack false, :poll-interval 100, :exchange nil}, :jobid dbafd5b4-0375-466a-8682-033af408ca50}]
tb_1             |  INFO [Thread-15] - Acking main message for step  convert-to-json  with thread stack  nil
tb_1             |  INFO [Thread-15] - Initializing a next step; next step [ update-purchase-dates ] was found among steps as step [ update-purchase-dates ]
tb_1             |  INFO [Thread-15] - Retrieved job [ 9bfa777a-4045-4b8f-9abb-0341cb46886e ] from jobs channel; Starting step [ update-purchase-dates ] with thread stack  [[convert-to-json :main]] ; Command is  nil
tb_1             |  INFO [Thread-14] - Initializing a next step; next step [ foobar ] was found among steps as step [ foobar ]
tb_1             |  INFO [Thread-14] - Retrieved job [ dbafd5b4-0375-466a-8682-033af408ca50 ] from jobs channel; Starting step [ foobar ] with thread stack  [[convert-to-json :thread-1]] ; Command is  nil
tb_1             |  INFO [Thread-14] - orchestrate-step called with results: state:  :finished  next-step:  nil  thread-stack:  [[convert-to-json :thread-1]]
tb_1             |  INFO [Thread-14] - Acking main message for step  foobar  with thread stack  [[convert-to-json :thread-1]]
tb_1             |  INFO [Thread-15] - orchestrate-step called with results: state:  :finished  next-step:  nil  thread-stack:  [[convert-to-json :main]]
tb_1             |  INFO [Thread-15] - Merging job with thread stack [ [[convert-to-json :thread-1]] ] into the main thread... 
tb_1             |  WARN [Thread-15] - Something went wrong during processing of a step. Stopping job...
tb_1             | java.lang.UnsupportedOperationException: nth not supported on this type: PersistentArrayMap
tb_1             | 	at clojure.lang.RT.nthFrom(RT.java:987)
tb_1             | 	at clojure.lang.RT.nth(RT.java:936)
tb_1             | 	at titanoboa.processor$finalize_job_BANG_.invokeStatic(processor.clj:816)
tb_1             | 	at titanoboa.processor$finalize_job_BANG_.doInvoke(processor.clj:800)
tb_1             | 	at clojure.lang.RestFn.invoke(RestFn.java:500)
tb_1             | 	at titanoboa.processor$start_processor_BANG_$fn__19454.invoke(processor.clj:905)
tb_1             | 	at titanoboa.processor$start_processor_BANG_.invokeStatic(processor.clj:867)
tb_1             | 	at titanoboa.processor$start_processor_BANG_.invoke(processor.clj:847)
tb_1             | 	at titanoboa.processor.JobWorker$fn__19550.invoke(processor.clj:932)
tb_1             | 	at clojure.lang.AFn.run(AFn.java:22)
tb_1             | 	at java.lang.Thread.run(Thread.java:748)

[mikub]

@bookvik looks like a bug, there has been a major change to how steps of type :join are processed recently, so there may have been some bugs introduced. Will look at it when available, would it be possible to share the workflow? Are the parallel threads of the workflow ever merged together with a join?

[bookvik]

YEP, but

The first time I encountered such a problem was when, instead of http, I put https on an internal call to the server. that is, the worker crashed with some kind of error but could not finish everything correctly

and this is generally a big problem because it is not clear how to kill a worker if he is busy for a very long time with some kind of garbage (and kill correctly), but how to say on the contrary that for a long time is right

{:first-step "load-purchase",
 :name "purchase-created",
 :revision 37,
 :type nil,
 :properties {:purchase-id ""},
 :steps [{:id "load-purchase",
          :type :http-client,
          :supertype :tasklet,
          :next [["*" "convert-to-json"]],
          :workload-fn #titanoboa.exp/Expression{:value "io.titanoboa.tasklet.httpclient/request",
                                                 :type nil},
          :properties {:url "http://root_server:3000/purchase",
                       :request-method :get,
                       :response-property-name :purchase,
                       :headers {:accept "application/vnd.pgrst.object+json"},
                       :query-params {:id #titanoboa.exp/Expression{:value "(format \"eq.%s\" (:purchase-id *properties*))",
                                                                    :type "clojure"},
                                      :select "*,stream(*),scenario(*,scenario_rule(*))"}},
          :revision 2}
         {:id "convert-to-json",
          :type :custom,
          :supertype :tasklet,
          :workload-fn #titanoboa.exp/Expression{:value "(fn [p] {:properties\n         {:purchase (clojure.data.json/read-str\n                      (:body (:purchase *properties*))\n                      :key-fn keyword)}})",
                                                 :type "clojure"},
          :next [["*" "update-purchase-dates"] ["*" "foobar"]],
          :properties {},
          :allow-parallel? true}
         {:id "update-purchase-dates",
          :type :http-client,
          :supertype :tasklet,
          :workload-fn #titanoboa.exp/Expression{:value "io.titanoboa.tasklet.httpclient/request",
                                                 :type nil},
          :next [],
          :properties {:url "http://root_server:3000/purchase",
                       :request-method :patch,
                       :response-property-name :rest-response,
                       :query-params {:id #titanoboa.exp/Expression{:value "(format \"eq.%s\" (:id (:purchase *properties*)))",
                                                                    :type "clojure"}},
                       :body #titanoboa.exp/Expression{:value "(let [purchase (:purchase *properties*)]\n  (let [stream (:stream purchase)]\n    (clojure.data.json/write-str\n      {:pre_started_at (:pre_started_at stream)\n       :started_at     (:started_at stream)\n       :ended_at       (:ended_at stream)})))",
                                                       :type "clojure"},
                       :content-type :json},
          :revision 2}
         {:id "foobar",
          :type :custom,
          :supertype :tasklet,
          :workload-fn #titanoboa.exp/Expression{:value "(fn [p] :ok)",
                                                 :type "clojure"},
          :next [],
          :properties {}}]}

[bookvik]

this is a certain process that gets a purchase on api and even makes a purchase in it

[bookvik]

and in general, even the syntax error of the tasklet must end correctly, because this is essentially a level inside the application when we can definitely avoid emergency stops

[bookvik]

by the way I made syntax errors and they end more correctly

but with what I described above, problems turned out

[bookvik]

I tried to remove parallel execution in the last step and everything is done

as soon as you add it back - it breaks

[bookvik]

launched a whole bunch of tasks enabled / disabled workers

looks even weirder now

[mikub]

Hi @bookvik,
so looking at the workflow definition it seems that it is an unsupported use case at this stage - you are not joining the workflow threads back in. Currently that is the assumption - branch into as many threads you like, but the threads assume that they will reach a :join step at some stage.

It shouldn't be that hard to add support for this though, but will require some thinking through. In the meantime - could you try to add a step of :supertype :join and see if that helps?

In regards to stopping hung jobs: if there is a chance it could happen you can use :suspendable attribute: https://github.com/mikub/titanoboa/wiki/Designing-Workflows#suspendable . If the job hangs you can then simply

1. suspend the job (this sends a suspend signal to all job's threads)
2. in "systems" tab stop the workers it ran on. You can see host of given node and worker number that is processing the given step in the job visualization - it is the last number at server info for the current step - i.e. format is IP:port (worker).
3. t-boa will attempt to restart the current steps on different workers, but then it will get the suspension singnal so it will stop the job