clojure - Clojure http-kit get request stuck on multiple async calls

Question

I have created small example which highlight the problem:

(->> (range 0 4)
     (mapv (fn [i]
             (http/get "http://http-kit.org/"
                       (fn [res]
                         (info "first callback")
                         (let [res2 @(http/get "http://http-kit.org/")]
                           (info "second callback ")))))))

It's stuck on printing the 4s first callback msg's.

If I change the range for 0..3 it will work, the sync version also works.

Update:

The (info) is a taoensso.timbre logging library

Answer 1

My current hypothesis is that you get into a deadlock by exhausting your thread-pool:

1. You create a thread per outer http/get
2. If you create less requests than available threads in the thread pool, there is room to service at least one inner http/get (which will require a new thread)
   1. Or if your first request is completed before you exhaust the thread-pool
3. Once there are no more threads in the thread-pool, the inner http/get cannot be serviced
4. Since the inner request cannot be completed, the outers are stuck forever

You can check the status of the thread-pool http-kit uses peeking http/default-pool. There you can see things like:

#object[java.util.concurrent.ThreadPoolExecutor 0x5a99e5c "java.util.concurrent.ThreadPoolExecutor@5a99e5c[Running, pool size = 8, active threads = 0, queued tasks = 0, completed tasks = 24]"]

when you did not get into the deadlock. Or

#object[java.util.concurrent.ThreadPoolExecutor 0x5a99e5c "java.util.concurrent.ThreadPoolExecutor@5a99e5c[Running, pool size = 8, active threads = 8, queued tasks = 8, completed tasks = 28]"]

when you did.

I have tested this in my machine (shows 8 as (.availableProcessors (Runtime/getRuntime))) and I got the results above. I walked into a deadlock when I run more than 8 requests.

Regards

Answer 2

It looks like the issue caused because the http-kit client thread pool doesn't relieve the thread until callback function is finished. And so it ends up with run out of threads.

So I started to think how to make callback function quicker and came up with this solution:

I have created async wrapper function for http-kit client get which using clojure.core.async/chan in callback to quickly put the result in channel and then wait for it results and execute the heavy callback:

(defn async-http-get
  [url opts callback]
  (let [channel (chan)]
    (http/get url opts #(go (>! channel %)))
    (go (callback (<! channel)))
    nil))

So now using async-http-get instead of http/get solve the problem for me.

(->> (range 0 4)
     (mapv (fn [i]
             (async-http-get "http://http-kit.org/"
                       (fn [res]
                         (info "first callback")
                         (let [res2 @(http/get "http://http-kit.org/")]
                           (info "second callback ")))))))