Context
I'm got a project where I return s3 input streams as :body, as a way to send s3 data through my app without loading the entire file in memory all at once. This works fine most of the time. But when the http request is aborted early, .close is not being called on the inputStream. In the case of java s3, not closing the inputSteams is quite bad as in that case, the library stops working after 50 open requests.
Example:
..
(:require [amazonica.aws.s3 :as s3])
...
(GET "/stream" _
{:status 200
:body (:input-stream (s3/get-object {:endpoint region} bucket key))})
curl http://localhost:4000/stream --compressed | head -c 5 > /dev/null
The above will not correctly close the s3 stream - I've verified this using a debugger.
Suspicious code
https://github.com/bertrandk/ring-gzip/blob/ccd0a70bdbbff491820bbfd2be12ed4a70359380/src/ring/middleware/gzip.clj#L70C1-L81C11
(defn- compress-body
[body]
(let [p-in (PipedInputStream.)
p-out (PipedOutputStream. p-in)]
(future
(with-open [out (GZIPOutputStream. p-out)]
(if (seq? body)
(doseq [string body] (io/copy (str string) out))
(io/copy body out))) <-------- io/copy throws IOException when out is closed early
(when (instance? Closeable body)
(.close ^Closeable body))) <--- in that case, .close isn't called! (should be in a finally)
p-in))
Thanks for your attention!
Context
I'm got a project where I return s3 input streams as :body, as a way to send s3 data through my app without loading the entire file in memory all at once. This works fine most of the time. But when the http request is aborted early, .close is not being called on the inputStream. In the case of java s3, not closing the inputSteams is quite bad as in that case, the library stops working after 50 open requests.
Example:
The above will not correctly close the s3 stream - I've verified this using a debugger.
Suspicious code
https://github.com/bertrandk/ring-gzip/blob/ccd0a70bdbbff491820bbfd2be12ed4a70359380/src/ring/middleware/gzip.clj#L70C1-L81C11
Thanks for your attention!