I am making a bunch of web requests and am converting the bodies to utf-8 (the input encoding will vary for each page). There seems to be a memory leak and the following code will get to 1GB of memory after about 4-5 minutes. Using the ConvertString method, however, results in no such leak. My best guess is that the converter is not being closed when using the NewReader method? Thanks for this package, btw!
package main
import (
"bytes"
"context"
"io/ioutil"
"log"
"net/http"
"sync"
"time"
iconv "github.com/djimenez/iconv-go"
)
var h = `<html><head></head><body></body></html>`
func job() {
time.Sleep(1 * time.Second) // simulate web request
bdy := ioutil.NopCloser(bytes.NewReader([]byte(h)))
defer bdy.Close()
resp := &http.Response{
Body: bdy,
}
defer resp.Body.Close()
_, err := iconv.NewReader(resp.Body, "windows-1252", "utf-8") // memory leak
//_, err := iconv.ConvertString(h, "windows-1252", "utf-8") // no memory leak
if err != nil {
log.Fatalln(err)
}
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()
n := 1000
stop := make(chan bool)
wg := sync.WaitGroup{}
for worker := 0; worker < n; worker++ {
wg.Add(1)
go func(stop chan bool) {
defer wg.Done()
for {
select {
case <-stop:
return
default:
job()
}
}
}(stop)
}
<-ctx.Done()
for i := 0; i < n; i++ {
stop <- true
}
wg.Wait()
}
I am making a bunch of web requests and am converting the bodies to utf-8 (the input encoding will vary for each page). There seems to be a memory leak and the following code will get to 1GB of memory after about 4-5 minutes. Using the ConvertString method, however, results in no such leak. My best guess is that the converter is not being closed when using the NewReader method? Thanks for this package, btw!