Setting rate limits

[Tochemey]

Hello I would like to know how to enable rate limits. Maybe there is a way with the library or I will have to go the normal net.http route.

By the way thanks for the great library.

[Tochemey]

I open to look into it in a PR if there is no a simple way to do it at the moment

[earthboundkid]

I'm not quite sure what you're asking. This is for client requests, not server handling. You can pass in an http.RoundTripper to use, which means you can set the http.Transport.MaxConnsPerHost if you just want it to block if there are too many simultaneous requests to the same host. Another option would be to use requests. RoundTripFunc to write your own transport that limits concurrency, like

limit := make(chan bool, 10)
var myTransport requests.RoundTripFunc = func(req *http.Request) (res *http.Response, err error) {
  limit <- true
  res, err = http.DefaultTransport(req)
  <-limit
  return res, err
}
err := requests.
  URL(url).
  Transport(myTransport).
  Fetch(ctx)

For more complicated limiting, see this blog post for some general thoughts about managing concurrency in Go.

[Tochemey]

@carlmjohnson what I am asking is about how to inject rate limiting using golang.org/x/time/rate where we can do something like:

package main

import (
	"context"
	"fmt"
	"net/http"
	"time"

	"golang.org/x/time/rate"
)

type ClientWrapper struct {
	client      *http.Client
	Ratelimiter *rate.Limiter
}

func (c *ClientWrapper) Do(req *http.Request) (*http.Response, error) {
	ctx := context.Background()
	err := c.Ratelimiter.Wait(ctx) // This is a blocking call. Honors the rate limit
	if err != nil {
		return nil, err
	}
	resp, err := c.client.Do(req)
	if err != nil {
		return nil, err
	}
	return resp, nil
}

//NewClient return http client with a ratelimiter
func NewClient(rl *rate.Limiter) *ClientWrapper {
	c := &RLHTTPClient{
		client:      http.DefaultClient,
		Ratelimiter: rl,
	}
	return c
}

func main() {
	rl := rate.NewLimiter(rate.Every(10*time.Second), 50) // 50 request every 10 seconds
	c := NewClient(rl)
	reqURL := "<some-url>"
	req, _ := http.NewRequest("GET", reqURL, nil)
	for i := 0; i < 300; i++ {
		resp, err := c.Do(req)
		if err != nil {
			fmt.Println(err.Error())
			fmt.Println(resp.StatusCode)
			return
		}
		if resp.StatusCode == 429 {
			fmt.Printf("Rate limit reached after %d requests", i)
			return
		}
	}
}

[Tochemey]

It is not hard to do it using the library. I was asking whether it is something we have already or we can incorporate into the library.

[earthboundkid]

I don't want to add more dependencies to this repo unless absolutely necessary, so I don't want to add a dependency on golang.org/x/time/rate, but it looks like it should be pretty easy to create a separate ratelimited.Transport type that you could interoperate with this package.

[Tochemey]

Ok thanks for the time.