feat: add a crawl delay function in kage clone to honor the Crawl-del…

[Xirui]

Add a crawl delay function in kage clone to honor the Crawl-delay directive parsed from robots.txt.

The next step would be adding a user flag (--crawl-delay) under clone cmd to resolve #6 .

[tamnd]

Love this. It adds more respect for robots.txt and lets users control the delay.

However, I found an awesome package that already implements the rate-limiting logic, and we could reuse it: https://pkg.go.dev/golang.org/x/time/rate

import "golang.org/x/time/rate"

Then add a field:

type Cloner struct {
    ...
    crawlLimiter *rate.Limiter
}

And the setup is just:

func (c *Cloner) setupCrawlDelayLimiter() {
    if !c.cfg.RespectRobots || c.robots == nil || c.robots.CrawlDelay <= 0 {
        c.crawlLimiter = nil
        return
    }

    c.crawlLimiter = rate.NewLimiter(rate.Every(c.robots.CrawlDelay), 1)
}

Then your function becomes:

func (c *Cloner) waitForCrawlDelay(ctx context.Context) bool {
    if c.crawlLimiter == nil {
        return true
    }

    return c.crawlLimiter.Wait(ctx) == nil
}

Underlying, it is almost same logic: https://cs.opensource.google/go/x/time/+/refs/tags/v0.15.0:rate/rate.go;l=253

[Xirui]

Thx for the suggestion!

I've amended the commit to use rate pkg and added the --crawl-delay user flag.

[tamnd]

lgtm, I will cut new release soon!