Skip to content

feat: send cookies with every request during a clone#58

Open
didimitrie wants to merge 1 commit into
tamnd:mainfrom
didimitrie:feat/request-cookies
Open

feat: send cookies with every request during a clone#58
didimitrie wants to merge 1 commit into
tamnd:mainfrom
didimitrie:feat/request-cookies

Conversation

@didimitrie

@didimitrie didimitrie commented Jun 22, 2026

Copy link
Copy Markdown

What

Adds a --cookie flag to kage clone so a site behind a login or a region/cookie wall can be mirrored. The cookies ride on every request a clone makes, not just the page render:

  • Chrome page navigations — seeded into the browser before the first load
  • Asset downloadsCookie header on the HTTP fetch
  • robots.txt fetch
  • sitemap.xml fetch

This keeps the authenticated session consistent across the whole run rather than only the rendered pages.

Usage

# single cookie
kage clone --cookie "session=abc" example.com

# repeat the flag
kage clone --cookie "session=abc" --cookie "theme=dark" example.com

# paste a whole Cookie header straight from devtools
kage clone --cookie "session=abc; theme=dark" example.com

Notes

  • Browser cookies are scoped to the seed host and its subdomains; the HTTP fetches send the same header.
  • Because off-domain assets are left on the live web by default, the cookies are not leaked to third-party hosts (a nice security property that falls out of the existing asset policy).
  • --cookie accepts a single name=value, repeats, or a full header in one value; malformed input is a clear usage error.

Tests

Added coverage for:

  • Config.CookieHeader serialisation (incl. empty-name skipping)
  • the asset downloader sending / omitting the Cookie header
  • fetchSitemap sending the cookie
  • the Chrome cookie-param building
  • CLI parseCookies, including malformed-input errors (the cli package had no tests before)

go build ./..., go vet ./..., gofmt, and the full go test ./... all pass.

README, CLI reference (docs/content/reference/cli.md), and CHANGELOG (Unreleased) updated.

🤖 Generated with Claude Code

Closes #9

Add a --cookie flag to `kage clone` that attaches cookies to all four
request paths a clone makes, so a site behind a login or a region/cookie
wall can be mirrored:

  - Chrome page navigations (seeded into the browser before first load)
  - asset downloads
  - the robots.txt fetch
  - the sitemap.xml fetch

The flag takes a single name=value pair, repeats for several cookies, or
accepts a whole Cookie header in one value ("a=1; b=2") pasted straight
from devtools. Browser cookies are scoped to the seed host and its
subdomains; the HTTP fetches send the same header. Off-domain assets are
left on the live web by default, so the cookies are not leaked to
third-party hosts.

Tests cover the header serialisation, the downloader/sitemap sending the
cookie, the Chrome cookie-param building, and the CLI flag parsing
(including malformed input). README, CLI reference, and CHANGELOG updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cookies?

1 participant