Skip to content

✨ feat: runtime, env, and headers for site scripts#14

Merged
vaayne merged 5 commits into
mainfrom
feat/tap-sites
May 12, 2026
Merged

✨ feat: runtime, env, and headers for site scripts#14
vaayne merged 5 commits into
mainfrom
feat/tap-sites

Conversation

@vaayne

@vaayne vaayne commented May 12, 2026

Copy link
Copy Markdown
Owner

Summary

Adds per-script metadata for declaring execution environment, environment variable dependencies, and HTTP headers. This enables:

  1. Pure API scripts that skip the browser entirely (runtime: "http")
  2. Auth-injected scripts that derive headers from env vars (headers: {"X-API-Key": "${EXA_API_KEY}"})

Phases implemented

Phase 1 — Type changes and env validation

  • Added EnvDef struct (mirrors ArgDef) and Env map[string]EnvDef to Meta
  • Added ValidateEnv() — checks required env vars are set, lists all missing
  • Added ResolveHeaders() — interpolates ${ENV_VAR} via os.LookupEnv, skips headers with unset optional vars
  • Wired ValidateEnv() into Client.RunScript before engine dispatch

Phase 2 — Runtime routing

  • Added enginesByRuntime() on Client — filters engine list by script-declared runtime
  • Maps: "http" → QuickJS only, "browser"/"lightpanda" → Browser only, "auto"/"" → fallback chain
  • Respects forceBrowser user override (all runtimes return browser-only)
  • Empty engine list returns a descriptive error before dispatch

Phase 3 — Header injection in engines

  • Added RunOpts{Headers} struct; changed Engine.Run signature to accept it
  • QuickJS.injectFetch: meta headers set first, JS-level headers override them
  • transport.BrowseEvalWithPause: injects network.SetExtraHTTPHeaders before Navigate when headers are non-empty
  • Client.RunScript resolves headers via s.Meta.ResolveHeaders() and passes them as RunOpts.Headers

Phase 4 — CLI display and sample script

  • site info shows Runtime, Env (with required/optional badges), and Headers (masked)
  • site list shows runtime badges (e.g., [http]) after descriptions
  • Updated sites/exa/search.js with runtime: "http", optional EXA_API_KEY, and X-API-Key header
  • Registry test validates all new fields parse correctly from the embedded script

Deferred (not in this PR)

  • Phase 5: authRequired cookie store check
  • Phase 6: Web/D1 schema updates

Verification

  • mise run test — all packages pass
  • mise run lint — 0 issues

- Add EnvDef/Env to script Meta with ValidateEnv and ResolveHeaders
- Add runtime-based engine routing (http → QuickJS, browser → Browser)
- Inject resolved headers into QuickJS fetch and Browser CDP
- Update CLI info/list commands to display runtime/env/headers
- Add full metadata to exa/search sample script
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented May 12, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
tap 832d2cc May 12 2026, 04:08 AM

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5861896d6d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tap.go
Comment on lines +101 to +104
return script.NewRegistry(
script.Source{Path: cacheDir, Type: script.ScriptSourceCache},
script.Source{FS: sites.FS, Type: script.ScriptSourceBuiltin},
script.Source{Path: overrideDir, Type: script.ScriptSourceOverride},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Exclude built-ins when caller requests local-only scripts

DefaultRegistry always injects sites.FS, so callers cannot construct a registry from only one filesystem path. In cmd/tap/main.go, --local-only sets WithSitesDir(localOverrideDir) specifically to expose only local override scripts, but this change now still loads embedded scripts as well, violating the flag’s documented behavior and causing site list/script resolution to include non-local entries.

Useful? React with 👍 / 👎.

vaayne added 4 commits May 12, 2026 10:21
- Narrow embed pattern to */*.js (exclude markdown files from binary)
- Remove lightpanda from runtime routing (needs --lightpanda CLI flag)
- Sort ValidateEnv error output for deterministic messages
- Fix siteInfoCmd formatting consistency (spacing, no header masking)
Sync script now accepts multiple directories; later dirs override
earlier ones by script name, matching the CLI registry priority.
- Add lightpanda as explicit browser runtime instead of falling through
- forceBrowser now returns only browser engines, not the full list
- Mask env-interpolated header values in site info output
- Fix const alignment in ScriptSource block
- Add twitter/{getxapi-tweet-detail,getxapi-article,post-tweet} scripts
- Fix auth bug: remove redundant authorization header from fetch() calls;
  meta headers are injected automatically by the engine
- Add field-level godoc to Meta struct in script/parser.go
- Add sites/CLAUDE.md explaining script structure and key rules
@vaayne vaayne merged commit fd06191 into main May 12, 2026
4 of 5 checks passed
@vaayne vaayne deleted the feat/tap-sites branch May 12, 2026 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant