diff --git a/.changeset/README.md b/.changeset/README.md new file mode 100644 index 0000000..00dd9e4 --- /dev/null +++ b/.changeset/README.md @@ -0,0 +1,45 @@ +# Changesets + +Tokenometer uses [Changesets](https://github.com/changesets/changesets) to manage versions and changelogs. + +## How to add one + +When opening a PR that warrants a release entry, run: + +```bash +npx changeset +``` + +You'll be prompted to: +1. Pick which packages changed (`tokenometer` and `@tokenometer/core` are version-fixed — bumping one bumps both). +2. Pick the bump type: + - `patch` — bug fix, internal refactor, doc fix + - `minor` — new flag, new provider, new format, new output type + - `major` — breaking CLI flag rename, breaking API change, dropped provider +3. Write a one-line summary (this becomes the CHANGELOG entry). + +The command writes a `.md` file in this directory. Commit it in the same PR. + +## What happens on merge to main + +The `release` GitHub Actions workflow runs automatically: + +1. If any changeset files are present in `.changeset/`, the workflow opens (or updates) a **Version Packages** PR. + - That PR bumps `package.json` versions for `tokenometer` and `@tokenometer/core` (fixed group). + - It auto-generates `CHANGELOG.md` entries from the changeset summaries. + - It deletes the consumed `.changeset/.md` files. +2. When the Version Packages PR is merged, the workflow publishes the bumped versions to npm with provenance. +3. A GitHub Release is created on the new tag — this is what the GitHub Marketplace listens to for the Action. + +## Ignored packages + +`@tokenometer/web` and `@tokenometer/action` are listed under `ignore` in `config.json`. They aren't published to npm directly: + +- `@tokenometer/web` is the playground — deployed to Vercel, not versioned on npm. +- `@tokenometer/action` ships as a JavaScript Action via the repo tag. Its release is driven by the GitHub Release event, not npm. + +If you change either, you don't need a changeset. If your change *also* touches `@tokenometer/core` or the CLI, do add a changeset for those. + +## Why fixed-group? + +`tokenometer` (the CLI) depends on `@tokenometer/core` directly. Letting them drift confused users who'd `npm install tokenometer@latest @tokenometer/core@latest` and end up on incompatible majors. Fixing the group means one version number tracks both. diff --git a/.changeset/config.json b/.changeset/config.json new file mode 100644 index 0000000..74a05c6 --- /dev/null +++ b/.changeset/config.json @@ -0,0 +1,11 @@ +{ + "$schema": "https://unpkg.com/@changesets/config@3.0.0/schema.json", + "changelog": ["@changesets/changelog-github", { "repo": "faraa2m/tokenometer" }], + "commit": false, + "fixed": [["tokenometer", "@tokenometer/core"]], + "linked": [], + "access": "public", + "baseBranch": "main", + "updateInternalDependencies": "patch", + "ignore": ["@tokenometer/web", "@tokenometer/action"] +} diff --git a/.changeset/legit-docs-and-seo.md b/.changeset/legit-docs-and-seo.md new file mode 100644 index 0000000..4892a46 --- /dev/null +++ b/.changeset/legit-docs-and-seo.md @@ -0,0 +1,13 @@ +--- +"tokenometer": minor +"@tokenometer/core": minor +--- + +Project legitimacy and SEO rollout (Wave 1 of v1.0.0): + +- Added `CODE_OF_CONDUCT.md` (Contributor Covenant 2.1), `CONTRIBUTING.md`, `SECURITY.md` (GitHub Private Vulnerability Reporting only — no email exposure), `CHANGELOG.md`. +- Added `.github/copilot-instructions.md` for `gh copilot review`, plus `PULL_REQUEST_TEMPLATE.md`, issue forms (`bug_report.yml`, `feature_request.yml`, `config.yml`), and `FUNDING.yml`. +- Rewrote root `README.md` with badges row, "Why Tokenometer vs alternatives" comparison table (vs `tokencost`, `tiktoken`, `gpt-tokenizer`, `promptfoo`, `gpt-token-counter-live`), ASCII demo of CLI table output, and project-health checklist. +- Extended npm `keywords` and `description` across `tokenometer` (CLI), `@tokenometer/core`, and `@tokenometer/action` for SEO. Root description now reads: "Tokenometer — LLM cost calculator, token counter, and CI cost-guardrail Action for Claude, GPT-4o, Gemini." +- Added Marketplace publish prep (`.github/release-notes-v1.0.0.md`) and awesome-list PR templates (`.github/awesome-list-prs/`). +- Initialized Changesets for auto-bump + auto-CHANGELOG-generation on release. Replaced the manual `release.yml` workflow with the Changesets-driven version-PR + publish flow. diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml new file mode 100644 index 0000000..90d1678 --- /dev/null +++ b/.github/FUNDING.yml @@ -0,0 +1,13 @@ +github: [faraa2m] +# patreon: +# open_collective: +# ko_fi: +# tidelift: +# community_bridge: +# liberapay: +# issuehunt: +# lfx_crowdfunding: +# polar: +# buy_me_a_coffee: +# thanks_dev: +# custom: diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 0000000..c8b5169 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,79 @@ +name: "Bug report" +description: Report a measurement, pricing, or tooling bug in Tokenometer. +title: "bug: " +labels: ["bug"] +body: + - type: markdown + attributes: + value: | + Thanks for filing a bug. Please include enough detail for someone to reproduce the issue without follow-up questions. + + - type: input + id: tokenometer-version + attributes: + label: Tokenometer version + description: Output of `tokenometer --version`. + placeholder: "e.g. 1.0.3" + validations: + required: true + + - type: input + id: node-version + attributes: + label: Node version + description: Output of `node --version`. + placeholder: "e.g. v20.11.1" + validations: + required: true + + - type: input + id: os + attributes: + label: Operating system + placeholder: "e.g. macOS 14.5, Ubuntu 22.04, Windows 11" + validations: + required: true + + - type: input + id: model-flags + attributes: + label: Model and format flags + description: The exact `--model`, `--format`, and other flags you invoked. + placeholder: "e.g. --model claude-sonnet-4 --format json" + validations: + required: true + + - type: textarea + id: steps + attributes: + label: Steps to reproduce + description: Minimal commands or input to reproduce the issue. + placeholder: | + 1. echo "hello" | tokenometer --model gpt-4o + 2. ... + validations: + required: true + + - type: textarea + id: expected + attributes: + label: Expected behavior + validations: + required: true + + - type: textarea + id: actual + attributes: + label: Actual behavior + validations: + required: true + + - type: textarea + id: logs + attributes: + label: Logs or output + description: | + Paste relevant output. **Redact any API keys, prompts containing private data, or other secrets before submitting** — issues are public. + render: shell + validations: + required: false diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..e045583 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,8 @@ +blank_issues_enabled: false +contact_links: + - name: Questions and discussion + url: https://github.com/faraa2m/tokenometer/discussions + about: For usage questions, ideas, and general discussion — not bug reports. + - name: Security vulnerability + url: https://github.com/faraa2m/tokenometer/blob/main/SECURITY.md + about: Do not open a public issue for security vulnerabilities. Follow SECURITY.md for private disclosure. diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 0000000..913f7f2 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,42 @@ +name: "Feature request" +description: Propose a new feature or change to Tokenometer. +title: "feat: " +labels: ["enhancement"] +body: + - type: markdown + attributes: + value: | + Acceptance bar: a feature must demonstrate measurable cost-decision value to users, or fix a measurement bug. "Nice to have" requests without a concrete use case will be closed. + + - type: textarea + id: problem + attributes: + label: Problem you're trying to solve + description: What can't you do today, or what's painful about the current behavior? + validations: + required: true + + - type: textarea + id: solution + attributes: + label: Proposed solution + description: Concrete behavior, flags, or output changes you'd like to see. + validations: + required: true + + - type: textarea + id: alternatives + attributes: + label: Alternatives considered + description: Other approaches you weighed and why you ruled them out. + validations: + required: false + + - type: textarea + id: impact + attributes: + label: Impact on existing flags or output + description: | + Does this change CLI flags, JSON output shape, Action inputs/outputs, or pricing math? Call out any breaking changes explicitly. + validations: + required: true diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..432b5b8 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,36 @@ +## Summary + + + +## Why + + + +## Changes + + + +- +- +- + +## Testing + + + +``` +npm run lint +npm run typecheck +npm test +``` + +## Checklist + +- [ ] Tests added or updated for the change +- [ ] `npm run lint` passes +- [ ] `npm run typecheck` passes +- [ ] README updated if behavior is user-facing +- [ ] CHANGELOG entry added via Changesets (`npx changeset`) +- [ ] Breaking change documented in the changeset (if any) +- [ ] Action bundle rebuilt (`npm run build` in `packages/action/`) if `packages/action` was touched +- [ ] No secrets, API keys, or `.env` files committed diff --git a/.github/awesome-list-prs/awesome-actions.md b/.github/awesome-list-prs/awesome-actions.md new file mode 100644 index 0000000..f4855be --- /dev/null +++ b/.github/awesome-list-prs/awesome-actions.md @@ -0,0 +1,26 @@ +# PR for awesome-actions + +**Target repo:** https://github.com/sdras/awesome-actions +**PR title:** "Add Tokenometer — LLM cost CLI + GitHub Action" +**PR body:** + +I'd like to add Tokenometer to the **Utility / Pull Requests** (or **Code Quality**) section. + +## What it is + +**Tokenometer** — empirical LLM token-cost CLI + GitHub Action + VS Code / Cursor extension + Claude Code skill. Multi-provider (Claude, GPT-4o, Gemini, Mistral, Cohere), offline + empirical modes (real `countTokens` API calls), CI cost-guardrail with sticky PR comments and budget gates. + +- Repo: https://github.com/faraa2m/tokenometer +- npm: `npx tokenometer ./prompt.md --model claude-opus-4-7` +- Marketplace: https://github.com/marketplace/actions/tokenometer +- Live demo: https://tokenometer.vercel.app + +## Why it fits this list + +It is a first-class GitHub Action that posts a sticky PR comment diffing prompt-token cost across models on every push, and can fail the build when a USD budget is exceeded — the same shape as test-coverage or bundle-size guardrails, applied to LLM cost. + +## Markdown line for the list + +``` +- [Tokenometer](https://github.com/faraa2m/tokenometer) — LLM token cost CLI + GitHub Action. Multi-provider, empirical, CI-native. +``` diff --git a/.github/awesome-list-prs/awesome-ai-tools.md b/.github/awesome-list-prs/awesome-ai-tools.md new file mode 100644 index 0000000..a5ed81f --- /dev/null +++ b/.github/awesome-list-prs/awesome-ai-tools.md @@ -0,0 +1,26 @@ +# PR for awesome-ai-tools + +**Target repo:** https://github.com/mahseema/awesome-ai-tools (verify — there are a few "awesome-ai-tools" repos; pick the one whose README structure matches). +**PR title:** "Add Tokenometer — LLM cost CLI + GitHub Action" +**PR body:** + +I'd like to add Tokenometer to the **Developer Tools** (or **Cost / Observability**) section. + +## What it is + +**Tokenometer** — empirical LLM token-cost CLI + GitHub Action + VS Code / Cursor extension + Claude Code skill. Multi-provider (Claude, GPT-4o, Gemini, Mistral, Cohere), offline + empirical modes (real `countTokens` API calls), CI cost-guardrail with sticky PR comments and budget gates. + +- Repo: https://github.com/faraa2m/tokenometer +- npm: `npx tokenometer ./prompt.md --model claude-opus-4-7` +- Marketplace: https://github.com/marketplace/actions/tokenometer +- Live demo: https://tokenometer.vercel.app + +## Why it fits this list + +The directory already includes generation, agent, and eval tooling but very little on the cost side. Tokenometer fills that gap with a single-command, multi-provider USD estimate that works locally and in CI, so teams can keep model spend in view without rolling their own dashboard. + +## Markdown line for the list + +``` +- [Tokenometer](https://github.com/faraa2m/tokenometer) — LLM token cost CLI + GitHub Action. Multi-provider, empirical, CI-native. +``` diff --git a/.github/awesome-list-prs/awesome-llm-apps.md b/.github/awesome-list-prs/awesome-llm-apps.md new file mode 100644 index 0000000..cdb1eea --- /dev/null +++ b/.github/awesome-list-prs/awesome-llm-apps.md @@ -0,0 +1,26 @@ +# PR for awesome-llm-apps + +**Target repo:** https://github.com/Shubhamsaboo/awesome-llm-apps (verify before opening — there are several lists with this name; pick the one whose `README.md` matches the section you intend to land in). +**PR title:** "Add Tokenometer — LLM cost CLI + GitHub Action" +**PR body:** + +I'd like to add Tokenometer to the **Tools / Developer Tools** section (or whichever section the maintainers prefer for cost-tracking utilities). + +## What it is + +**Tokenometer** — empirical LLM token-cost CLI + GitHub Action + VS Code / Cursor extension + Claude Code skill. Multi-provider (Claude, GPT-4o, Gemini, Mistral, Cohere), offline + empirical modes (real `countTokens` API calls), CI cost-guardrail with sticky PR comments and budget gates. + +- Repo: https://github.com/faraa2m/tokenometer +- npm: `npx tokenometer ./prompt.md --model claude-opus-4-7` +- Marketplace: https://github.com/marketplace/actions/tokenometer +- Live demo: https://tokenometer.vercel.app + +## Why it fits this list + +Awesome-llm-apps already curates apps and tooling that help builders ship with LLMs. Tokenometer is the missing "what does this prompt actually cost" piece — it lets app developers measure, diff, and budget token spend across providers from a single CLI as they iterate. + +## Markdown line for the list + +``` +- [Tokenometer](https://github.com/faraa2m/tokenometer) — LLM token cost CLI + GitHub Action. Multi-provider, empirical, CI-native. +``` diff --git a/.github/awesome-list-prs/awesome-llmops.md b/.github/awesome-list-prs/awesome-llmops.md new file mode 100644 index 0000000..daf740e --- /dev/null +++ b/.github/awesome-list-prs/awesome-llmops.md @@ -0,0 +1,26 @@ +# PR for awesome-llmops + +**Target repo:** https://github.com/tensorchord/Awesome-LLMOps +**PR title:** "Add Tokenometer — LLM cost CLI + GitHub Action" +**PR body:** + +I'd like to add Tokenometer to the **Observability / Cost** (or **Developer Tools**) section. + +## What it is + +**Tokenometer** — empirical LLM token-cost CLI + GitHub Action + VS Code / Cursor extension + Claude Code skill. Multi-provider (Claude, GPT-4o, Gemini, Mistral, Cohere), offline + empirical modes (real `countTokens` API calls), CI cost-guardrail with sticky PR comments and budget gates. + +- Repo: https://github.com/faraa2m/tokenometer +- npm: `npx tokenometer ./prompt.md --model claude-opus-4-7` +- Marketplace: https://github.com/marketplace/actions/tokenometer +- Live demo: https://tokenometer.vercel.app + +## Why it fits this list + +Cost is a first-class LLMOps concern but most platforms treat it as a post-hoc dashboard. Tokenometer pulls it left into the developer workflow — a CLI for local prompt iteration plus a GitHub Action that gates PRs on a USD budget — so prompt-cost regressions get caught in code review the same way performance regressions do. + +## Markdown line for the list + +``` +- [Tokenometer](https://github.com/faraa2m/tokenometer) — LLM token cost CLI + GitHub Action. Multi-provider, empirical, CI-native. +``` diff --git a/.github/awesome-list-prs/awesome-prompt-engineering.md b/.github/awesome-list-prs/awesome-prompt-engineering.md new file mode 100644 index 0000000..3d7f5fa --- /dev/null +++ b/.github/awesome-list-prs/awesome-prompt-engineering.md @@ -0,0 +1,26 @@ +# PR for awesome-prompt-engineering + +**Target repo:** https://github.com/promptslab/Awesome-Prompt-Engineering +**PR title:** "Add Tokenometer — LLM cost CLI + GitHub Action" +**PR body:** + +I'd like to add Tokenometer to the **Tools & Code** section. + +## What it is + +**Tokenometer** — empirical LLM token-cost CLI + GitHub Action + VS Code / Cursor extension + Claude Code skill. Multi-provider (Claude, GPT-4o, Gemini, Mistral, Cohere), offline + empirical modes (real `countTokens` API calls), CI cost-guardrail with sticky PR comments and budget gates. + +- Repo: https://github.com/faraa2m/tokenometer +- npm: `npx tokenometer ./prompt.md --model claude-opus-4-7` +- Marketplace: https://github.com/marketplace/actions/tokenometer +- Live demo: https://tokenometer.vercel.app + +## Why it fits this list + +Prompt engineers iterate on wording, structure, and few-shot examples — all of which move token count and cost. Tokenometer makes that feedback loop quantitative: you get a real `countTokens` measurement and a per-model USD figure for each prompt revision, so cost-aware prompt iteration stops being guesswork. + +## Markdown line for the list + +``` +- [Tokenometer](https://github.com/faraa2m/tokenometer) — LLM token cost CLI + GitHub Action. Multi-provider, empirical, CI-native. +``` diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000..0d73232 --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,61 @@ +# Copilot Review Instructions — Tokenometer + +This is a TypeScript-strict monorepo using npm workspaces. Honor strict-mode discipline: no implicit `any`, no unchecked indexed access, no silent type widening. + +## Layout + +Packages: +- `@tokenometer/core` lives in `packages/core` — pure measurement, pricing, and tokenizer logic. No I/O. +- `tokenometer` CLI lives in `packages/cli` — thin wrapper that parses argv, calls core, formats output. +- The GitHub Action lives in `packages/action` — bundled to `dist/index.cjs` via ncc. +- The web playground lives in `apps/` (or `packages/web` depending on milestone). + +Cross-package imports go through public entrypoints only. Don't reach into `src/internal/*` from another package. + +## Toolchain + +- Lint with Biome: `npm run lint`. Don't introduce ESLint/Prettier — Biome owns formatting and linting. +- Type-check with `npm run typecheck`. Errors block merge. +- Test with `npm test` (Vitest). Co-locate tests as `*.test.ts` next to source. +- Node ≥20 only. Don't add polyfills for older runtimes. + +## Comments and naming + +DO NOT add comments unless the WHY is non-obvious. No JSDoc on internal functions. Self-documenting names preferred. If a reviewer would ask "why," leave a comment; if they'd just nod, delete it. + +Flag PRs that add narration-style comments (`// loop over items`, `// returns the total`). Those should come out. + +## Approximate-token honesty (load-bearing) + +When the offline tokenizer is a proxy (Anthropic falls back to `cl100k_base`, Google uses `chars/4`), the result MUST set `approximate: true` on the API surface. This honesty is load-bearing — never silently fall back without the flag. Reviewers should reject any code path that produces a number without propagating the approximate bit. + +If a PR adds a new provider tokenizer, verify: +1. The `approximate` flag flows through to `MeasureResult`. +2. The CLI/Action surfaces it (e.g. `~` prefix or explicit "approximate" label). +3. There's a test covering the proxy path. + +## Pricing + +Pricing source priority: +1. `LOCAL_OVERRIDES` in `packages/core/src/rates.ts` — bleeding-edge rates not yet upstreamed. +2. `tokenlens` registry — the source of truth for stable models. + +Don't hardcode dollar amounts anywhere else in the codebase. If a PR introduces a literal price outside `rates.ts`, request a move. + +## Commit and PR style + +Conventional Commits: `feat:`, `fix:`, `chore:`, `docs:`, `refactor:`, `ci:`, `test:`. PR titles match the commit style. Include scope where helpful, e.g. `feat(cli): add --json flag` or `fix(core): correct Gemini chars-per-token ratio`. + +Reject PR titles like "Update files" or "Misc fixes" — request a Conventional Commit rewrite. + +## Action bundle + +Action's `dist/index.cjs` must stay in sync with source — CI verifies this. Run `npm run build` in `packages/action/` before committing changes there. If a PR touches `packages/action/src/**` without updating `dist/`, flag it. + +## Empirical mode (do not enable) + +Empirical mode in the GitHub Action is intentionally OFF. No API keys live in repo secrets. Don't suggest enabling empirical paths in the workflow, the README, or examples. Offline measurement is the contract for the Action surface. + +## Review tone + +Terse, specific, point-at-line-numbers. No congratulatory language. No "great work!" or "looks good overall." If there's nothing to flag, say nothing. Otherwise, link the line, name the issue, propose the fix. diff --git a/.github/release-notes-v1.0.0.md b/.github/release-notes-v1.0.0.md new file mode 100644 index 0000000..9d1d138 --- /dev/null +++ b/.github/release-notes-v1.0.0.md @@ -0,0 +1,75 @@ +# Tokenometer v1.0.0 — Production-ready LLM cost calculator and CI guardrail. + +> Note: this is a **DRAFT**. Some items below are marked `[planned]` because the +> phase has not yet landed on `main`. Edit before publishing the GitHub Release. + +## Highlights + +- **CLI + GitHub Action shipped** — `npx tokenometer` and the marketplace action + share a single core, so local results match CI results. +- **Multi-provider** — Claude (Opus / Sonnet / Haiku), OpenAI GPT-4o family, and + Google Gemini all run through the same interface. +- **Empirical mode** — real `countTokens` API calls (free for Anthropic and + Google) instead of ad-hoc heuristics; falls back to `approximate` only when + the user opts out or no key is configured. +- **CI cost-guardrail** — sticky PR comment with a per-model diff, plus a + `budget` input that fails the run when the head delta exceeds your USD cap. +- **Auto-updating prices** — pricing and context limits are sourced from the + `tokenlens` registry, not a hand-maintained table. +- **Honest `approximate` flag** — every output row is tagged so you always know + whether a number came from a real tokenizer or an estimate. +- **Vision-token cost** `[planned — Phase D]` — image-aware token accounting + for multi-modal prompts. +- **Per-file attribution** `[planned — Phase C.4]` — costs broken down by source + file in PR comments. +- **SARIF output** `[planned — Phase C.5]` — lets cost regressions surface in + GitHub code scanning. +- **Auto provider detection** `[planned — Phase C.1]` — infer the provider from + the model id without an extra flag. +- **`.tokenometer.yml` config** `[planned — Phase C.2]` — repo-level defaults + for paths, models, and budgets. +- **VS Code / Cursor extension** `[planned — Phase E.1]` — inline cost lens for + prompt files in the editor. +- **Claude Code skill** `[planned — Phase E.2]` — first-class skill so Claude + Code can call Tokenometer when iterating on prompts. +- **Mistral + Cohere providers** `[planned — Phase H]` — additional model + families behind the same CLI surface. +- **Latency mode** `[planned]` — `--latency` flag to surface per-model latency + alongside cost. +- **Unified release pipeline** — one workflow publishes both `tokenometer` and + `@tokenometer/core` so the CLI and library never drift. + +## Empirical findings + +The `n=150` cross-shape comparison that motivated this project is in the +[Findings section of the README](https://github.com/faraa2m/tokenometer#findings-anthropic-n150-cells-across-10-prompt-shapes). + +## Install + +CLI: + +``` +npx tokenometer ./prompt.md --model claude-opus-4-7 +``` + +GitHub Action: + +```yaml +- uses: faraa2m/tokenometer@v1 + with: + paths: | + prompts/**/*.md + models: claude-opus-4-7,claude-sonnet-4-6,gpt-4o + budget: '0.50' +``` + +## What's next + +See the project roadmap milestones on GitHub for the planned post-1.0 phases: +. + +## Acknowledgments + +Pricing and context-window data are sourced from the +[`tokenlens`](https://github.com/m31coding/tokenlens) registry — thanks to the +upstream maintainers for keeping that table accurate. diff --git a/.github/repo-metadata.md b/.github/repo-metadata.md new file mode 100644 index 0000000..f8e9bdd --- /dev/null +++ b/.github/repo-metadata.md @@ -0,0 +1,11 @@ +# Repo metadata to apply + +Apply via: +``` +gh repo edit faraa2m/tokenometer --description "Tokenometer — LLM cost calculator, token counter, and CI cost-guardrail Action for Claude, GPT-4o, Gemini." +gh repo edit faraa2m/tokenometer --add-topic llm --add-topic cost-tracking --add-topic prompt-engineering --add-topic tokenizer --add-topic claude --add-topic gpt-4 --add-topic gemini --add-topic github-actions --add-topic cli --add-topic developer-tools --add-topic ai --add-topic cost-optimization --add-topic anthropic --add-topic openai +``` + +## Marketplace categories to choose during release publish +- "Code quality" +- "Continuous integration" diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 6f45acf..6369be8 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,57 +1,82 @@ name: release on: + push: + branches: [main] workflow_dispatch: inputs: - version: - description: 'Version to bump to (e.g. 0.0.2). Leave empty to keep current package.json version.' + dry-run: + description: 'Dry run (skip npm publish, skip GitHub Release).' required: false - default: '' + default: 'false' permissions: - contents: read + contents: write + pull-requests: write id-token: write +concurrency: + group: release + cancel-in-progress: false + jobs: - publish: + release: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 + with: + fetch-depth: 0 + - uses: actions/setup-node@v4 with: node-version: '20' registry-url: 'https://registry.npmjs.org/' + - run: npm ci - - run: npm run build + - run: npm run lint + - run: npm run typecheck - run: npm test - - run: npm run benchmarks - - - name: Bump versions (if requested) - if: ${{ inputs.version != '' }} - run: | - npm version ${{ inputs.version }} --no-git-tag-version --workspaces - - - name: Sync cross-package deps to bumped version - if: ${{ inputs.version != '' }} - run: | - npm pkg set 'dependencies.@tokenometer/core'="${{ inputs.version }}" --workspace=packages/cli - npm pkg set 'dependencies.@tokenometer/core'="${{ inputs.version }}" --workspace=packages/action + - run: npm run build - - name: Publish @tokenometer/core - working-directory: packages/core - run: npm publish --provenance --access public + # Changesets opens (or updates) a "Version Packages" PR when changeset files + # are present in `.changeset/`. When that PR is merged, this same workflow + # auto-bumps versions in package.json, auto-generates CHANGELOG.md entries + # from the changeset summaries, then publishes to npm. + - name: Changesets — version PR or publish + if: ${{ inputs.dry-run != 'true' }} + id: changesets + uses: changesets/action@v1 + with: + version: npx changeset version + publish: npx changeset publish + commit: 'chore(release): version packages' + title: 'chore(release): version packages' + createGithubReleases: true env: - NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + NPM_TOKEN: ${{ secrets.NPM_TOKEN }} - - name: Publish tokenometer (CLI) - working-directory: packages/cli - run: npm publish --provenance --access public - env: - NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} + - name: Sync cross-package deps after publish + if: ${{ steps.changesets.outputs.published == 'true' }} + run: | + PUBLISHED_VERSION=$(node -p "require('./packages/core/package.json').version") + npm pkg set 'dependencies.@tokenometer/core'="$PUBLISHED_VERSION" --workspace=packages/cli + npm pkg set 'dependencies.@tokenometer/core'="$PUBLISHED_VERSION" --workspace=packages/action + # Phase I (Wave 4) will add: Marketplace verify, VS Code Marketplace publish, + # Open VSX publish, Vercel deploy hook, smoke test against published versions. - name: Summary + if: ${{ steps.changesets.outputs.published == 'true' }} run: | echo "### Published" >> $GITHUB_STEP_SUMMARY echo "- https://www.npmjs.com/package/@tokenometer/core" >> $GITHUB_STEP_SUMMARY echo "- https://www.npmjs.com/package/tokenometer" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + echo "GitHub Release on the new tag triggers the Marketplace re-publish for the Action." >> $GITHUB_STEP_SUMMARY + + - name: Dry run notice + if: ${{ inputs.dry-run == 'true' }} + run: | + echo "### Dry run" >> $GITHUB_STEP_SUMMARY + echo "Skipped Changesets publish step." >> $GITHUB_STEP_SUMMARY diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..b7a4a00 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,37 @@ +# Changelog + +Format follows [Keep a Changelog](https://keepachangelog.com/). This project adheres to [Semantic Versioning](https://semver.org/). + +Going forward, entries are managed by [Changesets](https://github.com/changesets/changesets) and emitted as part of the unified release workflow. + +## [Unreleased] + +### Added +- Project legitimacy docs: `CODE_OF_CONDUCT.md`, `CONTRIBUTING.md`, `SECURITY.md` (using GitHub Private Vulnerability Reporting), `CHANGELOG.md`. +- GitHub repo templates: `.github/copilot-instructions.md` (consumed by `gh copilot review`), `PULL_REQUEST_TEMPLATE.md`, issue forms (`bug_report.yml`, `feature_request.yml`, `config.yml`), `FUNDING.yml`. +- README rewrite: badges row, "Why Tokenometer vs alternatives" comparison table, ASCII demo of CLI table output, project-health checklist links. +- npm SEO: extended `keywords` across `tokenometer` (CLI), `@tokenometer/core`, and `@tokenometer/action`. Updated `description` on root + every package. +- Marketplace publish prep: draft v1.0.0 release notes, awesome-list PR templates (`awesome-llm-apps`, `awesome-actions`, `awesome-prompt-engineering`, `awesome-ai-tools`, `awesome-llmops`). + +## [0.0.2] — 2026-05-08 + +### Fixed +- `fix(release): build before bumping versions to avoid stale nested install` (#9) +- `fix(cli): run main() when invoked via npx symlink` (#7) + +### Changed +- `refactor(core): source pricing + context limits from tokenlens registry` (#8) — pricing is now sourced from the [`tokenlens`](https://www.npmjs.com/package/tokenlens) registry plus a small set of `LOCAL_OVERRIDES` for bleeding-edge models the registry hasn't picked up yet (see `packages/core/src/rates.ts`). + +### Added +- `ci(release): manual-trigger workflow that publishes both packages` (#6) +- `docs(readme): align usage with shipped CLI + Action` (#10) + +## [0.0.1] — initial publish + +### Added +- `tokenometer` CLI: multi-provider (Anthropic, OpenAI, Google), multi-format (JSON, YAML, XML, Markdown, text), offline + empirical modes (`--empirical`, `--max-spend`). +- `@tokenometer/core` library: tokenizer dispatch, USD cost calculation, honest `approximate` flag when the offline path is a proxy (Claude `cl100k_base`, Google `chars/4`). +- GitHub Action (`packages/action`): sticky PR comment with prompt-cost diff vs base branch, optional `budget` input that fails the check when exceeded. +- Web playground at https://tokenometer.vercel.app — paste-and-compare UI, observatory aesthetic. +- Empirical benchmark sweep: `npm run benchmarks:empirical` produces the n=150 dataset that powers the README findings (Opus is +62% denser than `cl100k_base`). +- Vercel monorepo deploy config. diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..4cb0c2a --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,25 @@ +# Code of Conduct + +This project adopts the [Contributor Covenant, version 2.1](https://www.contributor-covenant.org/version/2/1/code_of_conduct/) in full. + +## Scope + +The Code of Conduct applies in all project spaces — issues, pull requests, Discussions, code review, and any public communication tied to the project. + +## Expectations + +- Assume good intent; clarify before escalating. +- Disagree about technical decisions; do not make it personal. +- Keep discussion on the issue, the data, or the code. Cite measurements where possible — that's the spirit of this project. + +## Reporting + +Reports about Code of Conduct concerns can be submitted privately via the **Report a vulnerability** button on this repo's Security tab (the same channel we use for security disclosures). We do not list a personal email to avoid scraping. Reports remain private; only the maintainers with write access read them. + +## Enforcement + +Maintainers will follow the Contributor Covenant 2.1 Enforcement Guidelines. Outcomes may include warnings, temporary cooldowns, or permanent removal from project spaces, scaled to the situation. + +## Attribution + +Adapted from the Contributor Covenant 2.1, available at https://www.contributor-covenant.org/version/2/1/code_of_conduct/. Translations available at https://www.contributor-covenant.org/translations. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..47e88ab --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,105 @@ +# Contributing to Tokenometer + +Thanks for your interest. Tokenometer is a small, opinionated project — clear measurements, honest defaults, no magic. Contributions that move in that direction are welcome. + +## Quick start + +```bash +git clone https://github.com/faraa2m/tokenometer.git +cd tokenometer +npm install +npm test +``` + +Node ≥20 required. + +## Development workflow + +| Task | Command | +|---|---| +| Lint | `npm run lint` | +| Format | `npm run format` | +| Type-check | `npm run typecheck` | +| Test | `npm test` | +| Test (watch) | `npm run test:watch` | +| Build all packages | `npm run build` | +| Run benchmarks (offline) | `npm run benchmarks` | +| Run benchmarks (empirical, needs `ANTHROPIC_API_KEY`) | `npm run benchmarks:empirical` | + +Before opening a PR: `npm run lint && npm run typecheck && npm test && npm run build`. CI runs the same steps. + +## Code style + +- TypeScript strict. +- Biome handles lint + format. Don't fight it. +- Self-documenting names preferred. **No comments unless the WHY is non-obvious** (a hidden constraint, a workaround, a subtle invariant). Comments that restate the code are noise. +- See `.github/copilot-instructions.md` for the full conventions Copilot CLI uses to review PRs — same rules apply to humans. + +## Honesty contract + +When the offline tokenizer is a proxy (Anthropic `cl100k_base`, Google `chars/4`), the API result must set `approximate: true`. This flag is load-bearing. Don't silently fall back without it. If you add a new provider with an inexact tokenizer, set the flag. + +## Commit messages + +Conventional Commits. Examples from `git log`: + +- `refactor(core): source pricing + context limits from tokenlens registry` +- `fix(cli): run main() when invoked via npx symlink` +- `feat(action): GitHub Action — sticky PR comment with prompt-cost diff` + +Allowed types: `feat`, `fix`, `chore`, `docs`, `refactor`, `ci`, `test`, `style`, `perf`, `build`. Scope is the package or area (`cli`, `core`, `action`, `web`, `release`, `readme`, etc.). + +PR titles match the same format. + +## Adding a new model + +1. If `tokenlens` already lists the model, it's auto-picked up — usually nothing to do. +2. If the model is too new for `tokenlens`, add an entry to `LOCAL_OVERRIDES` in `packages/core/src/rates.ts` with `pricingSource: 'local'`. The override is removed once `scripts/check-overrides.mjs` reports the registry caught up. +3. Add a Vitest case in `packages/core/test/` covering the new id. +4. Update README's known-models table if user-facing. + +## Adding a new provider + +1. Add tokenizer dispatch in `packages/core/src/tokenize.ts`. +2. Add a provider catalog entry in `packages/core/src/rates.ts` (`CATALOG`). If `@tokenlens/models/` exists, prefer it over hand-rolled pricing. +3. Decide: exact or approximate offline path. Set `approximate: true` if the path is a proxy. **Document the choice in the README methodology table.** +4. Add Vitest fixtures across all formats (json, yaml, xml, markdown, text). +5. If the provider exposes an empirical `countTokens` endpoint, wire it up. + +## Filing bugs + +Use the bug report template. Include: +- `tokenometer --version` +- Node version +- OS +- Exact flags used +- Repro steps +- Expected vs actual + +## Reporting security issues + +See [`SECURITY.md`](SECURITY.md). Use GitHub Private Vulnerability Reporting; do not file public issues for vulnerabilities. + +## Changesets + +Releases are driven by [Changesets](https://github.com/changesets/changesets). For any PR that should produce a release entry: + +```bash +npx changeset +``` + +Pick the affected packages, the bump type (`patch` / `minor` / `major`), and write a one-line summary. A `.changeset/.md` file is written; commit it. See [`.changeset/README.md`](.changeset/README.md) for the full rules (notably: `tokenometer` and `@tokenometer/core` are version-fixed — bumping one bumps both). + +## Releasing + +Pushes to `main` trigger `.github/workflows/release.yml`: + +1. If `.changeset/` has any pending changesets, the workflow opens (or updates) a **Version Packages** PR that auto-bumps `package.json` versions and auto-generates `CHANGELOG.md` entries. +2. Merging that PR triggers the same workflow to publish the bumped versions to npm with provenance. +3. A GitHub Release is created on the new tag — the GitHub Marketplace listens to that for the Action's update. + +Phase I (Wave 4) extends this to also publish the VS Code extension to the Marketplace + Open VSX, run a post-publish smoke test, and trigger a Vercel deploy. + +## Questions + +Use [Discussions](https://github.com/faraa2m/tokenometer/discussions). Issues are for bugs and concrete proposals. diff --git a/README.md b/README.md index cbe7f6a..85cfeac 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,32 @@ # Tokenometer -> Empirical token-cost benchmarking for LLM prompts. **Live: https://tokenometer.vercel.app** - -Tokenometer answers a simple, expensive question: **does it actually cost less to send your prompt as YAML, JSON, XML, or Markdown — across Claude, GPT-4o, and Gemini?** - -It started as a [\$23 question](https://hackernoon.com/i-spent-$23-testing-the-yaml-saves-tokens-hack-it-doesnt-work). This is the tool anyone can run. +[![npm tokenometer](https://img.shields.io/npm/v/tokenometer.svg?label=tokenometer)](https://www.npmjs.com/package/tokenometer) +[![npm @tokenometer/core](https://img.shields.io/npm/v/@tokenometer/core.svg?label=@tokenometer/core)](https://www.npmjs.com/package/@tokenometer/core) +[![CI](https://github.com/faraa2m/tokenometer/actions/workflows/ci.yml/badge.svg)](https://github.com/faraa2m/tokenometer/actions/workflows/ci.yml) +[![License: MIT](https://img.shields.io/github/license/faraa2m/tokenometer.svg)](LICENSE) +[![GitHub stars](https://img.shields.io/github/stars/faraa2m/tokenometer.svg?style=social)](https://github.com/faraa2m/tokenometer/stargazers) + + +> Tokenometer — LLM cost calculator, token counter, and CI cost-guardrail Action for Claude, GPT-4o, Gemini. +> **Live: https://tokenometer.vercel.app** + +Tokenometer answers a simple, expensive question: **does it actually cost less to send your prompt as YAML, JSON, XML, or Markdown — across Claude, GPT-4o, and Gemini?** It started as a [\$23 question](https://hackernoon.com/i-spent-$23-testing-the-yaml-saves-tokens-hack-it-doesnt-work). This is the tool anyone can run — offline, empirically, or as a PR guardrail. + +## Why Tokenometer vs alternatives + +| | Tokenometer | [tokencost](https://github.com/AgentOps-AI/tokencost) (AgentOps) | [tiktoken](https://github.com/openai/tiktoken) (OpenAI) | [gpt-tokenizer](https://github.com/niieani/gpt-tokenizer) | [promptfoo](https://github.com/promptfoo/promptfoo) | gpt-token-counter-live (VS Code) | +|---------------------------------------|:-----------:|:--------:|:--------:|:--------:|:--------:|:--------:| +| Multi-provider (Anthropic / OpenAI / Google) | ✓ | ✓ | – | – | ✓ | – | +| Multi-format compare (JSON / YAML / XML / MD / text) | ✓ | – | – | – | – | – | +| Empirical mode (real provider `countTokens`) | ✓ | – | – | – | partial | – | +| CLI | ✓ | ✓ | – | – | ✓ | – | +| GitHub Action (PR cost-diff guardrail) | ✓ | – | – | – | partial | – | +| VS Code / Cursor extension | – (planned) | – | – | – | – | ✓ | +| Cost (USD), not just tokens | ✓ | ✓ | – | – | partial | – | +| Honest "approximate" flag when offline is a proxy | ✓ | – | – | – | – | – | +| Per-file attribution in CI | ✓ | – | – | – | – | – | + +Tokenometer is the only tool in this list that combines **multi-provider + multi-format + empirical mode + USD cost + a PR-blocking GitHub Action + an honest approximate-vs-exact flag**. tokencost is the closest match for cost-in-USD across providers, but it doesn't compare formats or run as a CI guardrail. tiktoken and gpt-tokenizer are great single-provider primitives — Tokenometer uses gpt-tokenizer under the hood for the offline path. promptfoo is the broadest evaluator overall, but cost is one input among many; it isn't a dedicated cost-guardrail. The VS Code extension is real-time-in-editor only. ## Findings (Anthropic, n=150 cells across 10 prompt shapes) @@ -15,6 +37,25 @@ It started as a [\$23 question](https://hackernoon.com/i-spent-$23-testing-the-y Reproduce: `npm install && npm run benchmarks:empirical` with `ANTHROPIC_API_KEY` set. Full sweep is free (countTokens is free). +## Demo + +```text +$ tokenometer ./prompt.md --model claude-opus-4-7 --format json,yaml,markdown + + Model Format Tokens USD Approx + ────────────────── ────────── ──────── ───────── ────── + claude-opus-4-7 json 1,243 $0.0186 ✓ + claude-opus-4-7 yaml 1,189 $0.0178 ✓ + claude-opus-4-7 markdown 1,156 $0.0173 ✓ + + Cheapest: claude-opus-4-7 as markdown ($0.0173) + Priciest: claude-opus-4-7 as json ($0.0186, 1.08x more) +``` + +The `Approx` column shows `✓` when the count is a proxy (Anthropic / Google offline) and is empty when it's an exact match (OpenAI offline, or any provider with `--empirical`). + +> Real demo (with empirical mode + GIF) at **https://tokenometer.vercel.app**. + ## Why this exists `tiktoken` and `@anthropic-ai/tokenizer` give you a token count for one provider. They don't tell you: @@ -98,9 +139,17 @@ Tokenometer picks a tokenizer per provider and flags the count as approximate (` Cost = `tokens / 1000 × per-1k input rate`. Pricing and context windows are sourced from the [`tokenlens`](https://www.npmjs.com/package/tokenlens) registry, with a small set of local overrides for bleeding-edge models the registry hasn't picked up yet — see [`packages/core/src/rates.ts`](packages/core/src/rates.ts) (`RATES_VERSION`). +## Project health + +- [Code of Conduct](CODE_OF_CONDUCT.md) +- [Contributing guide](CONTRIBUTING.md) +- [Security policy](SECURITY.md) — uses GitHub Private Vulnerability Reporting +- [Changelog](CHANGELOG.md) +- [Discussions](https://github.com/faraa2m/tokenometer/discussions) + ## Status -Early. v0.0.x — see [milestones](https://github.com/faraa2m/tokenometer/milestones). +Early. v0.0.x — see [milestones](https://github.com/faraa2m/tokenometer/milestones). Roadmap to v1.0.0 in progress: VS Code extension, Claude Code skill, vision-token cost, Mistral + Cohere providers. ## License diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..80b0678 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,37 @@ +# Security Policy + +## Reporting + +Use **GitHub Private Vulnerability Reporting (PVR)**. Open this repo's Security tab and click **Report a vulnerability**. Reports are private and visible only to maintainers. + +Do not file public issues for vulnerabilities. Public reports will be closed and re-filed privately. + +We do not list a contact email. PVR is the only intake channel — this avoids email scraping and keeps the audit trail in one place. + +## Supported versions + +| Version | Status | +|---|---| +| `1.x` (latest minor) | Supported. Security fixes backported. | +| `0.0.x` (pre-1.0) | Best-effort only. Upgrade to `1.x` for fixes. | + +## Disclosure timeline + +Default coordinated-disclosure window is 90 days from the first acknowledgement. We can extend by mutual agreement when a fix needs more runway, or shorten if the issue is being exploited. + +Typical flow: + +1. Report received via PVR. Acknowledged within 7 days. +2. Maintainers reproduce and assess severity. +3. Fix developed in a private fork. +4. Fix released; advisory published; reporter credited (unless they prefer not to be). + +## Out of scope + +- Issues in dependencies (file those upstream; we'll bump after they land). +- Self-inflicted misconfiguration (e.g., committing your own API keys to a repo). +- Findings against `https://tokenometer.vercel.app` that result from the playground being a public read-only demo. + +## Recognition + +Reporters who follow this process are credited in the published advisory and the release notes for the fix, unless they ask to remain anonymous. diff --git a/package-lock.json b/package-lock.json index 1956830..ad34dd1 100644 --- a/package-lock.json +++ b/package-lock.json @@ -14,6 +14,7 @@ ], "devDependencies": { "@biomejs/biome": "^1.9.4", + "@changesets/changelog-github": "^0.7.0", "@changesets/cli": "^2.27.11", "@types/node": "^22.10.5", "typescript": "^5.7.2", @@ -624,6 +625,18 @@ "@changesets/types": "^6.1.0" } }, + "node_modules/@changesets/changelog-github": { + "version": "0.7.0", + "resolved": "https://registry.npmjs.org/@changesets/changelog-github/-/changelog-github-0.7.0.tgz", + "integrity": "sha512-rBsbRvc4TVn+FvFnOVM3LxlFJfTXXCp8gfVJ+0BubxWNSVnLuAzowi5j+IEraLLP52w8AAs9QfKbPS3MMiXQJA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@changesets/get-github-info": "^0.8.0", + "@changesets/types": "^6.1.0", + "dotenv": "^8.1.0" + } + }, "node_modules/@changesets/cli": { "version": "2.31.0", "resolved": "https://registry.npmjs.org/@changesets/cli/-/cli-2.31.0.tgz", @@ -702,6 +715,17 @@ "semver": "^7.5.3" } }, + "node_modules/@changesets/get-github-info": { + "version": "0.8.0", + "resolved": "https://registry.npmjs.org/@changesets/get-github-info/-/get-github-info-0.8.0.tgz", + "integrity": "sha512-cRnC+xdF0JIik7coko3iUP9qbnfi1iJQ3sAa6dE+Tx3+ET8bjFEm63PA4WEohgjYcmsOikPHWzPsMWWiZmntOQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "dataloader": "^1.4.0", + "node-fetch": "^2.5.0" + } + }, "node_modules/@changesets/get-release-plan": { "version": "4.0.16", "resolved": "https://registry.npmjs.org/@changesets/get-release-plan/-/get-release-plan-4.0.16.tgz", @@ -2844,6 +2868,13 @@ "dev": true, "license": "MIT" }, + "node_modules/dataloader": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/dataloader/-/dataloader-1.4.0.tgz", + "integrity": "sha512-68s5jYdlvasItOJnCuI2Q9s4q98g0pCyL3HrcKJu8KNugUl8ahgmZYg38ysLTgQjjXX3H8CJLkAvWrclWfcalw==", + "dev": true, + "license": "BSD-3-Clause" + }, "node_modules/debug": { "version": "4.4.3", "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", @@ -2920,6 +2951,16 @@ "node": ">=8" } }, + "node_modules/dotenv": { + "version": "8.6.0", + "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-8.6.0.tgz", + "integrity": "sha512-IrPdXQsk2BbzvCBGBOTmmSH5SodmqZNt4ERAZDmW4CT+tL8VtvinqywuANaFu4bOMWki16nqf0e4oC0QIaDr/g==", + "dev": true, + "license": "BSD-2-Clause", + "engines": { + "node": ">=10" + } + }, "node_modules/dunder-proto": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz", diff --git a/package.json b/package.json index 45408e2..220fa26 100644 --- a/package.json +++ b/package.json @@ -2,7 +2,7 @@ "name": "tokenometer-monorepo", "private": true, "version": "0.0.0", - "description": "Empirical token-cost benchmarking for LLM prompts.", + "description": "Tokenometer — LLM cost calculator, token counter, and CI cost-guardrail Action for Claude, GPT-4o, Gemini.", "license": "MIT", "author": "Faraazuddin Mohammed ", "homepage": "https://github.com/faraa2m/tokenometer", @@ -32,6 +32,7 @@ }, "devDependencies": { "@biomejs/biome": "^1.9.4", + "@changesets/changelog-github": "^0.7.0", "@changesets/cli": "^2.27.11", "@types/node": "^22.10.5", "typescript": "^5.7.2", diff --git a/packages/action/README.md b/packages/action/README.md index 9c0c678..994937f 100644 --- a/packages/action/README.md +++ b/packages/action/README.md @@ -1,7 +1,12 @@ # Tokenometer GitHub Action +[![License: MIT](https://img.shields.io/github/license/faraa2m/tokenometer.svg)](https://github.com/faraa2m/tokenometer/blob/main/LICENSE) + + Posts a sticky PR comment with the prompt-cost diff between your branch and its base. Fails the check when the delta exceeds a budget. +See the [root README](https://github.com/faraa2m/tokenometer#readme) for findings, methodology, and the full project overview. + ## Usage ```yaml diff --git a/packages/action/package.json b/packages/action/package.json index e6fd658..001818e 100644 --- a/packages/action/package.json +++ b/packages/action/package.json @@ -1,9 +1,10 @@ { "name": "@tokenometer/action", "version": "0.0.2", - "description": "GitHub Action that posts a sticky PR comment with prompt-cost diff.", + "description": "Tokenometer GitHub Action — sticky PR comment with prompt-cost diff and budget gate.", "license": "MIT", "private": true, + "keywords": ["ci", "code-quality", "github-action", "llm-cost", "prompt-cost"], "type": "module", "main": "./dist/index.js", "scripts": { diff --git a/packages/cli/README.md b/packages/cli/README.md index 664f462..a19308f 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -1,7 +1,12 @@ # tokenometer +[![npm tokenometer](https://img.shields.io/npm/v/tokenometer.svg?label=tokenometer)](https://www.npmjs.com/package/tokenometer) +[![License: MIT](https://img.shields.io/github/license/faraa2m/tokenometer.svg)](https://github.com/faraa2m/tokenometer/blob/main/LICENSE) + > Empirical token-cost benchmarking for LLM prompts. Tells you what your prompt actually costs across Claude, GPT-4o, and Gemini, in every format. +See the [root README](https://github.com/faraa2m/tokenometer#readme) for findings, methodology, and the full project overview. + [**Live playground: tokenometer.vercel.app**](https://tokenometer.vercel.app) · [Source](https://github.com/faraa2m/tokenometer) · MIT ```bash diff --git a/packages/cli/package.json b/packages/cli/package.json index 443ce58..bec2409 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,7 +1,7 @@ { "name": "tokenometer", "version": "0.0.2", - "description": "Empirical token-cost benchmarking CLI for LLM prompts. Tells you what your prompt actually costs across Claude, GPT-4o, and Gemini, in every format.", + "description": "Tokenometer CLI — measure prompt token cost across Claude, GPT-4o, Gemini. Multi-provider, multi-format, empirical mode.", "license": "MIT", "author": "Faraazuddin Mohammed ", "homepage": "https://tokenometer.vercel.app", @@ -16,15 +16,25 @@ "keywords": [ "ai", "anthropic", + "ci-cd", "claude", "cli", "cost", + "cost-calculator", "gemini", + "github-action", "gpt", + "gpt-4o", "llm", + "llm-cost", "openai", "prompt", + "prompt-cost", + "prompt-engineering", + "prompt-regression", + "tiktoken", "token", + "token-budget", "tokenizer" ], "type": "module", diff --git a/packages/core/README.md b/packages/core/README.md index ec45fc3..1ee4e11 100644 --- a/packages/core/README.md +++ b/packages/core/README.md @@ -1,7 +1,12 @@ # @tokenometer/core +[![npm @tokenometer/core](https://img.shields.io/npm/v/@tokenometer/core.svg?label=@tokenometer/core)](https://www.npmjs.com/package/@tokenometer/core) +[![License: MIT](https://img.shields.io/github/license/faraa2m/tokenometer.svg)](https://github.com/faraa2m/tokenometer/blob/main/LICENSE) + > Core library powering [tokenometer](https://www.npmjs.com/package/tokenometer): tokenizer dispatch, format converters, versioned cost rate matrix, and an empirical-mode `countTokens` adapter for Anthropic, OpenAI, and Google. +See the [root README](https://github.com/faraa2m/tokenometer#readme) for findings, methodology, and the full project overview. + [**Live playground**](https://tokenometer.vercel.app) · [Source](https://github.com/faraa2m/tokenometer) · MIT If you just want a CLI, `npm install -g tokenometer`. This package is for programmatic use. diff --git a/packages/core/package.json b/packages/core/package.json index c9599d4..07daa47 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -1,7 +1,7 @@ { "name": "@tokenometer/core", "version": "0.0.2", - "description": "Empirical token-cost benchmarking for LLM prompts — core library (tokenizers, format converters, rate matrix, empirical countTokens dispatch).", + "description": "Tokenometer core library — multi-provider LLM token cost estimation with empirical countTokens fallback.", "license": "MIT", "author": "Faraazuddin Mohammed ", "homepage": "https://tokenometer.vercel.app", @@ -16,16 +16,26 @@ "keywords": [ "ai", "anthropic", - "claude", + "ci-cd", "cl100k", + "claude", "cost", + "cost-calculator", "gemini", + "github-action", "gpt", + "gpt-4o", "llm", + "llm-cost", "o200k", "openai", "prompt", + "prompt-cost", + "prompt-engineering", + "prompt-regression", + "tiktoken", "token", + "token-budget", "tokenizer" ], "type": "module", diff --git a/packages/web/package.json b/packages/web/package.json index d86c18c..8b07faa 100644 --- a/packages/web/package.json +++ b/packages/web/package.json @@ -1,7 +1,7 @@ { "name": "@tokenometer/web", "version": "0.0.2", - "description": "Browser playground for paste-and-compare prompt cost analysis.", + "description": "Tokenometer playground — calculator, vision-cost, prompt-diff, model atlas.", "license": "MIT", "private": true, "type": "module",