diff --git a/README.md b/README.md index 61ea9c0..b6f8bf6 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ The last one is easy to miss. The same prompt caches fine on Sonnet 4.6 (minimum | `lint request.json [--model ID]` | Checks one request for things that stop caching | it finds an error | | `diff previous.json next.json` | Explains why the second request missed the first one's cache | the cached part changed | | `probe request.json --model ID [--region R] [--stream]` | Sends the request twice and checks the second one read from cache | nothing was read from cache | -| `logs files-or-folders... [--by model\|principal] [--min-hit 0.5] [--price 3]` | Hit rate and input cost per model or IAM principal from Bedrock invocation logs, plus how much the misses cost | a group is under the threshold | +| `logs files-or-folders... [--by model\|principal] [--min-hit 0.5] [--price 3] [--ttl-check]` | Hit rate and input cost per model or IAM principal from Bedrock invocation logs, plus how much the misses cost. `--ttl-check` adds why calls had to write the cache again, and what those writes cost at the read and 1-hour write prices | a group is under the threshold | | `langfuse [--since 7d] [--environment E] [--by model\|name\|environment] [--min-hit 0.5] [--price 3]` | The same, from the generations your Langfuse project already records (no invocation logging needed) | a group is under the threshold | Exit code 2 means it couldn't run at all (bad file, no model access, no AWS credentials). `--json` and `--github` go before the command. @@ -137,6 +137,37 @@ How the dollars work: It reads S3 deliveries, CloudWatch exports and `aws logs filter-log-events` output, gzipped or not, including streamed responses. Bedrock logs InvokeModel calls under an inference profile ARN and Converse calls under the short model ID, so CacheCanary merges the two. +### Why calls had to write the cache again (`--ttl-check`) + +A low hit rate has different causes, and they have different fixes. If the cache expires between calls, the 1-hour cache helps. If something else is the cause, the 1-hour cache only makes every cache write cost more (2x the input price instead of 1.25x, on the Bedrock price list). `--ttl-check` looks at every call that read nothing from the cache but wrote to it, finds the previous call that started the same way (same model, same system prompt and tools), and tells you how long before it came: + +```text +$ cachecanary logs tests/fixtures/bedrock_invocation_logs_ttl_2026-11.json --ttl-check +us.anthropic.claude-sonnet-4-5-20250929-v1:0: hit 62% over 8 calls (read 105702, write 63422, uncached 104, unparsed 0) + input cost $0.35 at list price; about $0.22 of it lost to cache misses (target: 90% hit rate) +List prices: Amazon Bedrock on-demand, Oct 2026 (global. IDs at list, others +10%). Use --price for your own rate. + +Cache re-writes (calls that read nothing from the cache but wrote to it): +us.anthropic.claude-sonnet-4-5-20250929-v1:0: 3 re-write(s) in 8 calls, 63422 tokens written by them + 2 no earlier call that started the same way read or wrote the cache in these logs (42281 tokens) + 1 5 to 60 minutes after the previous call that started the same way (21141 tokens) + The 5-to-60-minute re-writes cost $0.09 as cache writes; the same tokens would cost under $0.01 as cache reads + All cache writes here cost $0.31; the same tokens at the 1-hour write price would cost $0.42 +``` + +That example comes from a live test on November 4 2026 (Claude Sonnet 4.5, us-west-2): two agents with the same size of system prompt (21,000 tokens), one on the 5-minute cache and one on the 1-hour cache, each called at 0, 2, 7 and 20 minutes. The token counts are what Bedrock returned ([the raw run](https://github.com/Haarris/cachecanary/blob/main/tests/fixtures/live_ttl_check_2026-11-04.jsonl), made by `scripts/live_ttl_check.py`); `scripts/build_ttl_fixture.py` turns it into log records with the prompts redacted and timestamps that follow the test schedule. The 5-minute agent was still read at minute 7 (every cache hit restarts the 5 minutes), then had to write everything again at minute 20. The 1-hour agent was read at minute 20. + +How to read it: + +- **5 to 60 minutes**: the cache had probably expired between calls. This is what the 1-hour cache is for. The two dollar lines are prices of what the logs show, not a forecast: what those re-writes cost and what the same tokens would cost as reads, and what all your cache writes cost and what they would cost at the 1-hour write price. If the first line's difference (writes minus reads) is much bigger than the second line's (the 1-hour price minus today's), and the re-written tokens are the shared start of the prompt (check two requests with `diff`), the 1-hour cache is worth a try. +- **Within 5 minutes**: the cache should not have expired yet. AWS says a cache entry is stored on the server that processed the request, and requests "served by a different server, such as one in another Region, or that are sent before an identical request has started its response, write a new entry". Other causes are a change later in the request with no cache point right after the system prompt and tools, or Bedrock's lookback limit. The 1-hour cache won't fix any of these. A different region is shown when the logs record it; run `diff` on two saved requests to rule out a prompt change. +- **Less than 1 second apart**: usually parallel requests sent before the first one was cached. +- **Latest write all on the 1-hour cache**: the cache write before this call went entirely to the 1-hour cache and this one still missed within the hour, so the lifetime wasn't the problem. + +Calls that read from the cache and also wrote aren't classified. In the logs, new text at the end of the prompt (the next turn of an agent loop) looks the same as one conversation's own history expiring while other conversations keep a shared system prompt cached. + +Matching by system prompt and tools needs the request body in the log record. Bedrock includes it when text data delivery is on and the body is 100 KB or less; bigger bodies (and binary data) go to separate S3 objects, for CloudWatch delivery only when an S3 location for large data is set. Calls without one, or without a system prompt or tools, are matched to the previous call from the same IAM caller and model instead, and the output says so. A call that never read or wrote the cache doesn't count as the previous call. With `--ttl-check`, `--json` output becomes `{"groups", "notes", "ttl_check", "ttl_check_skipped"}`; the GitHub summary doesn't include it yet. + ## Hit rates from Langfuse If you trace your agents with [Langfuse](https://langfuse.com), you don't need invocation logging (more detail in the [Langfuse guide](https://cachecanary.com/langfuse/)). Langfuse already stores the cache counts for every model call, and `langfuse` reads them through the Langfuse API: @@ -166,7 +197,7 @@ Only model calls (generations) are counted. Agent and chain spans can carry the ## How it's tested -The checks follow AWS's documented prompt-caching rules. The core behaviour was verified against live Amazon Bedrock: Converse and InvokeModel, streaming, both cache lifetimes (5 minutes and 1 hour), model minimums, prompt changes and the exact lookback limit (21 positions hit, 22 miss, re-measured October 30 2026 on Sonnet 4.6 and 5.5 after Bedrock changed how it counts tool calls). Every check added in 0.3 was measured live first, and `scripts/live_v03_checks.py` re-runs those cases against Bedrock and compares CacheCanary's verdict with what Bedrock did (15 of 15 agreed in October 2026). Every check also has unit tests. The log reader is tested against real Bedrock invocation logs (redacted copies are in [tests/fixtures](https://github.com/Haarris/cachecanary/tree/main/tests/fixtures)). The Langfuse reader is tested against records that Strands, LiteLLM and LangChain wrote into a real Langfuse 4.64 server, and against a production Langfuse Cloud project, where its numbers matched a separate recount for every model. +The checks follow AWS's documented prompt-caching rules. The core behaviour was verified against live Amazon Bedrock: Converse and InvokeModel, streaming, both cache lifetimes (5 minutes and 1 hour), model minimums, prompt changes and the exact lookback limit (21 positions hit, 22 miss, re-measured October 30 2026 on Sonnet 4.6 and 5.5 after Bedrock changed how it counts tool calls). Every check added in 0.3 was measured live first, and `scripts/live_v03_checks.py` re-runs those cases against Bedrock and compares CacheCanary's verdict with what Bedrock did (15 of 15 agreed in October 2026). Every check also has unit tests. The log reader is tested against real Bedrock invocation logs (redacted copies are in [tests/fixtures](https://github.com/Haarris/cachecanary/tree/main/tests/fixtures)), and `--ttl-check` against real logs with request bodies from Converse, ConverseStream and InvokeModel calls, including one stored in S3 (eu-west-1, November 2026). The Langfuse reader is tested against records that Strands, LiteLLM and LangChain wrote into a real Langfuse 4.64 server, and against a production Langfuse Cloud project, where its numbers matched a separate recount for every model. To run the live checks yourself, use an AWS Region without production traffic: `python scripts/live_test.py --region `. @@ -180,6 +211,7 @@ Everything runs on your machine or CI runner. Nothing is sent anywhere. `probe` - Bedrock keeps cache counts inside the logged response body, and only bodies up to 100 KB are stored inline. Bigger ones show up as `unparsed` and aren't in the dollar figures. - `langfuse` can only check a call's counts when the integration sent a total. Calls with no cache fields at all are counted as uncached and listed in the output, because Langfuse can't tell "caching was off" from "the integration doesn't report cache tokens". - Dollar figures use list prices, which change. Batch, provisioned throughput and priority tiers aren't covered; use `--price` for those. +- `--ttl-check` matches calls by system prompt and tools only when the request body is in the log record. The same prompt sent through Converse and InvokeModel, or as a plain string versus a list of blocks, is written differently in the logs, so it counts as two different starts. The served region (`inferenceRegion`) isn't in AWS's documented log format, though real records include it; without it, re-writes within 5 minutes show "region not in the logs". - Bedrock only for now. ## What's next diff --git a/scripts/build_ttl_fixture.py b/scripts/build_ttl_fixture.py new file mode 100644 index 0000000..6c7e7f2 --- /dev/null +++ b/scripts/build_ttl_fixture.py @@ -0,0 +1,46 @@ +"""Build tests/fixtures/bedrock_invocation_logs_ttl_2026-11.json from the live run of scripts/live_ttl_check.py. + +The live run records the usage Bedrock returned for each call and the call's minute in the schedule, +not a clock time, so the fixture's timestamps follow the schedule (16:03 + minute; the 1-hour agent one +second later). Prompts are replaced with a placeholder of the same role; account and request IDs are +dummies; inferenceRegion is left out because the run did not record it. + + python scripts/build_ttl_fixture.py +""" +import json +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +ARN = "arn:aws:bedrock:us-west-2:111122223333:inference-profile/us.anthropic.claude-sonnet-4-5-20250929-v1:0" + + +def build(rows): + recs = [] + for i, r in enumerate(sorted(rows, key=lambda r: (r["minute"], r["ttl"]))): + cc = {"type": "ephemeral"} if r["ttl"] == "5m" else {"type": "ephemeral", "ttl": "1h"} + usage = {"input_tokens": r["uncached"], "cache_read_input_tokens": r["read"], + "cache_creation_input_tokens": r["write"], "cache_creation": r["cache_creation"], "output_tokens": 2} + recs.append({ + "schemaType": "ModelInvocationLog", "schemaVersion": "1.0", + "timestamp": f"2026-11-04T16:{3 + r['minute']:02d}:{10 if r['ttl'] == '5m' else 11}Z", + "accountId": "111122223333", "region": "us-west-2", "requestId": f"00000000-0000-0000-0000-{i:012d}", + "operation": "InvokeModel", "modelId": ARN, + "identity": {"arn": f"arn:aws:sts::111122223333:assumed-role/ExampleRole/agent-{r['ttl']}"}, + "input": {"inputContentType": "application/json", "inputTokenCount": r["uncached"], + "inputBodyJson": {"anthropic_version": "bedrock-2023-05-31", "max_tokens": 20, + "system": [{"type": "text", "cache_control": cc, + "text": f"[{r['ttl']} agent: 21,000-token system prompt, redacted]"}], + "messages": [{"role": "user", "content": [ + {"type": "text", "text": f"Task {r['call']}: reply OK."}]}]}}, + "output": {"outputContentType": "application/json", "outputTokenCount": 2, + "outputBodyJson": {"type": "message", "role": "assistant", + "content": [{"type": "text", "text": "OK"}], "usage": usage}}}) + return recs + + +if __name__ == "__main__": + rows = [json.loads(line) for line in (ROOT / "tests/fixtures/live_ttl_check_2026-11-04.jsonl").read_text().splitlines() + if line.strip()] + out = ROOT / "tests/fixtures/bedrock_invocation_logs_ttl_2026-11.json" + out.write_text(json.dumps(build(rows), indent=1)) + print(f"wrote {out} ({len(rows)} calls)") diff --git a/scripts/live_ttl_check.py b/scripts/live_ttl_check.py new file mode 100644 index 0000000..9ade64a --- /dev/null +++ b/scripts/live_ttl_check.py @@ -0,0 +1,70 @@ +"""Live check behind the `logs --ttl-check` README example: 5-minute vs 1-hour cache with real gaps. + +Two agents with a ~21,000-token system prompt, one on the 5-minute cache and one on the 1-hour cache, +call Claude Sonnet 4.5 at 0, 2, 7 and 20 minutes. Each result line is appended to ttl_test.jsonl; +the November 4 2026 run is tests/fixtures/live_ttl_check_2026-11-04.jsonl, and +scripts/build_ttl_fixture.py turns it into the invocation-log fixture the README uses. + +Same request shape Claude Code sends on Bedrock (InvokeModel, Anthropic Messages body, cache_control +breakpoints on system and the last message). Calls at t = 0, +2 min, +7 min, +20 min. +Records cache read / write (with the 5m/1h split) and latency for each call. + + python scripts/live_ttl_check.py --region us-west-2 (uses your default AWS credentials; under $1) +""" +import argparse, json, sys, threading, time, uuid +import boto3 + +MODEL = "us.anthropic.claude-sonnet-4-5-20250929-v1:0" +GAPS_MIN = [0, 2, 7, 20] # minutes after the first call + + +def prefix(salt): + rules = [f"[{salt}] You are an operations agent for a small company workspace."] + for i in range(440): + rules.append(f"Skill note {i}: when a task touches area {i % 17}, read the linked runbook first, keep changes " + f"reversible, log what you did, and hand off to a human if the request is ambiguous or risky.") + return "\n".join(rules) + + +def body(sys_text, question, ttl): + cc = {"type": "ephemeral"} if ttl == "5m" else {"type": "ephemeral", "ttl": "1h"} + return {"anthropic_version": "bedrock-2023-05-31", "max_tokens": 20, + "system": [{"type": "text", "text": sys_text, "cache_control": cc}], + "messages": [{"role": "user", "content": [{"type": "text", "text": question}]}]} + + +def run_arm(br, ttl, out, t0): + sys_text = prefix(uuid.uuid4().hex[:10]) + for k, gap in enumerate(GAPS_MIN): + wait = t0 + gap * 60 - time.time() + if wait > 0: + time.sleep(wait) + start = time.time() + r = br.invoke_model(modelId=MODEL, body=json.dumps(body(sys_text, f"Task {k}: reply OK.", ttl))) + latency_ms = round((time.time() - start) * 1000) + u = json.loads(r["body"].read())["usage"] + row = {"ttl": ttl, "call": k, "minute": gap, "latency_ms": latency_ms, + "read": u.get("cache_read_input_tokens", 0), "write": u.get("cache_creation_input_tokens", 0), + "uncached": u.get("input_tokens", 0), "cache_creation": u.get("cache_creation")} + out.write(json.dumps(row) + "\n"); out.flush() + print(json.dumps(row), flush=True) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--region", required=True) + a = ap.parse_args() + br = boto3.client("bedrock-runtime", region_name=a.region) + assert br.meta.region_name == a.region + out = open("ttl_test.jsonl", "a") + t0 = time.time() + 5 + arms = [threading.Thread(target=run_arm, args=(br, ttl, out, t0)) for ttl in ("5m", "1h")] + for t in arms: + t.start() + for t in arms: + t.join() + print("DONE", flush=True) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/site/index.html b/site/index.html index 602f2cd..53298a9 100644 --- a/site/index.html +++ b/site/index.html @@ -31,7 +31,7 @@ "url": "https://cachecanary.com/", "applicationCategory": "DeveloperApplication", "operatingSystem": "macOS, Linux, Windows", - "softwareVersion": "0.4.1", + "softwareVersion": "0.5.0", "downloadUrl": "https://pypi.org/project/cachecanary/", "license": "https://www.apache.org/licenses/LICENSE-2.0", "offers": { @@ -142,6 +142,16 @@

1. Shows whether you're losing money right now

input cost $0.05 at list price; about $0.03 of it lost to cache misses (target: 80% hit rate) List prices: Amazon Bedrock on-demand, Oct 2026 (global. IDs at list, others +10%). Use --price for your own rate.

Here only half the calls used the cache, below the 80% target, so the command fails and shows what the misses cost on this small sample. Run it as a scheduled CI job and that failure becomes your alert. Prices are Bedrock's published list prices; pass --price if you pay a different rate.

+

A low hit rate can have different causes, and the fixes are different. Add --ttl-check and CacheCanary looks at every call that had to write the whole cache again, finds the previous call with the same system prompt and tools, and says how long before it came. If the cache expired between calls (5 to 60 minutes), the 1-hour cache helps. If it was within 5 minutes, the 1-hour cache only makes writes cost more.

+
$ cachecanary logs bedrock-logs/ --ttl-check
+...
+Cache re-writes (calls that read nothing from the cache but wrote to it):
+us.anthropic.claude-sonnet-4-5-20250929-v1:0: 3 re-write(s) in 8 calls, 63422 tokens written by them
+       2  no earlier call that started the same way read or wrote the cache in these logs (42281 tokens)
+       1  5 to 60 minutes after the previous call that started the same way (21141 tokens)
+  The 5-to-60-minute re-writes cost $0.09 as cache writes; the same tokens would cost under $0.01 as cache reads
+  All cache writes here cost $0.31; the same tokens at the 1-hour write price would cost $0.42
+

This is a live test: two agents with a 21,000-token prompt, one on the 5-minute cache and one on the 1-hour cache, called at 0, 2, 7 and 20 minutes. The 5-minute one had to write everything again at minute 20. The dollar lines are prices of what the logs show, not a forecast. The README explains each row.

Already trace your agents with Langfuse? You can skip the logging. Langfuse stores the cache counts for every model call, so one command reads your hit rate from there:

$ cachecanary langfuse --since 7d --environment production --min-hit 0.8

It needs your Langfuse project's API keys and uses only the token counts. On Langfuse v4 it doesn't download prompts or outputs at all. Some setups store numbers in Langfuse that don't add up, like older Strands versions, which make the cache look perfect. CacheCanary fixes those; the Langfuse guide shows what it found.

diff --git a/site/llms.txt b/site/llms.txt index 5123dae..57442e4 100644 --- a/site/llms.txt +++ b/site/llms.txt @@ -2,7 +2,7 @@ > CacheCanary is a free, open-source (Apache-2.0) Python CLI and GitHub Action that catches Claude prompt caching breaking silently on Amazon Bedrock. It shows the cache hit rate and what misses cost from Bedrock invocation logs or Langfuse traces, explains why one request missed the previous request's cache, and checks saved requests in CI. It runs locally and sends nothing anywhere. -Install with `pip install cachecanary`. Commands: `lint` (check one saved Converse or InvokeModel request), `diff` (explain why the second of two requests missed the first one's cache), `probe` (send a request twice to Bedrock and confirm the second reads from cache), `logs` (hit rate, input cost and dollars lost to misses from Bedrock invocation logs or a folder of them), `langfuse` (the same from the generations a Langfuse project already records, read through the Langfuse API: uses token counts only; on Langfuse v4 prompts and outputs are not downloaded). +Install with `pip install cachecanary`. Commands: `lint` (check one saved Converse or InvokeModel request), `diff` (explain why the second of two requests missed the first one's cache), `probe` (send a request twice to Bedrock and confirm the second reads from cache), `logs` (hit rate, input cost and dollars lost to misses from Bedrock invocation logs or a folder of them; with `--ttl-check` it also classifies every call that had to write the whole cache again by the time since the previous call with the same system prompt and tools, so you can tell cache expiry, which the 1-hour cache fixes, from other causes), `langfuse` (the same from the generations a Langfuse project already records, read through the Langfuse API: uses token counts only; on Langfuse v4 prompts and outputs are not downloaded). What it checks, each verified against live Amazon Bedrock (Claude Sonnet 4.6, us-west-2, October 2026): diff --git a/src/cachecanary/__init__.py b/src/cachecanary/__init__.py index 9ec0d26..4702780 100644 --- a/src/cachecanary/__init__.py +++ b/src/cachecanary/__init__.py @@ -1,3 +1,3 @@ """CacheCanary: know the moment your Claude prompt cache breaks on Amazon Bedrock.""" -__version__ = "0.4.1" +__version__ = "0.5.0" diff --git a/src/cachecanary/cli.py b/src/cachecanary/cli.py index 4984414..71717c1 100644 --- a/src/cachecanary/cli.py +++ b/src/cachecanary/cli.py @@ -3,7 +3,7 @@ cachecanary lint request.json [--model ID] cachecanary diff previous.json next.json [--model ID] cachecanary probe request.json --model ID [--region R] [--stream] - cachecanary logs FILE_OR_FOLDER ... [--by model|principal|model+principal] [--min-hit 0.5] [--price 3] + cachecanary logs FILE_OR_FOLDER ... [--by model|principal|model+principal] [--min-hit 0.5] [--price 3] [--ttl-check] cachecanary langfuse [--since 7d | --from DATE --to DATE] [--environment E] [--by model|name|environment] Exit codes: 0 ok, 1 caching problem found (use to gate CI), 2 could not run (bad input, AWS error). @@ -18,7 +18,7 @@ from datetime import datetime, timezone from pathlib import Path -from cachecanary import __version__, diff, langfuse, lint, logs, pricing, probe +from cachecanary import __version__, diff, langfuse, lint, logs, pricing, probe, ttl from cachecanary import github as gh from cachecanary.request import RequestError, normalize @@ -167,10 +167,65 @@ def cmd_logs(args) -> int: raise InputError(f"could not read {path}: {exc}") from exc groups = logs.aggregate(records, by=args.by) notes = [f"Skipped {stats.bad_lines} unreadable line(s)."] if stats.bad_lines else [] - return _report(groups, args, "logs", "No invocation log records found.", notes) + extra = _ttl_check(records, args.price) if args.ttl_check else None + return _report(groups, args, "logs", "No invocation log records found.", notes, extra) + + +def _ttl_check(records, custom_price) -> tuple[list[str], dict]: + """Text lines and JSON for `logs --ttl-check`.""" + calls, skipped = ttl.collect(records) + groups = ttl.check(calls, custom_price) + lines, data = ["", "Cache re-writes (calls that read nothing from the cache but wrote to it):"], {} + for model, g in sorted(groups.items(), key=lambda kv: -sum(kv[1].rewrite_tokens.values())): + total = sum(g.rewrites.values()) + money = g.priced + show_1h = money and g.supports_1h is True + data[model] = {"calls": g.calls, "calls_body_in_s3": g.calls_body_in_s3, "calls_no_body": g.calls_no_body, + "rewrites": g.rewrites, "rewrite_tokens": g.rewrite_tokens, + "rewrites_within_1_second": g.same_second, + "cache_write_tokens": g.write_tokens, "cache_write_tokens_1h": g.write_tokens_1h, + "rewrites_5_to_60m_cost_usd": round(g.cost_5to60_now, 6) if money else None, + "rewrites_5_to_60m_as_reads_usd": round(g.cost_5to60_as_reads, 6) if money else None, + "cache_writes_cost_usd": round(g.cost_writes_now, 6) if money else None, + "cache_writes_at_1h_rate_usd": round(g.cost_writes_at_1h, 6) if show_1h else None} + lines.append(f"{model}: {total} re-write(s) in {g.calls} calls, " + f"{sum(g.rewrite_tokens.values())} tokens written by them") + for row in ttl.ROWS: + if g.rewrites.get(row): + lines.append(f" {g.rewrites[row]:>6} {ttl.LABELS[row]} ({g.rewrite_tokens[row]} tokens)") + if g.same_second: + lines.append(f" {g.same_second} of these came less than 1 second after the previous call " + f"(likely sent before that call's response had started)") + missing = g.calls_body_in_s3 + g.calls_no_body + if missing: + why = [] + if g.calls_no_body: + why.append(f"{g.calls_no_body} with no body logged") + if g.calls_body_in_s3: + why.append(f"{g.calls_body_in_s3} with the body stored as a separate S3 object (over 100 KB, or binary)") + lines.append(f" {missing} of {g.calls} call(s) have no request body in the record ({', '.join(why)}); " + f"they are matched by caller and model, which can mix different prompts") + if not money: + lines.append(" no price for this model; add --price to see dollars") + continue + if g.rewrites.get(ttl.FIVE_TO_60M) or g.rewrites.get(f"{ttl.FIVE_TO_60M}/caller"): + lines.append(f" The 5-to-60-minute re-writes cost {_money(g.cost_5to60_now)} as cache writes; " + f"the same tokens would cost {_money(g.cost_5to60_as_reads)} as cache reads") + if show_1h: + lines.append(f" All cache writes here cost {_money(g.cost_writes_now)}; the same tokens at the " + f"1-hour write price would cost {_money(g.cost_writes_at_1h)}") + elif ttl.lookup(model)[1] is None: + lines.append(" CacheCanary has no 1-hour cache information for this model") + else: + lines.append(" The 1-hour cache is not documented for this model on Bedrock") + if not groups: + lines = ["", "Cache re-writes: no calls with usage and a timestamp."] + if skipped: + lines.append(f"Left out of the re-write check: {skipped} record(s) with no usage or no timestamp.") + return lines, {"ttl_check": data, "ttl_check_skipped": skipped} -def _report(groups, args, source: str, empty: str, notes: list[str]) -> int: +def _report(groups, args, source: str, empty: str, notes: list[str], extra=None) -> int: """Hit rate and dollars per group, shared by `logs` and `langfuse`.""" target = args.min_hit if args.min_hit is not None else DEFAULT_TARGET_HIT data, lines, failing = {}, [], False @@ -213,9 +268,12 @@ def _report(groups, args, source: str, empty: str, notes: list[str]) -> int: "Use --price for your own rate.") lines.extend(notes) if args.json: - print(json.dumps({"groups": data, "notes": notes} if source == "langfuse" else data, indent=2)) + if extra is not None: + print(json.dumps({"groups": data, "notes": notes, **extra[1]}, indent=2)) + else: + print(json.dumps({"groups": data, "notes": notes} if source == "langfuse" else data, indent=2)) else: - print("\n".join(lines if data else [empty, *notes])) + print("\n".join((lines if data else [empty, *notes]) + (extra[0] if extra is not None and data else []))) if args.github: rows = [] for key, d in data.items(): @@ -353,6 +411,10 @@ def build_parser() -> argparse.ArgumentParser: p.add_argument("--min-hit", type=_unit, help="fail if any group's hit ratio is below this (0-1); " "also the target for the dollars lost figure (default 0.9)") p.add_argument("--price", type=_price, help="your input price in USD per million tokens, instead of list prices") + p.add_argument("--ttl-check", action="store_true", + help="also show why calls had to write the cache again (time since the previous call that " + "started the same way, region), and what the logged cache writes cost at the read and " + "1-hour write prices") p.set_defaults(func=cmd_logs) p = sub.add_parser("langfuse", help="cache hit rates from the generations in a Langfuse project", diff --git a/src/cachecanary/ttl.py b/src/cachecanary/ttl.py new file mode 100644 index 0000000..ff38902 --- /dev/null +++ b/src/cachecanary/ttl.py @@ -0,0 +1,246 @@ +"""Why calls had to write the cache again, from Bedrock invocation logs (`logs --ttl-check`). + +A call that read nothing from the cache but wrote to it is a re-write: no cached copy was found. +Each re-write is matched to the previous call that started the same way: +- with the request body in the logs and a system prompt or tools in it: the same model and the + same system prompt and tools (cache markers ignored); +- otherwise: the same model and the same caller (IAM principal), which can mix different prompts, + so those rows say "from this caller". +Only calls that read or wrote the cache count as the previous call. + +Then it reports how long before the re-write that call happened and whether it was served in a +different region. The default cache lifetime is 5 minutes, reset by every cache hit; the optional +1-hour cache keeps it for an hour. AWS: a cache entry is stored on the server that processed the +request, so a request served by a different server (for example in another region), or sent before +an identical request has started its response, writes a new entry. + +Calls that read from the cache and also wrote are not classified: in the logs, new text at the end +of the prompt looks the same as a conversation's own history expiring while a shared start stays +cached. Dollar figures are prices of what the logs show, not predictions. +""" + +import hashlib +import json +from dataclasses import dataclass, field +from datetime import datetime + +from cachecanary.logs import _body, _is_invocation_record +from cachecanary.models import canonical_model_id, lookup +from cachecanary.pricing import Rates, rates_for +from cachecanary.usage import Usage, from_response + +FIVE_MINUTES = 300 +ONE_HOUR = 3600 + +# Re-write classes, in the order they are printed. +FIRST = "first" +FIVE_TO_60M = "5_to_60m" +OVER_1H = "over_1h" +ON_1H_CACHE = "on_1h_cache" +WITHIN_5M_OTHER_REGION = "within_5m_other_region" +WITHIN_5M_REGION_UNKNOWN = "within_5m_region_unknown" +WITHIN_5M = "within_5m" +CLASSES = (FIRST, FIVE_TO_60M, OVER_1H, ON_1H_CACHE, WITHIN_5M_OTHER_REGION, WITHIN_5M_REGION_UNKNOWN, WITHIN_5M) + + +@dataclass +class Call: + when: datetime + model: str # canonical ID: used for grouping and pricing (keeps the global./us. prefix) + caller: str + region: str | None + prompt_start: str | None # hash of system prompt + tools, when the body is logged and has either + body: str # "inline", "s3" (over 100 KB, stored as a separate object) or "none" + usage: Usage + + @property + def touched_cache(self) -> bool: + return self.usage.cache_read + self.usage.cache_write > 0 + + +@dataclass +class Group: + calls: int = 0 + calls_body_in_s3: int = 0 + calls_no_body: int = 0 + # Keyed "" when matched by system prompt and tools, "/caller" when matched by caller. + rewrites: dict[str, int] = field(default_factory=dict) + rewrite_tokens: dict[str, int] = field(default_factory=dict) + same_second: int = 0 # re-writes less than a second after the previous call + write_tokens: int = 0 # every cache write in the group (re-writes and other writes) + write_tokens_1h: int = 0 + supports_1h: bool | None = None + priced: bool = True + cost_writes_now: float = 0.0 # all cache writes, as logged (5-minute / 1-hour split) + cost_writes_at_1h: float = 0.0 # the same write tokens at the 1-hour write price + cost_5to60_now: float = 0.0 # the 5-60 minute re-writes, as logged + cost_5to60_as_reads: float = 0.0 # the same tokens at the cache read price + + +def _parse_time(value) -> datetime | None: + if not isinstance(value, str) or not value: + return None + try: + when = datetime.fromisoformat(value.replace("Z", "+00:00")) + except ValueError: + return None + return when if when.tzinfo else None + + +def _is_marker(key, value) -> bool: + return key == "cache_control" and isinstance(value, dict) and value.get("type") == "ephemeral" + + +def _strip_cache_markers(value): + """Cache markers don't change what is cached, so leave them out of the fingerprint.""" + if isinstance(value, dict): + if set(value) == {"cachePoint"}: + return None + return {k: _strip_cache_markers(v) for k, v in value.items() if not _is_marker(k, v)} + if isinstance(value, list): + return [x for x in (_strip_cache_markers(v) for v in value) if x is not None] + return value + + +def _request_body(rec: dict): + inp = rec.get("input") + body = inp.get("inputBodyJson") if isinstance(inp, dict) else None + if isinstance(body, str): + try: + body = json.loads(body) + except json.JSONDecodeError: + return None + return body if isinstance(body, dict) and "messages" in body else None + + +def _body_state(rec: dict) -> str: + if _request_body(rec) is not None: + return "inline" + inp = rec.get("input") + if isinstance(inp, dict) and inp.get("inputBodyS3Path"): + return "s3" + return "none" + + +def prompt_start(rec: dict) -> str | None: + """Fingerprint of the system prompt and tools in the logged request body (Converse or Claude Messages). + + None when there is no usable body, or when it has neither a system prompt nor tools (then the + fingerprint would match every unrelated conversation on the model). + """ + body = _request_body(rec) + if body is None: + return None + tools = body.get("tools") + if tools is None and isinstance(body.get("toolConfig"), dict): + tools = body["toolConfig"].get("tools") + system = _strip_cache_markers(body.get("system")) + tools = _strip_cache_markers(tools) + if not system and not tools: + return None + start = {"system": system, "tools": tools} + return hashlib.sha256(json.dumps(start, sort_keys=True, ensure_ascii=False).encode()).hexdigest()[:16] + + +def collect(records) -> tuple[list[Call], int]: + """Calls with usage and a timestamp, and how many invocation records were left out.""" + calls, skipped = [], 0 + for rec in records: + if not _is_invocation_record(rec): + continue + usage = from_response(_body(rec)) + when = _parse_time(rec.get("timestamp")) + if usage is None or when is None: + skipped += 1 + continue + identity = rec.get("identity") + caller = identity.get("arn") if isinstance(identity, dict) else None + region = rec.get("inferenceRegion") + calls.append(Call(when, str(canonical_model_id(rec.get("modelId")) or "?"), str(caller or "?"), + region if isinstance(region, str) and region else None, + prompt_start(rec), _body_state(rec), usage)) + return calls, skipped + + +def classify(gap_seconds: float | None, region: str | None, previous_region: str | None, + previous_all_1h: bool = False) -> str: + if gap_seconds is None: + return FIRST + if gap_seconds > ONE_HOUR: + return OVER_1H + if gap_seconds > FIVE_MINUTES: + return ON_1H_CACHE if previous_all_1h else FIVE_TO_60M + if not region or not previous_region: + return WITHIN_5M_REGION_UNKNOWN + if region != previous_region: + return WITHIN_5M_OTHER_REGION + return WITHIN_5M + + +def _write_cost(u: Usage, rates: Rates) -> float: + write_1h = min(u.cache_write_1h, u.cache_write) + return ((u.cache_write - write_1h) * rates.cache_write_5m + write_1h * rates.cache_write_1h) / 1_000_000 + + +def check(calls: list[Call], custom_price: float | None = None) -> dict[str, Group]: + """Re-writes per model, each classified by the time since the previous call that started the same way.""" + groups: dict[str, Group] = {} + last: dict[tuple, Call] = {} + last_write_all_1h: dict[tuple, bool] = {} # was the latest write in this stream entirely 1-hour? + for call in sorted(calls, key=lambda c: c.when): + g = groups.get(call.model) + if g is None: + _, limits = lookup(call.model) + g = groups[call.model] = Group(supports_1h=limits.supports_1h if limits else None) + g.calls += 1 + g.calls_body_in_s3 += call.body == "s3" + g.calls_no_body += call.body == "none" + u = call.usage + exact = call.prompt_start is not None + key = (call.model, "prompt", call.prompt_start) if exact else (call.model, "caller", call.caller) + previous = last.get(key) + priced = rates_for(call.model, custom_price) + if priced is None: + g.priced = False + if u.cache_write > 0: + g.write_tokens += u.cache_write + g.write_tokens_1h += min(u.cache_write_1h, u.cache_write) + if priced: + g.cost_writes_now += _write_cost(u, priced.rates) + g.cost_writes_at_1h += u.cache_write * priced.rates.cache_write_1h / 1_000_000 + if u.cache_write > 0 and u.cache_read == 0: + gap = (call.when - previous.when).total_seconds() if previous else None + cls = classify(gap, call.region, previous.region if previous else None, + last_write_all_1h.get(key, False)) + row = cls if exact else f"{cls}/caller" + g.rewrites[row] = g.rewrites.get(row, 0) + 1 + g.rewrite_tokens[row] = g.rewrite_tokens.get(row, 0) + u.cache_write + if gap is not None and gap < 1: + g.same_second += 1 + if cls == FIVE_TO_60M and priced: + g.cost_5to60_now += _write_cost(u, priced.rates) + g.cost_5to60_as_reads += u.cache_write * priced.rates.cache_read / 1_000_000 + if call.touched_cache: + # Calls with a prompt fingerprint also update their caller's timeline, so a caller-matched + # call (for example one whose body grew past 100 KB) sees every earlier call from that caller. + for k in {key, (call.model, "caller", call.caller)}: + if u.cache_write > 0: + last_write_all_1h[k] = u.cache_write_1h >= u.cache_write + last[k] = call + return groups + + +_WHEN = { + FIRST: "no earlier call {who} read or wrote the cache in these logs", + FIVE_TO_60M: "5 to 60 minutes after the previous call {who}", + OVER_1H: "more than 1 hour after the previous call {who}", + ON_1H_CACHE: "5 to 60 minutes after the previous call {who}; the latest cache write before it was all " + "on the 1-hour cache", + WITHIN_5M_OTHER_REGION: "within 5 minutes of the previous call {who}, served in a different region", + WITHIN_5M_REGION_UNKNOWN: "within 5 minutes of the previous call {who} (region not in the logs)", + WITHIN_5M: "within 5 minutes of the previous call {who}, same region", +} +ROWS = list(CLASSES) + [f"{c}/caller" for c in CLASSES] +LABELS = {**{c: t.format(who="that started the same way") for c, t in _WHEN.items()}, + **{f"{c}/caller": t.format(who="from this caller (matched by caller and model, not by prompt)") + for c, t in _WHEN.items()}} diff --git a/tests/fixtures/bedrock_invocation_logs_eu_ttl_2026-11.jsonl b/tests/fixtures/bedrock_invocation_logs_eu_ttl_2026-11.jsonl new file mode 100644 index 0000000..0be7d67 --- /dev/null +++ b/tests/fixtures/bedrock_invocation_logs_eu_ttl_2026-11.jsonl @@ -0,0 +1,10 @@ +{"timestamp": "2026-11-04T16:46:30Z", "accountId": "111122223333", "region": "eu-west-1", "requestId": "00000000-0000-0000-0000-000000000000", "operation": "InvokeModel", "modelId": "arn:aws:bedrock:eu-west-1:111122223333:inference-profile/eu.anthropic.claude-sonnet-4-6", "input": {"inputContentType": "application/json", "inputBodyJson": {"anthropic_version": "bedrock-2023-05-31", "max_tokens": 5, "system": [{"type": "text", "text": "[redacted text 1, 21953 chars]", "cache_control": {"type": "ephemeral"}}], "messages": [{"role": "user", "content": [{"type": "text", "text": "[redacted text 2, 9 chars]"}]}], "model": "eu.anthropic.claude-sonnet-4-6", "stream": false}, "inputTokenCount": 9, "cacheReadInputTokenCount": 0, "cacheWriteInputTokenCount": 5511}, "output": {"outputContentType": "application/json", "outputBodyJson": {"model": "claude-sonnet-4-6", "id": "msg_bdrk_i24to36mjnvhuanl5ouovjz6d47sjm5ylskyzypebbr3mwmoe22q", "type": "message", "role": "assistant", "content": [{"type": "text", "text": "[redacted text 3, 3 chars]"}], "container": null, "stop_reason": "end_turn", "stop_sequence": null, "stop_details": null, "usage": {"input_tokens": 9, "cache_creation_input_tokens": 5511, "cache_read_input_tokens": 0, "cache_creation": {"ephemeral_5m_input_tokens": 5511, "ephemeral_1h_input_tokens": 0}, "output_tokens": 5, "service_tier": "standard"}}, "outputTokenCount": 5}, "identity": {"arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/example-user"}, "inferenceRegion": "eu-west-1", "schemaVersion": "1.0", "schemaType": "ModelInvocationLog"} +{"timestamp": "2026-11-04T16:46:31Z", "accountId": "111122223333", "region": "eu-west-1", "requestId": "00000000-0000-0000-0000-000000000001", "operation": "Converse", "modelId": "eu.anthropic.claude-sonnet-4-6", "input": {"inputContentType": "application/json", "inputBodyJson": {"messages": [{"role": "user", "content": [{"text": "[redacted text 2, 9 chars]"}, {"cachePoint": {"type": "default"}}]}], "system": [{"text": "[redacted text 4, 21953 chars]"}, {"cachePoint": {"type": "default"}}], "inferenceConfig": {"maxTokens": 5}, "toolConfig": {"tools": [{"toolSpec": {"name": "tool_0", "description": "Looks up record type 0.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"toolSpec": {"name": "tool_1", "description": "Looks up record type 1.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"toolSpec": {"name": "tool_2", "description": "Looks up record type 2.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"toolSpec": {"name": "tool_3", "description": "Looks up record type 3.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"toolSpec": {"name": "tool_4", "description": "Looks up record type 4.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"cachePoint": {"type": "default"}}]}}, "inputTokenCount": 3, "cacheReadInputTokenCount": 0, "cacheWriteInputTokenCount": 6260}, "output": {"outputContentType": "application/json", "outputBodyJson": {"output": {"message": {"role": "assistant", "content": [{"text": "[redacted text 3, 3 chars]"}]}}, "stopReason": "end_turn", "metrics": {"latencyMs": 812}, "usage": {"inputTokens": 3, "cacheReadInputTokens": 0, "cacheWriteInputTokens": 6260, "outputTokens": 5, "totalTokens": 6268}}, "outputTokenCount": 5}, "identity": {"arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/example-user"}, "inferenceRegion": "eu-west-1", "schemaVersion": "1.0", "schemaType": "ModelInvocationLog"} +{"timestamp": "2026-11-04T16:46:32Z", "accountId": "111122223333", "region": "eu-west-1", "requestId": "00000000-0000-0000-0000-000000000002", "operation": "Converse", "modelId": "eu.anthropic.claude-sonnet-4-6", "input": {"inputContentType": "application/json", "inputBodyS3Path": "s3://cachecanary-logtest-111122223333-euw1/AWSLogs/111122223333/BedrockModelInvocationLogs/eu-west-1/2026/11/04/16/data/00000000-0000-0000-0000-000000000000_input.json.gz", "inputTokenCount": 3, "cacheReadInputTokenCount": 0, "cacheWriteInputTokenCount": 26368}, "output": {"outputContentType": "application/json", "outputBodyJson": {"output": {"message": {"role": "assistant", "content": [{"text": "[redacted text 5, 27 chars]"}]}}, "stopReason": "max_tokens", "metrics": {"latencyMs": 1292}, "usage": {"inputTokens": 3, "cacheReadInputTokens": 0, "cacheWriteInputTokens": 26368, "outputTokens": 5, "totalTokens": 26376}}, "outputTokenCount": 5}, "identity": {"arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/example-user"}, "inferenceRegion": "eu-west-1", "schemaVersion": "1.0", "schemaType": "ModelInvocationLog"} +{"timestamp": "2026-11-04T16:47:29Z", "accountId": "111122223333", "region": "eu-west-1", "requestId": "00000000-0000-0000-0000-000000000003", "operation": "Converse", "modelId": "eu.anthropic.claude-sonnet-4-6", "input": {"inputContentType": "application/json", "inputBodyJson": {"messages": [{"role": "user", "content": [{"text": "[redacted text 2, 9 chars]"}, {"cachePoint": {"type": "default"}}]}], "system": [{"text": "[redacted text 4, 21953 chars]"}, {"cachePoint": {"type": "default"}}], "inferenceConfig": {"maxTokens": 5}, "toolConfig": {"tools": [{"toolSpec": {"name": "tool_0", "description": "Looks up record type 0.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"toolSpec": {"name": "tool_1", "description": "Looks up record type 1.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"toolSpec": {"name": "tool_2", "description": "Looks up record type 2.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"toolSpec": {"name": "tool_3", "description": "Looks up record type 3.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"toolSpec": {"name": "tool_4", "description": "Looks up record type 4.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"cachePoint": {"type": "default"}}]}}, "inputTokenCount": 3, "cacheReadInputTokenCount": 6260, "cacheWriteInputTokenCount": 0}, "output": {"outputContentType": "application/json", "outputBodyJson": {"output": {"message": {"role": "assistant", "content": [{"text": "[redacted text 3, 3 chars]"}]}}, "stopReason": "end_turn", "metrics": {"latencyMs": 1565}, "usage": {"inputTokens": 3, "cacheReadInputTokens": 6260, "cacheWriteInputTokens": 0, "outputTokens": 5, "totalTokens": 6268}}, "outputTokenCount": 5}, "identity": {"arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/example-user"}, "inferenceRegion": "eu-west-1", "schemaVersion": "1.0", "schemaType": "ModelInvocationLog"} +{"timestamp": "2026-11-04T16:48:29Z", "accountId": "111122223333", "region": "eu-west-1", "requestId": "00000000-0000-0000-0000-000000000004", "operation": "InvokeModel", "modelId": "arn:aws:bedrock:eu-west-1:111122223333:inference-profile/eu.anthropic.claude-sonnet-4-6", "input": {"inputContentType": "application/json", "inputBodyJson": {"anthropic_version": "bedrock-2023-05-31", "max_tokens": 5, "system": [{"type": "text", "text": "[redacted text 1, 21953 chars]", "cache_control": {"type": "ephemeral"}}], "messages": [{"role": "user", "content": [{"type": "text", "text": "[redacted text 2, 9 chars]"}]}], "model": "eu.anthropic.claude-sonnet-4-6", "stream": false}, "inputTokenCount": 9, "cacheReadInputTokenCount": 5511, "cacheWriteInputTokenCount": 0}, "output": {"outputContentType": "application/json", "outputBodyJson": {"model": "claude-sonnet-4-6", "id": "msg_bdrk_5j6plzbsqa54focmaf4uc4ktgbw3bmukosv4yvnxshkktqwimz5q", "type": "message", "role": "assistant", "content": [{"type": "text", "text": "[redacted text 3, 3 chars]"}], "container": null, "stop_reason": "end_turn", "stop_sequence": null, "stop_details": null, "usage": {"input_tokens": 9, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 5511, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}, "output_tokens": 5, "service_tier": "standard"}}, "outputTokenCount": 5}, "identity": {"arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/example-user"}, "inferenceRegion": "eu-west-1", "schemaVersion": "1.0", "schemaType": "ModelInvocationLog"} +{"timestamp": "2026-11-04T16:49:29Z", "accountId": "111122223333", "region": "eu-west-1", "requestId": "00000000-0000-0000-0000-000000000005", "operation": "Converse", "modelId": "eu.anthropic.claude-sonnet-4-6", "input": {"inputContentType": "application/json", "inputBodyJson": {"messages": [{"role": "user", "content": [{"text": "[redacted text 2, 9 chars]"}, {"cachePoint": {"type": "default"}}]}], "system": [{"text": "[redacted text 6, 21979 chars]"}, {"cachePoint": {"type": "default"}}], "inferenceConfig": {"maxTokens": 5}, "toolConfig": {"tools": [{"toolSpec": {"name": "tool_0", "description": "Looks up record type 0.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"toolSpec": {"name": "tool_1", "description": "Looks up record type 1.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"toolSpec": {"name": "tool_2", "description": "Looks up record type 2.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"toolSpec": {"name": "tool_3", "description": "Looks up record type 3.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"toolSpec": {"name": "tool_4", "description": "Looks up record type 4.", "inputSchema": {"json": {"type": "object", "properties": {"id": {"type": "string"}}}}}}, {"cachePoint": {"type": "default"}}]}}, "inputTokenCount": 3, "cacheReadInputTokenCount": 0, "cacheWriteInputTokenCount": 6273}, "output": {"outputContentType": "application/json", "outputBodyJson": {"output": {"message": {"role": "assistant", "content": [{"text": "[redacted text 3, 3 chars]"}]}}, "stopReason": "end_turn", "metrics": {"latencyMs": 789}, "usage": {"inputTokens": 3, "cacheReadInputTokens": 0, "cacheWriteInputTokens": 6273, "outputTokens": 5, "totalTokens": 6281}}, "outputTokenCount": 5}, "identity": {"arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/example-user"}, "inferenceRegion": "eu-west-1", "schemaVersion": "1.0", "schemaType": "ModelInvocationLog"} +{"timestamp": "2026-11-04T16:50:29Z", "accountId": "111122223333", "region": "eu-west-1", "requestId": "00000000-0000-0000-0000-000000000006", "operation": "InvokeModel", "modelId": "arn:aws:bedrock:eu-west-1:111122223333:inference-profile/eu.anthropic.claude-sonnet-4-6", "input": {"inputContentType": "application/json", "inputBodyJson": {"anthropic_version": "bedrock-2023-05-31", "max_tokens": 5, "system": [{"type": "text", "text": "[redacted text 7, 21953 chars]", "cache_control": {"type": "ephemeral"}}], "messages": [{"role": "user", "content": [{"type": "text", "text": "[redacted text 2, 9 chars]"}]}], "model": "eu.anthropic.claude-sonnet-4-6", "stream": false}, "inputTokenCount": 9, "cacheReadInputTokenCount": 0, "cacheWriteInputTokenCount": 5512}, "output": {"outputContentType": "application/json", "outputBodyJson": {"model": "claude-sonnet-4-6", "id": "msg_bdrk_e6uuxduqz62s5kvm7cirragszadogx6xagksdbtzb2ttrqukju2q", "type": "message", "role": "assistant", "content": [{"type": "text", "text": "[redacted text 3, 3 chars]"}], "container": null, "stop_reason": "end_turn", "stop_sequence": null, "stop_details": null, "usage": {"input_tokens": 9, "cache_creation_input_tokens": 5512, "cache_read_input_tokens": 0, "cache_creation": {"ephemeral_5m_input_tokens": 5512, "ephemeral_1h_input_tokens": 0}, "output_tokens": 5, "service_tier": "standard"}}, "outputTokenCount": 5}, "identity": {"arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/example-user"}, "inferenceRegion": "eu-west-1", "schemaVersion": "1.0", "schemaType": "ModelInvocationLog"} +{"timestamp": "2026-11-04T16:50:30Z", "accountId": "111122223333", "region": "eu-west-1", "requestId": "00000000-0000-0000-0000-000000000007", "operation": "InvokeModel", "modelId": "arn:aws:bedrock:eu-west-1:111122223333:inference-profile/eu.anthropic.claude-sonnet-4-6", "input": {"inputContentType": "application/json", "inputBodyJson": {"anthropic_version": "bedrock-2023-05-31", "max_tokens": 5, "system": [{"type": "text", "text": "[redacted text 7, 21953 chars]", "cache_control": {"type": "ephemeral"}}], "messages": [{"role": "user", "content": [{"type": "text", "text": "[redacted text 2, 9 chars]"}]}], "model": "eu.anthropic.claude-sonnet-4-6", "stream": false}, "inputTokenCount": 9, "cacheReadInputTokenCount": 5512, "cacheWriteInputTokenCount": 0}, "output": {"outputContentType": "application/json", "outputBodyJson": {"model": "claude-sonnet-4-6", "id": "msg_bdrk_qg5tny5ofqsbeaoiek3damlqculcttjjbebkmi6vhruodc6wl6da", "type": "message", "role": "assistant", "content": [{"type": "text", "text": "[redacted text 3, 3 chars]"}], "container": null, "stop_reason": "end_turn", "stop_sequence": null, "stop_details": null, "usage": {"input_tokens": 9, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 5512, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}, "output_tokens": 5, "service_tier": "standard"}}, "outputTokenCount": 5}, "identity": {"arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/example-user"}, "inferenceRegion": "eu-west-1", "schemaVersion": "1.0", "schemaType": "ModelInvocationLog"} +{"timestamp": "2026-11-04T16:51:29Z", "accountId": "111122223333", "region": "eu-west-1", "requestId": "00000000-0000-0000-0000-000000000008", "operation": "ConverseStream", "modelId": "eu.anthropic.claude-sonnet-4-6", "input": {"inputContentType": "application/json", "inputBodyJson": {"messages": [{"role": "user", "content": [{"text": "[redacted text 2, 9 chars]"}]}], "system": [{"text": "[redacted text 8, 21953 chars]"}, {"cachePoint": {"type": "default"}}], "inferenceConfig": {"maxTokens": 5}}, "inputTokenCount": 9, "cacheReadInputTokenCount": 0, "cacheWriteInputTokenCount": 5511}, "output": {"outputContentType": "application/json", "outputBodyJson": {"output": {"message": {"role": "assistant", "content": [{"text": "[redacted text 3, 3 chars]"}]}}, "stopReason": "end_turn", "metrics": {"latencyMs": 836}, "usage": {"inputTokens": 9, "cacheReadInputTokens": 0, "cacheWriteInputTokens": 5511, "outputTokens": 5, "totalTokens": 5525}}, "outputTokenCount": 5}, "identity": {"arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/example-user"}, "inferenceRegion": "eu-west-1", "schemaVersion": "1.0", "schemaType": "ModelInvocationLog"} +{"timestamp": "2026-11-04T16:54:29Z", "accountId": "111122223333", "region": "eu-west-1", "requestId": "00000000-0000-0000-0000-000000000009", "operation": "InvokeModel", "modelId": "arn:aws:bedrock:eu-west-1:111122223333:inference-profile/eu.anthropic.claude-sonnet-4-6", "input": {"inputContentType": "application/json", "inputBodyJson": {"anthropic_version": "bedrock-2023-05-31", "max_tokens": 5, "system": [{"type": "text", "text": "[redacted text 1, 21953 chars]", "cache_control": {"type": "ephemeral"}}], "messages": [{"role": "user", "content": [{"type": "text", "text": "[redacted text 2, 9 chars]"}]}], "model": "eu.anthropic.claude-sonnet-4-6", "stream": false}, "inputTokenCount": 9, "cacheReadInputTokenCount": 0, "cacheWriteInputTokenCount": 5511}, "output": {"outputContentType": "application/json", "outputBodyJson": {"model": "claude-sonnet-4-6", "id": "msg_bdrk_tr3x5ovcfar52c2nz5hni4tdli4r75h4nrmbybmo2mbkb32jw6rq", "type": "message", "role": "assistant", "content": [{"type": "text", "text": "[redacted text 3, 3 chars]"}], "container": null, "stop_reason": "end_turn", "stop_sequence": null, "stop_details": null, "usage": {"input_tokens": 9, "cache_creation_input_tokens": 5511, "cache_read_input_tokens": 0, "cache_creation": {"ephemeral_5m_input_tokens": 5511, "ephemeral_1h_input_tokens": 0}, "output_tokens": 5, "service_tier": "standard"}}, "outputTokenCount": 5}, "identity": {"arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/example-user"}, "inferenceRegion": "eu-west-1", "schemaVersion": "1.0", "schemaType": "ModelInvocationLog"} diff --git a/tests/fixtures/bedrock_invocation_logs_ttl_2026-11.json b/tests/fixtures/bedrock_invocation_logs_ttl_2026-11.json new file mode 100644 index 0000000..73a6a66 --- /dev/null +++ b/tests/fixtures/bedrock_invocation_logs_ttl_2026-11.json @@ -0,0 +1,526 @@ +[ + { + "schemaType": "ModelInvocationLog", + "schemaVersion": "1.0", + "timestamp": "2026-11-04T16:03:11Z", + "accountId": "111122223333", + "region": "us-west-2", + "requestId": "00000000-0000-0000-0000-000000000000", + "operation": "InvokeModel", + "modelId": "arn:aws:bedrock:us-west-2:111122223333:inference-profile/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + "identity": { + "arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/agent-1h" + }, + "input": { + "inputContentType": "application/json", + "inputTokenCount": 13, + "inputBodyJson": { + "anthropic_version": "bedrock-2023-05-31", + "max_tokens": 20, + "system": [ + { + "type": "text", + "cache_control": { + "type": "ephemeral", + "ttl": "1h" + }, + "text": "[1h agent: 21,000-token system prompt, redacted]" + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "Task 0: reply OK." + } + ] + } + ] + } + }, + "output": { + "outputContentType": "application/json", + "outputTokenCount": 2, + "outputBodyJson": { + "type": "message", + "role": "assistant", + "content": [ + { + "type": "text", + "text": "OK" + } + ], + "usage": { + "input_tokens": 13, + "cache_read_input_tokens": 0, + "cache_creation_input_tokens": 21140, + "cache_creation": { + "ephemeral_5m_input_tokens": 0, + "ephemeral_1h_input_tokens": 21140 + }, + "output_tokens": 2 + } + } + } + }, + { + "schemaType": "ModelInvocationLog", + "schemaVersion": "1.0", + "timestamp": "2026-11-04T16:03:10Z", + "accountId": "111122223333", + "region": "us-west-2", + "requestId": "00000000-0000-0000-0000-000000000001", + "operation": "InvokeModel", + "modelId": "arn:aws:bedrock:us-west-2:111122223333:inference-profile/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + "identity": { + "arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/agent-5m" + }, + "input": { + "inputContentType": "application/json", + "inputTokenCount": 13, + "inputBodyJson": { + "anthropic_version": "bedrock-2023-05-31", + "max_tokens": 20, + "system": [ + { + "type": "text", + "cache_control": { + "type": "ephemeral" + }, + "text": "[5m agent: 21,000-token system prompt, redacted]" + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "Task 0: reply OK." + } + ] + } + ] + } + }, + "output": { + "outputContentType": "application/json", + "outputTokenCount": 2, + "outputBodyJson": { + "type": "message", + "role": "assistant", + "content": [ + { + "type": "text", + "text": "OK" + } + ], + "usage": { + "input_tokens": 13, + "cache_read_input_tokens": 0, + "cache_creation_input_tokens": 21141, + "cache_creation": { + "ephemeral_5m_input_tokens": 21141, + "ephemeral_1h_input_tokens": 0 + }, + "output_tokens": 2 + } + } + } + }, + { + "schemaType": "ModelInvocationLog", + "schemaVersion": "1.0", + "timestamp": "2026-11-04T16:05:11Z", + "accountId": "111122223333", + "region": "us-west-2", + "requestId": "00000000-0000-0000-0000-000000000002", + "operation": "InvokeModel", + "modelId": "arn:aws:bedrock:us-west-2:111122223333:inference-profile/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + "identity": { + "arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/agent-1h" + }, + "input": { + "inputContentType": "application/json", + "inputTokenCount": 13, + "inputBodyJson": { + "anthropic_version": "bedrock-2023-05-31", + "max_tokens": 20, + "system": [ + { + "type": "text", + "cache_control": { + "type": "ephemeral", + "ttl": "1h" + }, + "text": "[1h agent: 21,000-token system prompt, redacted]" + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "Task 1: reply OK." + } + ] + } + ] + } + }, + "output": { + "outputContentType": "application/json", + "outputTokenCount": 2, + "outputBodyJson": { + "type": "message", + "role": "assistant", + "content": [ + { + "type": "text", + "text": "OK" + } + ], + "usage": { + "input_tokens": 13, + "cache_read_input_tokens": 21140, + "cache_creation_input_tokens": 0, + "cache_creation": { + "ephemeral_5m_input_tokens": 0, + "ephemeral_1h_input_tokens": 0 + }, + "output_tokens": 2 + } + } + } + }, + { + "schemaType": "ModelInvocationLog", + "schemaVersion": "1.0", + "timestamp": "2026-11-04T16:05:10Z", + "accountId": "111122223333", + "region": "us-west-2", + "requestId": "00000000-0000-0000-0000-000000000003", + "operation": "InvokeModel", + "modelId": "arn:aws:bedrock:us-west-2:111122223333:inference-profile/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + "identity": { + "arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/agent-5m" + }, + "input": { + "inputContentType": "application/json", + "inputTokenCount": 13, + "inputBodyJson": { + "anthropic_version": "bedrock-2023-05-31", + "max_tokens": 20, + "system": [ + { + "type": "text", + "cache_control": { + "type": "ephemeral" + }, + "text": "[5m agent: 21,000-token system prompt, redacted]" + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "Task 1: reply OK." + } + ] + } + ] + } + }, + "output": { + "outputContentType": "application/json", + "outputTokenCount": 2, + "outputBodyJson": { + "type": "message", + "role": "assistant", + "content": [ + { + "type": "text", + "text": "OK" + } + ], + "usage": { + "input_tokens": 13, + "cache_read_input_tokens": 21141, + "cache_creation_input_tokens": 0, + "cache_creation": { + "ephemeral_5m_input_tokens": 0, + "ephemeral_1h_input_tokens": 0 + }, + "output_tokens": 2 + } + } + } + }, + { + "schemaType": "ModelInvocationLog", + "schemaVersion": "1.0", + "timestamp": "2026-11-04T16:10:11Z", + "accountId": "111122223333", + "region": "us-west-2", + "requestId": "00000000-0000-0000-0000-000000000004", + "operation": "InvokeModel", + "modelId": "arn:aws:bedrock:us-west-2:111122223333:inference-profile/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + "identity": { + "arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/agent-1h" + }, + "input": { + "inputContentType": "application/json", + "inputTokenCount": 13, + "inputBodyJson": { + "anthropic_version": "bedrock-2023-05-31", + "max_tokens": 20, + "system": [ + { + "type": "text", + "cache_control": { + "type": "ephemeral", + "ttl": "1h" + }, + "text": "[1h agent: 21,000-token system prompt, redacted]" + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "Task 2: reply OK." + } + ] + } + ] + } + }, + "output": { + "outputContentType": "application/json", + "outputTokenCount": 2, + "outputBodyJson": { + "type": "message", + "role": "assistant", + "content": [ + { + "type": "text", + "text": "OK" + } + ], + "usage": { + "input_tokens": 13, + "cache_read_input_tokens": 21140, + "cache_creation_input_tokens": 0, + "cache_creation": { + "ephemeral_5m_input_tokens": 0, + "ephemeral_1h_input_tokens": 0 + }, + "output_tokens": 2 + } + } + } + }, + { + "schemaType": "ModelInvocationLog", + "schemaVersion": "1.0", + "timestamp": "2026-11-04T16:10:10Z", + "accountId": "111122223333", + "region": "us-west-2", + "requestId": "00000000-0000-0000-0000-000000000005", + "operation": "InvokeModel", + "modelId": "arn:aws:bedrock:us-west-2:111122223333:inference-profile/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + "identity": { + "arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/agent-5m" + }, + "input": { + "inputContentType": "application/json", + "inputTokenCount": 13, + "inputBodyJson": { + "anthropic_version": "bedrock-2023-05-31", + "max_tokens": 20, + "system": [ + { + "type": "text", + "cache_control": { + "type": "ephemeral" + }, + "text": "[5m agent: 21,000-token system prompt, redacted]" + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "Task 2: reply OK." + } + ] + } + ] + } + }, + "output": { + "outputContentType": "application/json", + "outputTokenCount": 2, + "outputBodyJson": { + "type": "message", + "role": "assistant", + "content": [ + { + "type": "text", + "text": "OK" + } + ], + "usage": { + "input_tokens": 13, + "cache_read_input_tokens": 21141, + "cache_creation_input_tokens": 0, + "cache_creation": { + "ephemeral_5m_input_tokens": 0, + "ephemeral_1h_input_tokens": 0 + }, + "output_tokens": 2 + } + } + } + }, + { + "schemaType": "ModelInvocationLog", + "schemaVersion": "1.0", + "timestamp": "2026-11-04T16:23:11Z", + "accountId": "111122223333", + "region": "us-west-2", + "requestId": "00000000-0000-0000-0000-000000000006", + "operation": "InvokeModel", + "modelId": "arn:aws:bedrock:us-west-2:111122223333:inference-profile/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + "identity": { + "arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/agent-1h" + }, + "input": { + "inputContentType": "application/json", + "inputTokenCount": 13, + "inputBodyJson": { + "anthropic_version": "bedrock-2023-05-31", + "max_tokens": 20, + "system": [ + { + "type": "text", + "cache_control": { + "type": "ephemeral", + "ttl": "1h" + }, + "text": "[1h agent: 21,000-token system prompt, redacted]" + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "Task 3: reply OK." + } + ] + } + ] + } + }, + "output": { + "outputContentType": "application/json", + "outputTokenCount": 2, + "outputBodyJson": { + "type": "message", + "role": "assistant", + "content": [ + { + "type": "text", + "text": "OK" + } + ], + "usage": { + "input_tokens": 13, + "cache_read_input_tokens": 21140, + "cache_creation_input_tokens": 0, + "cache_creation": { + "ephemeral_5m_input_tokens": 0, + "ephemeral_1h_input_tokens": 0 + }, + "output_tokens": 2 + } + } + } + }, + { + "schemaType": "ModelInvocationLog", + "schemaVersion": "1.0", + "timestamp": "2026-11-04T16:23:10Z", + "accountId": "111122223333", + "region": "us-west-2", + "requestId": "00000000-0000-0000-0000-000000000007", + "operation": "InvokeModel", + "modelId": "arn:aws:bedrock:us-west-2:111122223333:inference-profile/us.anthropic.claude-sonnet-4-5-20250929-v1:0", + "identity": { + "arn": "arn:aws:sts::111122223333:assumed-role/ExampleRole/agent-5m" + }, + "input": { + "inputContentType": "application/json", + "inputTokenCount": 13, + "inputBodyJson": { + "anthropic_version": "bedrock-2023-05-31", + "max_tokens": 20, + "system": [ + { + "type": "text", + "cache_control": { + "type": "ephemeral" + }, + "text": "[5m agent: 21,000-token system prompt, redacted]" + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "type": "text", + "text": "Task 3: reply OK." + } + ] + } + ] + } + }, + "output": { + "outputContentType": "application/json", + "outputTokenCount": 2, + "outputBodyJson": { + "type": "message", + "role": "assistant", + "content": [ + { + "type": "text", + "text": "OK" + } + ], + "usage": { + "input_tokens": 13, + "cache_read_input_tokens": 0, + "cache_creation_input_tokens": 21141, + "cache_creation": { + "ephemeral_5m_input_tokens": 21141, + "ephemeral_1h_input_tokens": 0 + }, + "output_tokens": 2 + } + } + } + } +] \ No newline at end of file diff --git a/tests/fixtures/live_ttl_check_2026-11-04.jsonl b/tests/fixtures/live_ttl_check_2026-11-04.jsonl new file mode 100644 index 0000000..59be49b --- /dev/null +++ b/tests/fixtures/live_ttl_check_2026-11-04.jsonl @@ -0,0 +1,8 @@ +{"ttl": "5m", "call": 0, "minute": 0, "latency_ms": 6865, "read": 0, "write": 21141, "uncached": 13, "cache_creation": {"ephemeral_5m_input_tokens": 21141, "ephemeral_1h_input_tokens": 0}} +{"ttl": "1h", "call": 0, "minute": 0, "latency_ms": 7745, "read": 0, "write": 21140, "uncached": 13, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 21140}} +{"ttl": "5m", "call": 1, "minute": 2, "latency_ms": 3695, "read": 21141, "write": 0, "uncached": 13, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}} +{"ttl": "1h", "call": 1, "minute": 2, "latency_ms": 4740, "read": 21140, "write": 0, "uncached": 13, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}} +{"ttl": "5m", "call": 2, "minute": 7, "latency_ms": 4947, "read": 21141, "write": 0, "uncached": 13, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}} +{"ttl": "1h", "call": 2, "minute": 7, "latency_ms": 5493, "read": 21140, "write": 0, "uncached": 13, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}} +{"ttl": "5m", "call": 3, "minute": 20, "latency_ms": 4106, "read": 0, "write": 21141, "uncached": 13, "cache_creation": {"ephemeral_5m_input_tokens": 21141, "ephemeral_1h_input_tokens": 0}} +{"ttl": "1h", "call": 3, "minute": 20, "latency_ms": 6608, "read": 21140, "write": 0, "uncached": 13, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}} diff --git a/tests/test_ttl.py b/tests/test_ttl.py new file mode 100644 index 0000000..d7f2ae7 --- /dev/null +++ b/tests/test_ttl.py @@ -0,0 +1,369 @@ +"""`logs --ttl-check`: why calls had to write the cache again.""" + +import json + +import pytest + +from cachecanary import cli, ttl + +MODEL = "global.anthropic.claude-sonnet-4-5-20250929-v1:0" # $3 input, $0.30 read, $3.75 5m write, $6 1h write + + +def rec(ts, read, write, *, uncached=10, system="S", tools=None, caller="role/a", region="us-east-1", + model=MODEL, body=True, write_1h=0, body_as_string=False, s3_body=False): + usage = {"inputTokens": uncached, "cacheReadInputTokens": read, "cacheWriteInputTokens": write, "outputTokens": 5} + if write_1h: + usage["cacheDetails"] = [{"ttl": "1h", "inputTokens": write_1h}] + r = {"schemaType": "ModelInvocationLog", "timestamp": ts, "modelId": model, "identity": {"arn": caller}, + "inferenceRegion": region, "input": {"inputTokenCount": uncached}, + "output": {"outputBodyJson": {"usage": usage}}} + if s3_body: + r["input"]["inputBodyS3Path"] = "s3://bucket/AWSLogs/data/x.json" + elif body: + b = {"system": [{"text": system}, {"cachePoint": {"type": "default"}}], "messages": [{"role": "user"}]} + if tools is not None: + b["toolConfig"] = {"tools": tools} + r["input"]["inputBodyJson"] = json.dumps(b) if body_as_string else b + return r + + +def run(records): + calls, skipped = ttl.collect(records) + return ttl.check(calls), skipped + + +def write_file(tmp_path, records): + path = tmp_path / "l.jsonl" + path.write_text("\n".join(json.dumps(r) for r in records)) + return path + + +# --- what counts as a re-write ------------------------------------------------ + +def test_only_calls_that_read_nothing_are_rewrites(): + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000), + rec("2026-11-01T10:01:00Z", 20000, 1500), # read and wrote: not classified + rec("2026-11-01T10:30:00Z", 2000, 20000), # partial read 29 min later: not classified either + rec("2026-11-01T10:31:00Z", 21500, 0)]) + g = groups[MODEL] + assert g.rewrites == {ttl.FIRST: 1} + assert g.write_tokens == 20000 + 1500 + 20000 + + +@pytest.mark.parametrize("second,region,expected", [ + ("2026-11-01T10:04:59Z", "us-east-1", ttl.WITHIN_5M), + ("2026-11-01T10:05:00Z", "us-east-1", ttl.WITHIN_5M), # exactly 5 minutes is still within + ("2026-11-01T10:05:01Z", "us-east-1", ttl.FIVE_TO_60M), + ("2026-11-01T11:00:00Z", "us-east-1", ttl.FIVE_TO_60M), # exactly 1 hour + ("2026-11-01T11:00:01Z", "us-east-1", ttl.OVER_1H), + ("2026-11-01T10:03:00Z", "us-east-2", ttl.WITHIN_5M_OTHER_REGION), + ("2026-11-01T10:03:00Z", None, ttl.WITHIN_5M_REGION_UNKNOWN), + ("2026-11-01T10:20:00Z", "us-east-2", ttl.FIVE_TO_60M), # region only matters inside 5 minutes +]) +def test_gap_classes(second, region, expected): + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000), rec(second, 0, 20000, region=region)]) + assert groups[MODEL].rewrites == {ttl.FIRST: 1, expected: 1} + + +def test_gap_counts_from_any_call_that_touched_the_cache(): + # A read at 10:04 resets the 5 minutes, so a re-write at 10:08 is 4 minutes after it. + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000), rec("2026-11-01T10:04:00Z", 20000, 0), + rec("2026-11-01T10:08:00Z", 0, 20000)]) + assert groups[MODEL].rewrites == {ttl.FIRST: 1, ttl.WITHIN_5M: 1} + + +def test_a_call_that_never_touched_the_cache_does_not_count_as_previous(): + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000), + rec("2026-11-01T10:20:00Z", 0, 0, uncached=20000), + rec("2026-11-01T10:22:00Z", 0, 20000)]) + assert groups[MODEL].rewrites == {ttl.FIRST: 1, ttl.FIVE_TO_60M: 1} + + +def test_same_second_rewrites_are_noted(): + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000), rec("2026-11-01T10:00:00.400Z", 0, 20000)]) + assert groups[MODEL].same_second == 1 + + +# --- matching ------------------------------------------------------------------ + +def test_prompts_from_one_caller_are_tracked_separately(): + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000, system="A"), + rec("2026-11-01T10:01:00Z", 0, 20000, system="B"), + rec("2026-11-01T10:30:00Z", 20000, 0, system="A"), + rec("2026-11-01T10:31:00Z", 0, 20000, system="B")]) + g = groups[MODEL] + assert g.rewrites == {ttl.FIRST: 2, ttl.FIVE_TO_60M: 1} + assert g.calls_no_body == g.calls_body_in_s3 == 0 + + +def test_without_request_bodies_matching_is_by_caller_and_says_so(): + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000, body=False), + rec("2026-11-01T10:10:00Z", 0, 20000, body=False), + rec("2026-11-01T10:11:00Z", 0, 20000, body=False, caller="role/b")]) + g = groups[MODEL] + assert g.calls_no_body == 3 + assert g.rewrites == {f"{ttl.FIRST}/caller": 2, f"{ttl.FIVE_TO_60M}/caller": 1} + assert "matched by caller and model, not by prompt" in ttl.LABELS[f"{ttl.FIVE_TO_60M}/caller"] + assert "started the same way" in ttl.LABELS[ttl.FIVE_TO_60M] + + +def test_body_in_s3_is_counted_separately_and_sees_the_callers_earlier_calls(): + # The conversation grew past 100 KB: its next call has no inline body. It is matched by caller, + # and the caller's earlier (inline) call at 10:00 is its previous call, so it is not "first". + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000), rec("2026-11-01T10:20:00Z", 0, 30000, s3_body=True)]) + g = groups[MODEL] + assert g.calls_body_in_s3 == 1 and g.calls_no_body == 0 + assert g.rewrites == {ttl.FIRST: 1, f"{ttl.FIVE_TO_60M}/caller": 1} + + +def test_a_newer_bodyless_call_does_not_override_the_matched_previous_call(): + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000), + rec("2026-11-01T10:28:00Z", 0, 5000, body=False), + rec("2026-11-01T10:30:00Z", 0, 20000)]) + rows = groups[MODEL].rewrites + assert rows[ttl.FIVE_TO_60M] == 1 # 30 minutes after the call that started the same way + assert ttl.WITHIN_5M not in rows + + +def test_no_system_and_no_tools_is_matched_by_caller_not_merged(): + a = rec("2026-11-01T10:00:00Z", 0, 20000, caller="role/a") + b = rec("2026-11-01T10:01:00Z", 0, 20000, caller="role/b") + for r in (a, b): + r["input"]["inputBodyJson"] = {"messages": [{"role": "user", "content": [{"text": r["identity"]["arn"]}]}]} + groups, _ = run([a, b]) + assert ttl.prompt_start(a) is None + assert groups[MODEL].rewrites == {f"{ttl.FIRST}/caller": 2} + + +def test_invokemodel_arn_and_short_id_are_one_stream(): + arn = "arn:aws:bedrock:us-east-1:111122223333:inference-profile/" + MODEL + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000, model=arn), rec("2026-11-01T10:20:00Z", 0, 20000)]) + assert list(groups) == [MODEL] + assert groups[MODEL].rewrites == {ttl.FIRST: 1, ttl.FIVE_TO_60M: 1} + + +# --- the 1-hour cache in the logs ------------------------------------------------ + +def test_previous_write_all_on_1h_is_not_called_an_expiry(): + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000, write_1h=20000), + rec("2026-11-01T10:20:00Z", 0, 20000)]) + assert groups[MODEL].rewrites == {ttl.FIRST: 1, ttl.ON_1H_CACHE: 1} + + +def test_previous_write_partly_on_1h_still_counts_as_possible_expiry(): + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 50000, write_1h=10000), + rec("2026-11-01T10:20:00Z", 0, 50000)]) + assert groups[MODEL].rewrites == {ttl.FIRST: 1, ttl.FIVE_TO_60M: 1} + + +# --- fingerprint ----------------------------------------------------------------- + +def test_fingerprint_ignores_cache_markers_but_not_content(): + base = rec("t", 0, 1) + moved = json.loads(json.dumps(base)) + moved["input"]["inputBodyJson"]["system"] = [{"text": "S", "cache_control": {"type": "ephemeral", "ttl": "1h"}}] + changed = rec("t", 0, 1, system="S plus today's date") + as_string = rec("t", 0, 1, body_as_string=True) + with_tools = rec("t", 0, 1, tools=[{"toolSpec": {"name": "x"}}]) + assert ttl.prompt_start(base) == ttl.prompt_start(moved) == ttl.prompt_start(as_string) + assert ttl.prompt_start(base) != ttl.prompt_start(changed) + assert ttl.prompt_start(base) != ttl.prompt_start(with_tools) + + +def test_schema_property_named_cache_control_is_part_of_the_fingerprint(): + a = rec("t", 0, 1, tools=[{"toolSpec": {"name": "x", "inputSchema": {"json": {"properties": { + "cache_control": {"type": "string"}}}}}}]) + b = rec("t", 0, 1, tools=[{"toolSpec": {"name": "x", "inputSchema": {"json": {"properties": {}}}}}]) + assert ttl.prompt_start(a) != ttl.prompt_start(b) + + +def test_claude_messages_body_is_fingerprinted(): + r = rec("t", 0, 1) + r["input"]["inputBodyJson"] = {"anthropic_version": "bedrock-2023-05-31", "system": "S", + "tools": [{"name": "x"}], "messages": [{"role": "user"}]} + assert ttl.prompt_start(r) is not None + + +@pytest.mark.parametrize("body", [None, "not json", {"no": "messages"}, ["list"]]) +def test_unusable_bodies_give_no_fingerprint(body): + r = rec("t", 0, 1, body=False) + r["input"]["inputBodyJson"] = body + assert ttl.prompt_start(r) is None + + +# --- records -------------------------------------------------------------------- + +def test_records_without_usage_or_timestamp_are_counted_not_guessed(): + no_time = rec("", 0, 1) + bad_time = rec("yesterday", 0, 1) + naive_time = rec("2026-11-01T10:00:00", 0, 1) # no timezone: can't be ordered against UTC records + no_usage = rec("2026-11-01T10:00:00Z", 0, 1) + no_usage["output"] = {"outputBodyS3Path": "s3://b/k"} + calls, skipped = ttl.collect([no_time, bad_time, naive_time, no_usage, {"some": "other file"}, + rec("2026-11-01T10:00:00.123+00:00", 0, 1)]) + assert len(calls) == 1 and skipped == 4 + + +def test_odd_record_shapes_do_not_crash(): + weird = [rec("2026-11-01T10:00:00Z", 0, 20000), rec("2026-11-01T10:01:00Z", 0, 20000)] + weird[0]["input"] = "x" + weird[1]["identity"] = {"arn": {"not": "a string"}} + weird[1]["input"].pop("inputBodyJson") + weird.append({**rec("2026-11-01T10:02:00Z", 0, 5), "identity": "role/c", "inferenceRegion": 7}) + calls, skipped = ttl.collect(weird) + assert len(calls) == 3 and skipped == 0 + ttl.check(calls) + + +# --- dollars: prices of what the logs show ---------------------------------------- + +def test_dollar_figures_by_hand(): + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000, uncached=0), + rec("2026-11-01T10:02:00Z", 20000, 1000, uncached=0), + rec("2026-11-01T10:20:00Z", 0, 30000, uncached=0, write_1h=10000)]) + g = groups[MODEL] + assert g.write_tokens == 51000 and g.write_tokens_1h == 10000 + assert g.cost_writes_now == pytest.approx((21000 * 3.75 + 20000 * 3.75 + 10000 * 6) / 1e6) + assert g.cost_writes_at_1h == pytest.approx(51000 * 6 / 1e6) + assert g.cost_5to60_now == pytest.approx((20000 * 3.75 + 10000 * 6) / 1e6) + assert g.cost_5to60_as_reads == pytest.approx(30000 * 0.30 / 1e6) + + +def test_arn_and_global_ids_are_priced_at_the_global_rate(): + arn = "arn:aws:bedrock:us-east-1:111122223333:inference-profile/" + MODEL + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000, model=arn)]) + assert groups[MODEL].cost_writes_now == pytest.approx(20000 * 3.75 / 1e6) + + +def test_regional_ids_cost_10_percent_more(): + regional = "us.anthropic.claude-sonnet-4-5-20250929-v1:0" + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000, model=regional)]) + assert groups[regional].cost_writes_now == pytest.approx(20000 * 3.75 * 1.1 / 1e6) + + +def test_unknown_models_are_not_priced(): + app = "arn:aws:bedrock:us-east-1:111122223333:application-inference-profile/abc" + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000, model=app)]) + assert not groups[app].priced + + +# --- CLI ------------------------------------------------------------------------- + +def test_cli_without_flag_is_unchanged(tmp_path, capsys): + path = write_file(tmp_path, [rec("2026-11-01T10:00:00Z", 0, 20000), rec("2026-11-01T10:20:00Z", 0, 20000)]) + assert cli.main(["logs", str(path)]) == 0 + assert "re-write" not in capsys.readouterr().out + assert cli.main(["--json", "logs", str(path)]) == 0 + assert set(json.loads(capsys.readouterr().out)) == {MODEL} # same shape as before: groups only + + +def test_cli_with_flag_text_and_json(tmp_path, capsys): + path = write_file(tmp_path, [rec("2026-11-01T10:00:00Z", 0, 30000), rec("2026-11-01T10:20:00Z", 0, 30000), + rec("2026-11-01T10:21:00Z", 30000, 300)]) + assert cli.main(["logs", str(path), "--ttl-check"]) == 0 + out = capsys.readouterr().out + assert "2 re-write(s) in 3 calls" in out + assert "5 to 60 minutes after the previous call that started the same way" in out + assert "The 5-to-60-minute re-writes cost $0.11 as cache writes" in out # 30,000 x $3.75/M + assert "the same tokens at the 1-hour write price would cost $0.36" in out # 60,300 x $6/M + assert "at least" not in out + assert cli.main(["--json", "logs", str(path), "--ttl-check"]) == 0 + data = json.loads(capsys.readouterr().out) + assert set(data) == {"groups", "notes", "ttl_check", "ttl_check_skipped"} + t = data["ttl_check"][MODEL] + assert t["rewrites"] == {ttl.FIRST: 1, ttl.FIVE_TO_60M: 1} + assert t["cache_writes_at_1h_rate_usd"] == pytest.approx(60300 * 6 / 1e6) + + +@pytest.mark.parametrize("model,show", [(MODEL, True), + ("us.anthropic.claude-3-7-sonnet-20250219-v1:0", False), + ("us.anthropic.claude-sonnet-4-20250514-v1:0", False)]) +def test_1h_price_line_only_for_models_with_documented_1h_cache(tmp_path, capsys, model, show): + path = write_file(tmp_path, [rec("2026-11-01T10:00:00Z", 0, 20000, model=model)]) + cli.main(["logs", str(path), "--ttl-check", "--price", "3"]) + out = capsys.readouterr().out + assert ("1-hour write price" in out) is show + assert ("not documented for this model" in out) is (not show) + + +def test_cli_body_wording_tells_s3_from_not_logged(tmp_path, capsys): + path = write_file(tmp_path, [rec("2026-11-01T10:00:00Z", 0, 20000, body=False), + rec("2026-11-01T10:20:00Z", 0, 20000, s3_body=True)]) + cli.main(["logs", str(path), "--ttl-check"]) + out = capsys.readouterr().out + assert "1 with no body logged" in out + assert "1 with the body stored as a separate S3 object (over 100 KB, or binary)" in out + + +def test_cli_flag_exit_code_still_follows_min_hit(tmp_path): + path = write_file(tmp_path, [rec("2026-11-01T10:00:00Z", 0, 20000)]) + assert cli.main(["logs", str(path), "--ttl-check", "--min-hit", "0.5"]) == 1 + assert cli.main(["logs", str(path), "--ttl-check"]) == 0 + + +def test_github_summary_unchanged_without_flag(tmp_path, capsys, monkeypatch): + summary = tmp_path / "summary.md" + monkeypatch.setenv("GITHUB_STEP_SUMMARY", str(summary)) + path = write_file(tmp_path, [rec("2026-11-01T10:00:00Z", 0, 20000)]) + assert cli.main(["--github", "logs", str(path)]) == 0 + assert "re-write" not in capsys.readouterr().out + assert "re-write" not in summary.read_text() + + +# --- round-3 review cases ---------------------------------------------------------- + +def test_first_means_no_earlier_cache_touching_call_not_first_call(): + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 0, uncached=20000), rec("2026-11-01T10:01:00Z", 0, 20000)]) + assert groups[MODEL].rewrites == {ttl.FIRST: 1} + assert ttl.LABELS[ttl.FIRST].startswith("no earlier call that started the same way read or wrote the cache") + + +def test_1h_label_follows_the_latest_write_even_when_the_previous_call_only_read(): + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000, write_1h=20000), + rec("2026-11-01T10:10:00Z", 20000, 0), + rec("2026-11-01T10:20:00Z", 0, 20000)]) + assert groups[MODEL].rewrites == {ttl.FIRST: 1, ttl.ON_1H_CACHE: 1} + assert "the latest cache write before it was all on the 1-hour cache" in ttl.LABELS[ttl.ON_1H_CACHE] + + +def test_within_5_minutes_region_is_reported_even_on_the_1h_cache(): + groups, _ = run([rec("2026-11-01T10:00:00Z", 0, 20000, write_1h=20000), + rec("2026-11-01T10:03:00Z", 0, 20000, region="us-east-2")]) + assert groups[MODEL].rewrites == {ttl.FIRST: 1, ttl.WITHIN_5M_OTHER_REGION: 1} + + +def test_unknown_model_says_no_1h_information(tmp_path, capsys): + path = write_file(tmp_path, [rec("2026-11-01T10:00:00Z", 0, 20000, model="anthropic.claude-future-9")]) + cli.main(["logs", str(path), "--ttl-check", "--price", "3"]) + out = capsys.readouterr().out + assert "CacheCanary has no 1-hour cache information for this model" in out + assert "not documented" not in out + + +def test_skipped_count_is_kept_when_nothing_could_be_checked(tmp_path, capsys): + path = write_file(tmp_path, [rec("", 0, 20000)]) + cli.main(["logs", str(path), "--ttl-check"]) + out = capsys.readouterr().out + assert "no calls with usage and a timestamp" in out + assert "Left out of the re-write check: 1 record(s)" in out + + +# --- real invocation logs --------------------------------------------------------- + +def test_real_eu_west_1_logs_nov_2026(): + """Real Bedrock invocation logs (S3 delivery, text data on), eu-west-1, November 4 2026, redacted: + account, request IDs and caller replaced; every text replaced by a placeholder, the same text by the + same placeholder. Calls: InvokeModel prompt A at 0, 2 and 8 min; Converse with tools prompt B at 0 and + 1 min and with a changed date line at 3 min; a ~130 KB Converse request (body stored in S3); InvokeModel + prompt D twice at 4 min; ConverseStream prompt S at 5 min.""" + from pathlib import Path + from cachecanary import logs + path = Path(__file__).parent / "fixtures" / "bedrock_invocation_logs_eu_ttl_2026-11.jsonl" + calls, skipped = ttl.collect(list(logs.iter_records(path))) + assert skipped == 0 and len(calls) == 10 + g = ttl.check(calls)["eu.anthropic.claude-sonnet-4-6"] + assert g.rewrites == {ttl.FIRST: 5, ttl.FIVE_TO_60M: 1, f"{ttl.WITHIN_5M}/caller": 1} + assert g.rewrite_tokens[ttl.FIVE_TO_60M] == 5511 + assert g.calls_body_in_s3 == 1 and g.calls_no_body == 0 + assert all(c.region == "eu-west-1" for c in calls) + assert sum(c.prompt_start is not None for c in calls) == 9