Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/website/public/llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ New, but no longer unused. It has been run end to end against a real live paid e
- [Scoreboard](https://tokenizen.nl/en/scoreboard): how many of the findings that human auditors already found in closed contests our smart contract scan finds, one row per measurement, each linking to the rule written before the run and the result after it, with the sealed run and the contests after today still empty on purpose
- [Free x402 endpoint check](https://tokenizen.nl/en/x402-check): paste the URL of an x402 route you run and see, from one anonymous request that never pays, whether the payment request passes the official buyer schema and how far up the receipt ladder it gets. Nothing is stored
- [We measured our scan. It found 4 of 105.](https://tokenizen.nl/en/notes/we-measured-our-scan): the one run on 8 sealed audit contests, written down before it ran: production found 1 of 105 known findings, the new version 4, our own bar was 5 more and it found 3, so production stays as it is. The results page lists the misses.
- [The scan now finds 8 of 105 on the same contests](https://tokenizen.nl/en/notes/scan-version-3-1): version 3.1 is live; on the 8 contests of the sealed run it found 8 of 105 known findings against 4 and 1 before; two changes at once, contests no longer secret, one run, 97 of 105 still not found
- [Check. Pay. Prove.](https://tokenizen.nl/en/notes/check-pay-prove): one real AI-agent payment on Base mainnet, checked before, paid through x402 and proven from the chain, with the commands to repeat it, six numbers each with a place to check it, and the limits stated first
- [A signed receipt that cannot name its own payment](https://tokenizen.nl/en/notes/a-signed-receipt-that-cannot-name-its-own-payment): a live x402 receipt that verifies and still cannot say which payment it is about, the measured cause in the reference implementation, and the corrections that followed in public
- [733,203 attestations were taken back](https://tokenizen.nl/en/notes/733203-attestations-were-taken-back): a count of revocations across the three largest EAS indexers, showing that every revocation is an issuer withdrawing its own statement and the counterparty has no move at all
Expand Down
5 changes: 5 additions & 0 deletions packages/website/public/sitemap.xml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,11 @@
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
<url>
<loc>https://tokenizen.nl/en/notes/scan-version-3-1</loc>
<changefreq>monthly</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc>https://tokenizen.nl/en/notes/official-sources</loc>
<changefreq>monthly</changefreq>
Expand Down
36 changes: 35 additions & 1 deletion packages/website/src/data/blog-posts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,38 @@ Money is starting to move between machines. The people who build those machines
},
],
},
{
slug: "scan-version-3-1",
kind: "note",
date: "2026-10-25",
title: "The scan now finds 8 of 105 on the same contests",
dek: "A newer version of the scan is live. On the 8 contests of our sealed run it found 8 of 105 known findings, up from 4. What that means and what it does not.",
description:
"Version 3.1 of our scan found 8 of 105 known findings on the same 8 contests, against 4 and 1 before. Two changes at once, one run.",
tags: ["Measurements", "Tools"],
readMinutes: 2,
blocks: [
{
type: "md",
text: `On 24 October we published a run of our scan on 8 Sherlock contests nobody here had touched. The scan in production found 1 of 105 known High and Medium findings, a newer version 4. Since the evening of 25 October the newer version, V3.1, is what paid scans run on.

We ran it once on the same 8 contests. It found 8 of 105 (7.6 percent). Precision went from 5.0 to 7.5 percent. A scan takes about 20 to 45 seconds and costs us 0.02 to 0.15 dollar.

## What this does not say

- Two things changed at once, a newer model and a fix of which files count, so we cannot say which one did the work.
- Those contests are no longer secret, and the models may have seen them.
- It is one run, so a repeat could land anywhere from about 6 to 10.
- 97 of the 105 known findings were still not found. This is not a scan that finds what a human audit finds.

Before the run we wrote down, in our own notes, how we would read the result: 8 or more is a clear gain. It landed exactly on that line. Those notes are not public and carry no hash, so this is less verifiable than the sealed run.

## What you can check

The judgements of both judges, the hypotheses and the hashes are in [the results folder](https://github.com/holistis/bug-bounty-intelligence-mcp/blob/main/results/2026-10-25-v31/README.md), and the row is on the [scoreboard](https://tokenizen.nl/en/scoreboard). How the scan finds things stays private. The next test that no model can have seen is the contests that appear after today, scored when their results are published. The scoreboard has that row waiting.`,
},
],
},
{
slug: "official-sources",
kind: "note",
Expand Down Expand Up @@ -260,7 +292,9 @@ It also says nothing about how good the scan is. The scan returns AI-generated h
blocks: [
{
type: "md",
text: `Bug Bounty Intelligence reads the Solidity of a public repository and returns a list of hypotheses to check by hand. We sell it for 5 USDC through x402. It is not an audit.
text: `Update, 25 October: a newer version of the scan is live and found 8 of 105 on the same contests. See [the note](/notes/scan-version-3-1).

Bug Bounty Intelligence reads the Solidity of a public repository and returns a list of hypotheses to check by hand. We sell it for 5 USDC through x402. It is not an audit.

The first time we measured it, it found none of 40 known findings. We changed it, wrote down a test before running it, and ran that test once on 8 contests that nobody here had run anything on. The version in production found 1 of 105 known findings. The new version found 4. Our own bar was 5 more than production, and it found 3 more. We missed it, so production stays as it is.`,
},
Expand Down
18 changes: 16 additions & 2 deletions packages/website/src/data/scoreboard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ export interface ScoreRun {
links: RunLink[];
}

export const SCORE_UPDATED = "2026-10-24";
export const SCORE_UPDATED = "2026-10-25";

export const SEALED_SET_HASH = "de1a23a4aabc17def5145229c8ee4b4b216e25e8a5ec990e0c5bb37620b61ca1";
export const SEED_BLOCK = { number: 53022890, hash: "0x91b7371d9902f60b3515633ffda7f89f7537bcce9a132a62fef21694f54ddd1d" };
Expand All @@ -72,6 +72,20 @@ export const SCORE_RUNS: ScoreRun[] = [
note: "The cleanest test, because no model can have seen the answers. The hash of a system's report is committed before the contest results are published, and scored afterwards with the same two judges. Takes weeks, since results arrive per contest.",
links: [{ label: "How the judges decide", href: `${REPO}/PREREGISTRATION-v3-2026-10-24.md` }],
},
{
id: "v31-same-eight",
date: "2026-10-25",
title: "Version 3.1 on the same eight contests",
system: "Version 3.1 of the scan, live since the evening of 25 October",
set: "The 8 contests of the sealed run, 105 known findings. No longer secret: their results were published on 24 October",
status: "measured",
known: 105,
found: 8,
hypotheses: 120,
tunedOn: "no",
note: "6 of 59 easy findings, 1 of 14 middle, 1 of 32 hard. Precision 7.5 percent, about 20 to 45 seconds and 0.02 to 0.15 dollar per scan. Two things changed at once, a newer model and a fix of which files count, so we cannot say which one did the work. One run, so a repeat could land between about 6 and 10. The models may have seen these contests. 97 of 105 known findings were still not found. The rule for reading the result was written in our own notes before the run, which are not public, so this row is less verifiable than the sealed run.",
links: [{ label: "The results, judgements and hashes", href: `${REPO}/results/2026-10-25-v31/README.md` }],
},
{
id: "sealed-v3",
date: "2026-10-24",
Expand All @@ -94,7 +108,7 @@ export const SCORE_RUNS: ScoreRun[] = [
id: "sealed-s0",
date: "2026-10-24",
title: "The sealed run: the scan as it runs",
system: "Version 2.1 of the scan, the one in production, with the pattern block",
system: "Version 2.1 of the scan, the one in production until the evening of 25 October, with the pattern block",
set: "Sealed set: 8 contests, 105 known findings",
status: "measured",
known: 105,
Expand Down
Loading