Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions packages/website/public/llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ New, but no longer unused. It has been run end to end against a real live paid e
- [Scoreboard](https://tokenizen.nl/en/scoreboard): how many of the findings that human auditors already found in closed contests our smart contract scan finds, one row per measurement, each linking to the rule written before the run and the result after it, with the sealed run and the contests after today still empty on purpose
- [Free x402 endpoint check](https://tokenizen.nl/en/x402-check): paste the URL of an x402 route you run and see, from one anonymous request that never pays, whether the payment request passes the official buyer schema and how far up the receipt ladder it gets. Nothing is stored
- [We measured our scan. It found 4 of 105.](https://tokenizen.nl/en/notes/we-measured-our-scan): the one run on 8 sealed audit contests, written down before it ran: production found 1 of 105 known findings, the new version 4, our own bar was 5 more and it found 3, so production stays as it is. The results page lists the misses.
- [Two free tools found none of the known findings on the same contests](https://tokenizen.nl/en/notes/free-tools-same-contests): Slither found 0 of 34 and Aderyn 0 of 70 on the same Code4rena contests, judged like our scan; static tools are not built for contest logic bugs; plan hashed before judging
- [A standard x402 client can now buy the scan, and a quick scan costs 1 dollar](https://tokenizen.nl/en/notes/quick-scan-standard-clients): the endpoint could not be bought by a standard client, fixed and checked with two real purchases on Base; the official client refuses payments above 1 dollar, so a quick scan costs exactly 1
- [The scan held its rate on nine newer contests](https://tokenizen.nl/en/notes/scan-newer-contests): on nine Code4rena contests from the last year the first 15 hypotheses held 5 of 66 known findings (7.6 percent), the first 30 held 8; one run, small numbers, rule written beforehand; reports can now show a second list
- [The scan signs its offer, and we paid it once to see the receipt](https://tokenizen.nl/en/notes/scan-signs-its-offer): the paid endpoint signs its x402 offer (level 2 of the open check), binds the signer in DNS, and returned a signed receipt on one real self-test payment on Base mainnet, with a script to check it
- [The scan now finds 8 of 105 on the same contests](https://tokenizen.nl/en/notes/scan-version-3-1): version 3.1 is live; on the 8 contests of the sealed run it found 8 of 105 known findings against 4 and 1 before; two changes at once, contests no longer secret, one run, 97 of 105 still not found
Expand Down
10 changes: 10 additions & 0 deletions packages/website/public/sitemap.xml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,16 @@
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
<url>
<loc>https://tokenizen.nl/en/notes/free-tools-same-contests</loc>
<changefreq>monthly</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc>https://tokenizen.nl/en/notes/quick-scan-standard-clients</loc>
<changefreq>monthly</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc>https://tokenizen.nl/en/notes/scan-newer-contests</loc>
<changefreq>monthly</changefreq>
Expand Down
63 changes: 63 additions & 0 deletions packages/website/src/data/blog-posts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,69 @@ Money is starting to move between machines. The people who build those machines
},
],
},
{
slug: "free-tools-same-contests",
kind: "note",
date: "2026-10-25",
title: "Two free tools found none of the known findings on the same contests",
dek: "We ran Slither and Aderyn with default settings on the contests where our scan was measured. Neither matched a known High or Medium finding. What that says and what it does not.",
description:
"Slither found 0 of 34 and Aderyn 0 of 70 known findings on the same Code4rena contests, judged like our scan. Static tools are not built for contest logic bugs.",
tags: ["Measurements", "Tools"],
readMinutes: 2,
blocks: [
{
type: "md",
text: `We wanted to know how our scan compares with free tools people already use, on the same contests and with the same two judges. We ran Slither 0.11.6 and Aderyn 0.6.8 with default settings and turned each finding into a hypothesis. The plan was written down with a hash before the judges saw anything.

Neither tool matched a known High or Medium finding. Slither found 0 of 34 on the contests where it ran, and Aderyn 0 of 70. On the same contests our scan found 6 of 34 and 10 of 70 in its first 15 hypotheses. On the four newer contests, which our scan was not built on, it was 3 of 15 against 0 of 15 for each tool.

## What this does not say

- Static tools are not built to find the business-logic flaws that audit contests reward. They are free and fast and find other things, such as patterns and missing checks. A finding of theirs can be a valid Low remark that is not on the list of High and Medium findings.
- The tools ran on only part of the contests in our setup. Slither ran on 8 of 15 and Aderyn on 10 of 15, and the reasons for the others are listed. A user with a working project setup would get results on more.
- The older contests may be in the training data of the model behind our scan; the tools do not have that. The judges are language models and this is our own measurement.
- Small numbers. Read it as a description, not a ranking.

## What you can check

The hypotheses as the judges saw them, every judgement, the list of runs that did not work and the plan file are in [the results folder](https://github.com/holistis/bug-bounty-intelligence-mcp/blob/main/results/2026-10-25-free-tools/README.md), and both tools have a row on the [scoreboard](https://tokenizen.nl/en/scoreboard). The same procedure can be applied to the output of any other tool on the same contests.`,
},
],
},
{
slug: "quick-scan-standard-clients",
kind: "note",
date: "2026-10-25",
title: "A standard x402 client can now buy the scan, and a quick scan costs 1 dollar",
dek: "The scan endpoint could not be bought by a standard x402 client. That is fixed and checked with two real purchases. The official client also refuses payments above 1 dollar, so there is now a quick scan for exactly that.",
description:
"A standard x402 client can now buy our scan, checked with real purchases on Base. The official client caps payments at 1 dollar, hence a quick scan for 1.",
tags: ["x402", "Tools"],
readMinutes: 3,
blocks: [
{
type: "md",
text: `Until today the paid scan endpoint accepted only a transaction hash from a payment you had made by hand. It answered 400 to the signed authorization that a standard x402 client such as @x402/fetch sends, although its own 402 answer said that would work. We found this by testing and fixed it.

## Checked with real purchases

A throwaway address bought the full scan (5 dollars) with @x402/fetch 2.28.0 and got a signed receipt back. The six checks on the receipt pass: the signature, the signer of the live offer, the DNS record that binds the signer to the domain, the transaction on Base, the USDC transfer to our payment address, and the payer. It was a self test with our own money, so it is not a customer.

## The 1 dollar cap

The official x402 client refuses a payment above 1 dollar per request unless its owner raises the limit. The scan costs 5 dollars, so a default agent will refuse it. There is now a quick scan for exactly 1 dollar that returns the 15 hypotheses the model is most confident about; the full scan adds 15 more with lower confidence. A second throwaway address bought the quick scan with the default limit left in place, and the same six checks pass.

## What this does not show

- Two purchases, both by us. Nobody else has used it yet.
- A receipt says that the service saw a payment. It says nothing about how good the report is.
- The hit rates we published (8 of 105 and 5 of 66) are for the first 15 hypotheses, so they are the numbers of the quick scan.

The transactions, the answers, the delivered reports and the scripts are in the [full scan folder](https://github.com/holistis/bug-bounty-intelligence-mcp/blob/main/results/2026-10-25-v31/standard-client-purchase/README.md) and the [quick scan folder](https://github.com/holistis/bug-bounty-intelligence-mcp/blob/main/results/2026-10-25-v31/quick-scan-purchase/README.md).`,
},
],
},
{
slug: "scan-newer-contests",
kind: "note",
Expand Down
28 changes: 28 additions & 0 deletions packages/website/src/data/scoreboard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,34 @@ export const SCORE_RUNS: ScoreRun[] = [
note: "The cleanest test, because no model can have seen the answers. The hash of a system's report is committed before the contest results are published, and scored afterwards with the same two judges. Takes weeks, since results arrive per contest.",
links: [{ label: "How the judges decide", href: `${REPO}/PREREGISTRATION-v3-2026-10-24.md` }],
},
{
id: "slither-same-contests",
date: "2026-10-25",
title: "Slither 0.11.6 on the same contests",
system: "Slither 0.11.6 with default settings, a free static analysis tool",
set: "Code4rena contests where it ran in our setup: 4 of 9 newer and 4 of 6 older, 34 known findings",
status: "measured",
known: 34,
found: 0,
hypotheses: 167,
tunedOn: "not applicable",
note: "Strict hits: none. Two hypotheses on one contest matched a known finding for one judge only. Static tools are not built for the business-logic flaws that contests reward, they ran on only part of the contests in our setup, and the judges are language models. On the same contests our scan found 6 of 34 in its first 15 hypotheses; the older contests may be in the training data of the model behind our scan.",
links: [{ label: "The results, judgements and hashes", href: `${REPO}/results/2026-10-25-free-tools/README.md` }],
},
{
id: "aderyn-same-contests",
date: "2026-10-25",
title: "Aderyn 0.6.8 on the same contests",
system: "Aderyn 0.6.8 with default settings, a free static analysis tool",
set: "Code4rena contests where it ran in our setup: 4 of 9 newer and 6 of 6 older, 70 known findings",
status: "measured",
known: 70,
found: 0,
hypotheses: 202,
tunedOn: "not applicable",
note: "Strict hits: none. Static tools are not built for the business-logic flaws that contests reward, they ran on only part of the contests in our setup, and the judges are language models. On the same contests our scan found 10 of 70 in its first 15 hypotheses; the older contests may be in the training data of the model behind our scan.",
links: [{ label: "The results, judgements and hashes", href: `${REPO}/results/2026-10-25-free-tools/README.md` }],
},
{
id: "c4-fresh-nine",
date: "2026-10-25",
Expand Down
Loading