diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 0000000..f291ce1 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,22 @@ +name: Something on the site is wrong +description: A wrong number, a dead link, or a page that does not render +labels: ["bug"] +body: + - type: input + id: page + attributes: + label: Which page + placeholder: https://dkautomation23.github.io/ucp-survey.html + validations: + required: true + - type: textarea + id: problem + attributes: + label: What is wrong + description: For a wrong number, say what it should be and how you know. + validations: + required: true + - type: input + id: browser + attributes: + label: Browser and screen width, if it is a layout problem diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..6b7337c --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,8 @@ +blank_issues_enabled: false +contact_links: + - name: Question, or paid work + url: mailto:hello@dkautomation.dev + about: Anything that is not a bug report or a feature request + - name: Security report + url: https://github.com/dkautomation23/.github/blob/main/SECURITY.md + about: Report a vulnerability privately, never in a public issue diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml new file mode 100644 index 0000000..2ef0372 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -0,0 +1,15 @@ +name: Suggest something +description: A page that would be worth having, or data worth publishing +labels: ["enhancement"] +body: + - type: textarea + id: idea + attributes: + label: What would you want to find here? + validations: + required: true + - type: textarea + id: why + attributes: + label: Who would use it, and for what? + description: This site publishes measurements, not opinions. diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..32c4dfc --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,7 @@ +version: 2 +updates: + - package-ecosystem: github-actions + directory: "/" + schedule: + interval: weekly + open-pull-requests-limit: 5 diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 0000000..11d3b7c --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,31 @@ +# GitHub's own static analysis, reading the paths the tests do not. +# +# Findings appear under Security -> Code scanning. A false positive is +# dismissed with a written reason, never by deleting the query. +name: CodeQL +on: + push: + branches: [main] + pull_request: + branches: [main] + schedule: + - cron: '0 6 * * 1' +permissions: + contents: read + security-events: write +jobs: + analyze: + name: analyze (${{ matrix.language }}) + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + language: ['javascript-typescript'] + steps: + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + - uses: github/codeql-action/init@1c5b675653bb5c22dbe9b12b556ec555138e09fd # v4 + with: + languages: ${{ matrix.language }} + queries: security-extended + - uses: github/codeql-action/autobuild@d8073367669608af8fbcc5f63dd0a0d52bb90cff # v4 + - uses: github/codeql-action/analyze@d8073367669608af8fbcc5f63dd0a0d52bb90cff # v4 diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..8df0083 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,83 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience +* Focusing on what is best not just for us as individuals, but for the overall community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or advances of any kind +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or email address, without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at hello@dkautomation.dev. All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series of actions. + +**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within the community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1]. + +Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder][Mozilla CoC]. + +For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at [https://www.contributor-covenant.org/translations][translations]. + +[homepage]: https://www.contributor-covenant.org +[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html +[Mozilla CoC]: https://github.com/mozilla/diversity +[FAQ]: https://www.contributor-covenant.org/faq +[translations]: https://www.contributor-covenant.org/translations diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..6ee73b0 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,26 @@ +# Contributing + +This is a personal site, so pull requests that change its wording or its claims +will usually be declined. Two kinds of change are welcome: + +- **A factual correction.** If a number, a date or a claim on the site is + wrong, open an issue with the evidence. Published numbers are meant to be + checkable; being wrong in public is worse than being corrected. +- **A broken page.** Layout that fails at some width, a dead link, a page that + does not render. + +## Working on it + +No dependencies to install. Node 22 or newer: + +```console +$ node build.mjs +$ node survey.mjs ../ucp-audit/survey/2026-09-18-aggregate.json +$ python -m http.server 8765 # then open http://127.0.0.1:8765 +``` + +Generated files (`tools/*.html`, `ucp-survey.html`, `sitemap.xml`) are committed, +so run the generator and commit its output rather than editing the result. + +Never hand-edit a number on a page. It comes from the aggregate file the audit +wrote; if it is wrong, the run is wrong, and that is what gets fixed. diff --git a/README.md b/README.md new file mode 100644 index 0000000..1147683 --- /dev/null +++ b/README.md @@ -0,0 +1,34 @@ +# dkautomation23.github.io + +The site at : a page per tool, plus the +open-data pages. + +Everything is static and generated by two scripts with Node's standard library +and nothing else. + +## Building it + +```console +$ node build.mjs # a page per tool, plus sitemap.xml +$ node survey.mjs ../ucp-audit/survey/-aggregate.json # the survey page +``` + +`build.mjs` reads `tools.json`. `survey.mjs` reads the aggregate a real audit +run wrote, so a number that is not in that file cannot appear on the page. + +## What is where + +| Path | What it is | +|---|---| +| `index.html` | the front page, edited by hand | +| `tools.json` | one entry per tool; `build.mjs` turns each into a page | +| `survey.mjs` | builds `ucp-survey.html` from measured data | +| `style.css` | the whole stylesheet | +| `sitemap.xml`, `robots.txt` | generated and committed | + +## Rules this site keeps + +- No tracking, no analytics, no cookies. There is nothing to consent to. +- Every published number names its method, its date and the script that + produced it. +- Shop names from the survey never reach this repository — only totals. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..29b341d --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,26 @@ +# Security policy + +This repository is a static site. There is no server, no database and no user +input: GitHub Pages serves files from the default branch. + +## Reporting + +**Security → Report a vulnerability** on this repository, or +**hello@dkautomation.dev**. First reply within 2 working days. + +Do not open a public issue for a security problem. + +## In scope + +- Content injection: anything that makes a published page execute code or load + a resource the repository does not contain. +- A page or a data file that exposes personal data. The survey publishes totals + only, and a shop name appearing anywhere on the site is a bug worth reporting. +- A link or a redirect that sends a visitor somewhere other than where it says. + +## Out of scope + +- Missing security headers that GitHub Pages does not let a site set. +- The absence of a Content Security Policy header, for the same reason; the + pages load no third-party script. +- Findings about GitHub Pages itself — report those to GitHub. diff --git a/build.mjs b/build.mjs index a6be3a7..e2d26a5 100644 --- a/build.mjs +++ b/build.mjs @@ -136,6 +136,7 @@ const urls = [ // The survey is the only page here that is data rather than a tool, and the // one most likely to be linked to from elsewhere. { loc: `${SITE}/ucp-survey.html`, priority: "0.9" }, + { loc: `${SITE}/security-txt-survey.html`, priority: "0.9" }, ...tools.map((tool) => ({ loc: `${SITE}/tools/${tool.slug}.html`, priority: "0.8" })), ]; writeFileSync( diff --git a/index.html b/index.html index 4ee2939..b614430 100644 --- a/index.html +++ b/index.html @@ -45,6 +45,12 @@

Dmytro Galko

Open data — what AI shopping agents actually see

+
+

Half the security.txt files at the top of the web are invalid

+

A security.txt tells a researcher where to report a hole in your site, and RFC 9116 makes its Expires field mandatory — an expired file is, in the words of the specification, as good as no file. Of the 158 such files among the 500 most visited sites, 92 carry no Expires at all and 6 have expired: 62% are not valid. Nothing warns when that happens.

+

500 sites2026-09-18raw results publishedrepeatable in one command

+
+

A census of 5,356 live Shopify storefronts

Google and Shopify published the Universal Commerce Protocol so an agent can read a shop and buy from it. Nobody had published a count of how many shops actually speak it, so I measured it: 98.9% publish a valid profile, every one of them on the current release, none with a blocker. The only thing they differ on is whether an agent may sign the shopper into their own account — and 47.5% do not allow it, which turns a returning customer into an anonymous visitor.

@@ -60,25 +66,25 @@

Command-line tools — Rust

csvdiff

Diffs two CSV exports by key and reports what actually changed. Two million rows in 2.8 seconds, a per-column breakdown with counted transitions, and number/null normalisation — so a migration report reads as 9 809 real changes instead of 995 000 rows of timestamp noise.

-

Rust13 testsbenchmark in repoprebuilt binaries

+

Rust16 testsbenchmark in repoprebuilt binaries

leakscan

Offline check of what leaves with a file before you send it: tracked changes with the deleted text still readable, review comments, hidden worksheets, PDF attachments, GPS coordinates in photos, validated card and IBAN numbers.

-

Rust38 testsno network at any point

+

Rust41 testsno network at any point

imgdupe

Perceptual-hash duplicate finder. Complete-linkage grouping, so unrelated pictures never chain together through a path of small steps — and it writes a delete script for you to read instead of deleting anything itself.

-

Rust19 testsbenchmark in repo

+

Rust23 testsbenchmark in repo

linkscan

Crawls a site and reports the links that are genuinely broken, keeping bot protection and slow third parties out of the "broken" column — which is why the report still gets read after the second run.

-

Rust15 testsexit code for CI

+

Rust28 testsexit code for CI

@@ -90,13 +96,19 @@

Integration tooling — TypeScript

webhook-rewind

Record the webhooks a provider sends you once, then replay them at your own code as often as you need — re-signed for Meta, GitHub, Shopify or Stripe, so the receiver runs its real verification instead of having it switched off for the debugging session.

-

TypeScript34 testsno runtime dependencies

+

TypeScript35 testsno runtime dependencies

+
+ +
+

well-known-audit

+

A site publishes a dozen small files at its root — security.txt, robots.txt, llms.txt, assetlinks.json, mta-sts.txt — each checked by a different tool or by nobody. This reads all of them in one run and says which are missing, malformed, or quietly expired. RFC 9116 gives security.txt an Expires date; past it, the file is no longer valid, and almost nobody notices.

+

TypeScript106 testsno runtime dependenciesrefuses private addresses

ucp-audit

Google and Shopify's Universal Commerce Protocol lets AI agents shop: every business publishes a profile at /.well-known/ucp and agents read it to decide what they can do. The failure mode is silence — offer checkout without declaring a searchable catalogue and no agent ever surfaces your products. This reads the profile the way an agent does.

-

TypeScript53 testsno runtime dependenciesrefuses private addresses

+

TypeScript59 testsno runtime dependenciesrefuses private addresses

@@ -108,7 +120,7 @@

cra-report

api-drift

Records the shape of a JSON API and reports when it changes: removed fields, changed types, values that can now be null. Exits non-zero in CI, so a partner's silent rename stops being something you find out about two weeks later.

-

TypeScript29 testsruns in cron or CI

+

TypeScript31 testsruns in cron or CI

diff --git a/security-txt-survey.html b/security-txt-survey.html new file mode 100644 index 0000000..ea09c22 --- /dev/null +++ b/security-txt-survey.html @@ -0,0 +1,141 @@ + + + + + +Half the security.txt files at the top of the web are invalid + + + + + + + + + + + + + + + + +
+
+

Dmytro Galko · open data

+

Where to report a hole, and nobody watching

+

security.txt across the 500 most visited sites · 2026-09-18

+

+ A security.txt file tells a researcher who found a hole in + your site where to send it. RFC 9116 + makes one field mandatory — Expires — precisely because a + contact address nobody has confirmed in three years is worse than none: + the specification treats an expired file as invalid, exactly like a + missing one. +

+

+ Nothing breaks when that date passes. No browser warns, no scanner + complains, no dashboard turns red. So this is a count of how often it has + already happened at the top of the web. +

+
+ The tool and the raw data + How it was measured +
+
+
+ +
+
+

Of the 158 files that exist

+ + + + + + +
Valid today6038.0%
No Expires field at all9258.2%
Expired63.8%
+

+ 98 of 158 — 62.0% — are not valid files under the + specification they are written to. The commonest failure is not an expired + date but no date at all: the field was never added, and nothing ever asked + for it. +

+
+
+ +
+
+

Who publishes one in the first place

+ + + + + + +
Sites that answered33266.4%
Publish a security.txt15847.6%
Never answered16833.6%
+

+ The 168 that never answered are counted here and nowhere else. A ranking of + the most visited domains is full of content-delivery and API hostnames + that serve nothing at their root; calling those "sites without a + security.txt" would be a made-up number. +

+
+
+ +
+
+

And the rest of what sites publish at their root

+ + + + + + + + + + + + + + + +
robots.txt
what crawlers may read		23370.2%
security.txt
where to report a hole		15847.6%
assetlinks.json
Android app links		13841.6%
apple-app-site-association
iOS universal links		12738.3%
change-password
where a password manager sends you		7723.2%
llms.txt
what an AI assistant should read		7021.1%
gpc.json
Global Privacy Control		216.3%
openid-configuration
OIDC discovery		195.7%
mta-sts.txt
mail transport security		61.8%
ai.txt
what AI crawlers may use		41.2%
ucp
Universal Commerce Protocol profile		41.2%
dnt-policy.txt
Do Not Track policy		41.2%
+
+
+ +
+
+

How it was measured

+
    +
  • Population. The Tranco top 500, taken as it is — no filtering by kind of site.
    • +
  • Measurement. One HTTPS GET per file, four at a time for a single domain, one domain at a time, with a pause and a user agent naming the tool. Both locations RFC 9116 allows are checked.
    • +
  • Date. 2026-09-18.
    • +
  • Tool. well-known-audit — open source, zero runtime dependencies, tests on recorded responses.
    • +
  • Checked by hand before publishing. Four of the flagged sites were confirmed with curl, because a number like this is only worth having if it is right.
    • +
  • Repeat it. well-known-audit --batch survey/domains-top500.txt --csv out.csv rebuilds every table above.
    • +
+

What this does not say

+
    +
  • An invalid security.txt does not mean a site is insecure. It means the one document a researcher reads first cannot be relied on.
    • +
  • This was run from one machine. A site that serves different files by region, or blocks an unknown user agent, is counted as unverified rather than guessed at.
    • +
  • No site is named on this page. The per-domain results are in the repository for anyone who wants to check their own.
    • +
+

+ Want to know what your own domains publish? npx well-known-audit yourdomain.com + once it is released, or write to hello@dkautomation.dev. +

+
+
+ + + + + diff --git a/securitytxt.mjs b/securitytxt.mjs new file mode 100644 index 0000000..b21fa6e --- /dev/null +++ b/securitytxt.mjs @@ -0,0 +1,225 @@ +#!/usr/bin/env node +/** + * Builds /security-txt-survey.html from the aggregate a real run produced. + * + * Same contract as survey.mjs: every number is read out of the file the audit + * wrote, so a figure that is not in that file cannot appear on the page, and no + * site name ever reaches the site - the aggregate carries totals only. + * + * node securitytxt.mjs ../well-known-audit/survey/2026-09-18-top500-aggregate.json + */ + +import { readFileSync, writeFileSync } from "node:fs"; + +const HERE = import.meta.dirname; +const SITE = "https://dkautomation23.github.io"; +const GITHUB = "https://github.com/dkautomation23"; +const AUTHOR = "Dmytro Galko"; +const EMAIL = "hello@dkautomation.dev"; +const REPO = `${GITHUB}/well-known-audit`; + +const source = process.argv[2]; +if (!source) { + process.stderr.write("usage: node securitytxt.mjs \n"); + process.exit(2); +} + +const run = JSON.parse(readFileSync(source, "utf8")); +const s = run.summary; +const sec = s.securityTxt; +const date = (run.checkedAt ?? "").slice(0, 10); + +const escape = (text) => + String(text).replace(/&/g, "&").replace(//g, ">").replace(/"/g, """); +const pct = (part, whole) => (whole === 0 ? "—" : `${((part / whole) * 100).toFixed(1)}%`); +const num = (value) => Number(value).toLocaleString("en-US"); + +const invalid = sec.withoutExpires + sec.expired; +const headline = `${pct(invalid, sec.published)} of the security.txt files at the top of the web are not valid under RFC 9116`; + +const FILE_LABEL = { + "robots.txt": "what crawlers may read", + "security.txt": "where to report a hole", + "llms.txt": "what an AI assistant should read", + "ai.txt": "what AI crawlers may use", + "assetlinks.json": "Android app links", + "apple-app-site-association": "iOS universal links", + "change-password": "where a password manager sends you", + "openid-configuration": "OIDC discovery", + "mta-sts.txt": "mail transport security", + "gpc.json": "Global Privacy Control", + "dnt-policy.txt": "Do Not Track policy", + ucp: "Universal Commerce Protocol profile", +}; + +const files = Object.entries(s.filePresence).sort((a, b) => b[1] - a[1]); + +const jsonLd = { + "@context": "https://schema.org", + "@type": "Dataset", + name: "security.txt validity across the most visited sites", + description: + `A census of the ${num(s.total)} most visited sites, taken on ${date}: how many publish a ` + + "security.txt, how many of those carry the Expires field RFC 9116 requires, and how many have expired.", + url: `${SITE}/security-txt-survey.html`, + license: "https://opensource.org/licenses/MIT", + creator: { "@type": "Person", name: AUTHOR, url: SITE, email: EMAIL }, + dateCreated: date, + temporalCoverage: date, + isAccessibleForFree: true, + variableMeasured: ["security.txt published", "Expires field present", "Expires date in the past"], + keywords: [ + "security.txt", "RFC 9116", "Expires", "vulnerability disclosure", + "well-known files", "site audit", "open data", + ].join(", "), + distribution: [ + { + "@type": "DataDownload", + name: "Per-domain results", + encodingFormat: "text/csv", + contentUrl: `${REPO}/blob/main/survey/${date}-top500.csv`, + }, + ], + measurementTechnique: + "One HTTPS GET per file, four at a time per domain, one domain at a time. Domains that never answered are reported separately and never counted as publishing nothing.", +}; + +const html = ` + + + + +Half the security.txt files at the top of the web are invalid + + + + + + + + + + + + + + + + +
+
+

${escape(AUTHOR)} · open data

+

Where to report a hole, and nobody watching

+

security.txt across the ${num(s.total)} most visited sites · ${date}

+

+ A security.txt file tells a researcher who found a hole in + your site where to send it. RFC 9116 + makes one field mandatory — Expires — precisely because a + contact address nobody has confirmed in three years is worse than none: + the specification treats an expired file as invalid, exactly like a + missing one. +

+

+ Nothing breaks when that date passes. No browser warns, no scanner + complains, no dashboard turns red. So this is a count of how often it has + already happened at the top of the web. +

+
+ The tool and the raw data + How it was measured +
+
+
+ +
+
+

Of the ${num(sec.published)} files that exist

+ + + + + + +
Valid today${num(sec.valid)}${pct(sec.valid, sec.published)}
No Expires field at all${num(sec.withoutExpires)}${pct(sec.withoutExpires, sec.published)}
Expired${num(sec.expired)}${pct(sec.expired, sec.published)}
+

+ ${num(invalid)} of ${num(sec.published)} — ${pct(invalid, sec.published)} — are not valid files under the + specification they are written to. The commonest failure is not an expired + date but no date at all: the field was never added, and nothing ever asked + for it. +

+
+
+ +
+
+

Who publishes one in the first place

+ + + + + + +
Sites that answered${num(s.checked)}${pct(s.checked, s.total)}
Publish a security.txt${num(sec.published)}${pct(sec.published, s.checked)}
Never answered${num(s.unreachable)}${pct(s.unreachable, s.total)}
+

+ The ${num(s.unreachable)} that never answered are counted here and nowhere else. A ranking of + the most visited domains is full of content-delivery and API hostnames + that serve nothing at their root; calling those "sites without a + security.txt" would be a made-up number. +

+
+
+ +
+
+

And the rest of what sites publish at their root

+ + +${files + .map( + ([file, count]) => + ` `, + ) + .join("\n")} + +
${escape(file)}
${escape(FILE_LABEL[file] ?? "")}		${num(count)}${pct(count, s.checked)}
+
+
+ +
+
+

How it was measured

+
    +
  • Population. The Tranco top ${num(s.total)}, taken as it is — no filtering by kind of site.
    • +
  • Measurement. One HTTPS GET per file, four at a time for a single domain, one domain at a time, with a pause and a user agent naming the tool. Both locations RFC 9116 allows are checked.
    • +
  • Date. ${date}.
    • +
  • Tool. well-known-audit — open source, zero runtime dependencies, tests on recorded responses.
    • +
  • Checked by hand before publishing. Four of the flagged sites were confirmed with curl, because a number like this is only worth having if it is right.
    • +
  • Repeat it. well-known-audit --batch survey/domains-top500.txt --csv out.csv rebuilds every table above.
    • +
+

What this does not say

+
    +
  • An invalid security.txt does not mean a site is insecure. It means the one document a researcher reads first cannot be relied on.
    • +
  • This was run from one machine. A site that serves different files by region, or blocks an unknown user agent, is counted as unverified rather than guessed at.
    • +
  • No site is named on this page. The per-domain results are in the repository for anyone who wants to check their own.
    • +
+

+ Want to know what your own domains publish? npx well-known-audit yourdomain.com + once it is released, or write to ${EMAIL}. +

+
+
+ + + + + +`; + +writeFileSync(`${HERE}/security-txt-survey.html`, html, "utf8"); +process.stdout.write(`security-txt-survey.html (${num(sec.published)} files, ${num(invalid)} invalid)\n`); diff --git a/sitemap.xml b/sitemap.xml index 5d0b510..392a112 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -10,6 +10,11 @@ 2026-09-18 0.9 + + https://dkautomation23.github.io/security-txt-survey.html + 2026-09-18 + 0.9 + https://dkautomation23.github.io/tools/ucp-audit.html 2026-09-18 @@ -20,6 +25,11 @@ 2026-09-18 0.8 + + https://dkautomation23.github.io/tools/well-known-audit.html + 2026-09-18 + 0.8 + https://dkautomation23.github.io/tools/webhook-rewind.html 2026-09-18 diff --git a/survey.mjs b/survey.mjs index ecbcf85..0c2e33f 100644 --- a/survey.mjs +++ b/survey.mjs @@ -23,11 +23,16 @@ const REPO = `${GITHUB}/ucp-audit`; const source = process.argv[2]; if (!source) { - process.stderr.write("usage: node survey.mjs \n"); + process.stderr.write("usage: node survey.mjs [census-aggregate.json]\n"); process.exit(2); } +// The second measurement is optional: the page renders without it, and the +// first one keeps its own date rather than being rewritten. +const censusSource = process.argv[3]; + const run = JSON.parse(readFileSync(source, "utf8")); +const census = censusSource ? JSON.parse(readFileSync(censusSource, "utf8")) : null; const s = run.summary; const date = (run.checkedAt ?? "").slice(0, 10); @@ -278,7 +283,57 @@ ${blockers ` } -
+${ + !census + ? "" + : (() => { + const c = census.summary; + const answered = c.total - c.byOutcome.unreachable - (c.byOutcome.blocked ?? 0); + const date = (census.checkedAt ?? "").slice(0, 10); + return `
+
+

And outside Shopify: ${num(c.checked)} of ${num(answered)}

+

+ The measurement above covers one platform, which was the honest limit of + it. So the same question was put to the web at large: of the + ${num(c.total)} most visited sites in the world, asked on ${date}, how many + publish a Universal Commerce Protocol profile at all? No commerce + classifier is involved - publishing one is itself the signal, and every + retailer large enough to matter to a shopping agent is inside that range. +

+ + + + + + +
Published a profile${num(c.checked)}${pct(c.checked, answered)}
Answered, no profile${num(c.byOutcome["no-profile"] ?? 0)}${pct(c.byOutcome["no-profile"] ?? 0, answered)}
Answered with something that is not a profile${num((c.byOutcome["not-json"] ?? 0) + (c.byOutcome.invalid ?? 0))}${pct((c.byOutcome["not-json"] ?? 0) + (c.byOutcome.invalid ?? 0), answered)}
+

+ ${num(c.byOutcome.unreachable + (c.byOutcome.blocked ?? 0))} of the ${num(c.total)} did not answer or refused the + request, and are excluded from every percentage above rather than counted + as "no profile". Many of them are not websites at all - the list of most + visited domains is full of content-delivery and API hostnames that serve + nothing at their root - and the rest were asked with a ${census.method.timeouts.split(";")[0].replace("s on the first pass", "-second")} limit, + which a slow site can miss - so every one of them was asked again with a + longer limit, and ${num(census.retried ?? 0)} answered on that second pass. What still did not + answer is counted as unverified rather than quietly folded into a nicer + number. +

+ ${census.finding ? `

${escape(census.finding)}

` : ""} +

+ Taken together, the two measurements say something neither says alone: + inside Shopify the protocol is universal, and outside it, at the top of + the web, it is not there yet. An agent's shopping world is currently the + shops one company switched on. +

+
+
+ +
`; + })() +} + +

How it was measured

    diff --git a/tools.json b/tools.json index 3f070cf..5c9abaf 100644 --- a/tools.json +++ b/tools.json @@ -14,7 +14,7 @@ "proof": "Run against chewy.com on the day it was written: checkout, fulfilment and order declared, neither catalogue capability, protocol seven months behind. allbirds.com the same day: eight capabilities on the current version. Both profiles are recorded in the repository as test fixtures.", "stack": [ "TypeScript", - "53 tests", + "59 tests", "no runtime dependencies", "refuses private addresses" ], @@ -58,6 +58,39 @@ "devsecops" ] }, + { + "slug": "well-known-audit", + "name": "well-known-audit", + "tagline": "Every file your site publishes at its root, checked at once", + "search": "security.txt expired check, RFC 9116 Expires validator, well-known files audit, llms.txt robots.txt assetlinks checker", + "summary": "A site publishes a dozen small files at its root and under /.well-known/ that nobody owns: security.txt, robots.txt, llms.txt, assetlinks.json, apple-app-site-association, mta-sts.txt and the rest. Each one is checked by a different tool, or by nobody. This reads all of them in one run and says which are missing, which are malformed, and which have quietly expired - starting with security.txt, whose Expires field makes the file formally invalid once the date passes.", + "how": [ + "Reads twelve well-known and root files in one pass, four requests at a time, and reports them together.", + "Validates security.txt against RFC 9116: a Contact line is required, an Expires line is required, and a date in the past means the file is no longer valid - it counts the days either way.", + "Catches the common silent failure: apple-app-site-association or assetlinks.json served as HTML, which iOS and Android quietly ignore.", + "Refuses private and loopback addresses, checks every redirect hop rather than the first URL, and stops reading a body at 5 MB.", + "Audits a list of domains in one run and writes the totals as CSV, one polite domain at a time." + ], + "proof": "Run against the Tranco top 500 on 2026-09-18: of the 158 sites that publish a security.txt, 92 have no Expires field at all and 6 have expired - nearly two thirds are invalid under RFC 9116. Four of the flagged sites were checked by hand with curl before the number was published. 106 tests, none of which touch a network.", + "stack": [ + "TypeScript", + "99 tests", + "no runtime dependencies", + "refuses private addresses" + ], + "keywords": [ + "security.txt", + "RFC 9116", + "well-known", + "llms.txt", + "robots.txt", + "assetlinks.json", + "apple-app-site-association", + "mta-sts", + "site audit", + "cli" + ] + }, { "slug": "webhook-rewind", "name": "webhook-rewind", @@ -70,10 +103,10 @@ "Gives Stripe a fresh timestamp, because its tolerance window rejects the recorded one.", "Redacts Authorization, Cookie and API-key headers before anything reaches disk." ], - "proof": "Every signing scheme is checked against an independent implementation of the verifying side, not against a frozen digest. 34 tests, none of which touch a network.", + "proof": "Every signing scheme is checked against an independent implementation of the verifying side, not against a frozen digest. 35 tests, none of which touch a network.", "stack": [ "TypeScript", - "34 tests", + "35 tests", "no runtime dependencies" ], "keywords": [ @@ -101,7 +134,7 @@ "proof": "Reproducible from two fixtures in the repository: it reports a rename as its two halves, a string that became a number, and a value that can now be null.", "stack": [ "TypeScript", - "29 tests", + "31 tests", "runs in cron or CI" ], "keywords": [ @@ -127,7 +160,7 @@ "proof": "The benchmark is a script in the repository: it builds both exports from a fixed seed, so the timing repeats on any machine.", "stack": [ "Rust", - "13 tests", + "16 tests", "prebuilt binaries for Linux, macOS and Windows" ], "keywords": [ @@ -149,7 +182,7 @@ "Validates card and IBAN numbers rather than matching digit patterns, which is the difference between a finding and noise.", "Reports what it found and where, and changes nothing." ], - "proof": "38 tests across office documents, PDFs and images.", + "proof": "41 tests across office documents, PDFs and images.", "stack": [ "Rust", "38 tests", diff --git a/tools/api-drift.html b/tools/api-drift.html index 5734d4b..8f5dc9d 100644 --- a/tools/api-drift.html +++ b/tools/api-drift.html @@ -49,7 +49,7 @@

    What it does

    How it is proved

    Reproducible from two fixtures in the repository: it reports a rename as its two halves, a string that became a number, and a value that can now be null.

    -

    TypeScript29 testsruns in cron or CI

    +

    TypeScript31 testsruns in cron or CI

diff --git a/tools/csvdiff.html b/tools/csvdiff.html index 3df8d6e..d4b157d 100644 --- a/tools/csvdiff.html +++ b/tools/csvdiff.html @@ -49,7 +49,7 @@

What it does

How it is proved

The benchmark is a script in the repository: it builds both exports from a fixed seed, so the timing repeats on any machine.

-

Rust13 testsprebuilt binaries for Linux, macOS and Windows

+

Rust16 testsprebuilt binaries for Linux, macOS and Windows

diff --git a/tools/leakscan.html b/tools/leakscan.html index d46315e..d31d836 100644 --- a/tools/leakscan.html +++ b/tools/leakscan.html @@ -47,7 +47,7 @@

What it does

How it is proved

-

38 tests across office documents, PDFs and images.

+

41 tests across office documents, PDFs and images.

Rust38 testsno networkprebuilt binaries

diff --git a/tools/ucp-audit.html b/tools/ucp-audit.html index e005c33..1002fe1 100644 --- a/tools/ucp-audit.html +++ b/tools/ucp-audit.html @@ -49,7 +49,7 @@

What it does

How it is proved

Run against chewy.com on the day it was written: checkout, fulfilment and order declared, neither catalogue capability, protocol seven months behind. allbirds.com the same day: eight capabilities on the current version. Both profiles are recorded in the repository as test fixtures.

-

TypeScript53 testsno runtime dependenciesrefuses private addresses

+

TypeScript59 testsno runtime dependenciesrefuses private addresses

diff --git a/tools/webhook-rewind.html b/tools/webhook-rewind.html index ae46130..2775ba8 100644 --- a/tools/webhook-rewind.html +++ b/tools/webhook-rewind.html @@ -48,8 +48,8 @@

What it does

How it is proved

-

Every signing scheme is checked against an independent implementation of the verifying side, not against a frozen digest. 34 tests, none of which touch a network.

-

TypeScript34 testsno runtime dependencies

+

Every signing scheme is checked against an independent implementation of the verifying side, not against a frozen digest. 35 tests, none of which touch a network.

+

TypeScript35 testsno runtime dependencies

diff --git a/tools/well-known-audit.html b/tools/well-known-audit.html new file mode 100644 index 0000000..9e51b0f --- /dev/null +++ b/tools/well-known-audit.html @@ -0,0 +1,78 @@ + + + + + +well-known-audit — Every file your site publishes at its root, checked at once + + + + + + + + + + + + + + + + +
+
+

Dmytro Galko · open-source tools

+

well-known-audit

+

Every file your site publishes at its root, checked at once

+

A site publishes a dozen small files at its root and under /.well-known/ that nobody owns: security.txt, robots.txt, llms.txt, assetlinks.json, apple-app-site-association, mta-sts.txt and the rest. Each one is checked by a different tool, or by nobody. This reads all of them in one run and says which are missing, which are malformed, and which have quietly expired - starting with security.txt, whose Expires field makes the file formally invalid once the date passes.

+
+ Source on GitHub + Work with me +
+
+
+ +
+
+

What it does

+
    +
  • Reads twelve well-known and root files in one pass, four requests at a time, and reports them together.
    • +
  • Validates security.txt against RFC 9116: a Contact line is required, an Expires line is required, and a date in the past means the file is no longer valid - it counts the days either way.
    • +
  • Catches the common silent failure: apple-app-site-association or assetlinks.json served as HTML, which iOS and Android quietly ignore.
    • +
  • Refuses private and loopback addresses, checks every redirect hop rather than the first URL, and stops reading a body at 5 MB.
    • +
  • Audits a list of domains in one run and writes the totals as CSV, one polite domain at a time.
    • +
+
+
+ +
+
+

How it is proved

+

Run against the Tranco top 500 on 2026-09-18: of the 158 sites that publish a security.txt, 92 have no Expires field at all and 6 have expired - nearly two thirds are invalid under RFC 9116. Four of the flagged sites were checked by hand with curl before the number was published. 106 tests, none of which touch a network.

+

TypeScript99 testsno runtime dependenciesrefuses private addresses

+
+
+ +
+
+

People find this looking for

+

security.txt expired check, RFC 9116 Expires validator, well-known files audit, llms.txt robots.txt assetlinks checker.

+

+ If that is your week and you would rather someone else did it, write to + hello@dkautomation.dev + or open an issue. +

+
+
+ + + + + diff --git a/ucp-survey.html b/ucp-survey.html index 4e5bd8e..dcaa8da 100644 --- a/ucp-survey.html +++ b/ucp-survey.html @@ -144,7 +144,48 @@

Blockers found: none

+
+
+

And outside Shopify: 1 of 1,154

+

+ The measurement above covers one platform, which was the honest limit of + it. So the same question was put to the web at large: of the + 2,000 most visited sites in the world, asked on 2026-09-18, how many + publish a Universal Commerce Protocol profile at all? No commerce + classifier is involved - publishing one is itself the signal, and every + retailer large enough to matter to a shopping agent is inside that range. +

+ + + + + + +
Published a profile10.1%
Answered, no profile93981.4%
Answered with something that is not a profile21418.5%
+

+ 846 of the 2,000 did not answer or refused the + request, and are excluded from every percentage above rather than counted + as "no profile". Many of them are not websites at all - the list of most + visited domains is full of content-delivery and API hostnames that serve + nothing at their root - and the rest were asked with a 2.5-second limit, + which a slow site can miss - so every one of them was asked again with a + longer limit, and 668 answered on that second pass. What still did not + answer is counted as unverified rather than quietly folded into a nicer + number. +

+

The one domain that publishes a profile serves it from wyzecom.myshopify.com - a Shopify storefront. Outside that one platform the protocol is not present at the top of the web.

+

+ Taken together, the two measurements say something neither says alone: + inside Shopify the protocol is universal, and outside it, at the top of + the web, it is not there yet. An agent's shopping world is currently the + shops one company switched on. +

+
+
+
+ +

How it was measured