diff --git a/CHANGELOG.md b/CHANGELOG.md
index fbef6c7c9..9f0bc2181 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,7 @@ service can change a gate's answer, and a pull request number is only knowable f
## 2026-10-19
+- The version-basis tooling supports the next guide layout (maintainer ruling, 2026-10-19): a short introduction paragraph under the title, then the line "If an AI assistant is helping you secure a deployment, give it the prompt below:" with the prompt, then the versions and dates paragraph, then the guide; the end-of-file note shrinks to a short explanation with a link back to the prompt, which now appears once; `tools/version_basis.py --introduce` adds a guide's introduction and checks that nothing else in the guide changes; guides keep validating in the current layout until they move, and no guide changes in this PR (#701)
- Component names no longer repeat the version their header entry or summary line appends, in `deployment-lifecycle.md`, `gitops-controllers.md`, `host.md`, `mfa.md` and `self-signed.md`: headers now read, for example, "DNF 4.14.0", "Dex source v2.45.1" and "MySQL multifactor authentication 8.0 and 8.4", with each version stated once; no version, source, claim or guide body changes (#702)
- `authentication.md`, the last guide, moves to the two-section layout (maintainer request, 2026-10-18), with its own topics named first in its header; all 100 guides now use the layout; no guide body or digest changes (#700)
- Nineteen more guides move to the two-section layout (maintainer request, 2026-10-18), with each guide's own tool named first in its header; `mfa.md`, already in the layout, now lists authentik, Keycloak and oauth2-proxy in its header (Authelia has no recorded release version, so it stays in the folded count); no guide body or digest changes. Guides: `dotnet.md`, `php.md`, `ruby.md`, `nextjs.md`, `frontend-frameworks.md`, `web-exposure.md`, `admin-uis.md`, `bi-dashboards.md`, `devops-uis.md`, `server-admin-panels.md`, `low-code-builders.md`, `python.md`, `deployment-lifecycle.md`, `gpu-clouds.md`, `mlflow.md`, `paas.md`, `sqlite.md`, `sqlite-http-frontends.md`, `transactional-email-posture.md` (#699)
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index ad14f9e61..c4614be6e 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -160,15 +160,33 @@ pipe and backtick escaping, and unmatched backticks remain escaped. A rendered c
that still contains a literal `<` needs the pinned parser for this check. Component fields and the plain header retain their existing
escaping.
-The two-section layout has no front matter. Its H1 is followed by a generated plain-Markdown
-header, the unchanged human body and Sources, and a generated AI section at EOF. The existing
-single marker pair wraps the fixed H2 "About the AI assistant section below", the human note
-and its `text` prompt, the summary, and the complete metadata in a four-backtick `json` fence.
-END is the last non-blank line. Other headings, fences and HTML inside the pair are refused.
+The two-section layout has no front matter. After the H1, write one short intro paragraph
+describing the tool, application or service and explaining that this guide helps the reader's
+AI assistant check that it is configured securely. The generated top block follows that
+paragraph: the `ai-assistant-prompt` HTML anchor, the invitation line, the unchanged per-guide
+`text` prompt with its raw URL, and the versions/dates header. The intro is ordinary digested
+body text. The rest of the human body and Sources follow the generated top block.
+The single EOF marker pair wraps the fixed H2 "About the AI assistant section below", a
+two-sentence note linking back to the prompt anchor, the summary, and the complete metadata
+in a four-backtick `json` fence. The prompt appears only at the top. END is the last non-blank
+line. Other headings, fences and HTML inside the pair are refused.
The JSON generator refuses fence-looking payload lines. The llms bundle and plugin retain the
whole two-section guide, including the JSON. Legacy link-check preprocessing remains active;
two-section JSON is already fenced, and its component URLs must also occur in Sources.
+For incremental migration, the exact new top block may also immediately follow the H1.
+The parser first tries that boundary; otherwise it accepts only the boundary immediately
+after the first paragraph. H1 and paragraph separators are exactly two LFs. The paragraph
+must parse as one unindented CommonMark paragraph with the pinned parser, without HTML,
+blank lines or control characters other than LF; wrapped prose is allowed. Other placements,
+edited generated bytes and duplicate generated blocks fail closed, even during rebind.
+The historical two-section form (header after H1, prompt in the EOF note) remains supported
+byte for byte. Its EOF text fence selects that historical grammar, which cannot be combined
+with a new top block. Ordinary check/write operations preserve all these positions.
+Verify marking masks only an exact new top block before scanning, retaining physical line
+numbers; its prompt is never a guide Verify fence. TLS safety masks it only after canonical
+guide validation. Bare-URL checks retain their ordinary prose/code rules, as for the old header.
+
The header names the first component in metadata order first, with its release basis or
`(pinned to a commit)` for a lowercase 7-to-40-digit hexadecimal basis, otherwise
`(no recorded version)` for `unknown`. It merges identical component names in metadata order
@@ -186,6 +204,18 @@ migrated, stale or invalid targets fail. Migration compares old and new digested
metadata, refusing any difference. It never rebinds. Rebuild both bundles and run the gates in
the same change as each migration batch. `--check` and `--write` preserve each guide's layout.
+For a two-section guide with its generated block immediately after the H1, run
+`python3 tools/version_basis.py --introduce GUIDE --intro 'One human-written paragraph.'`.
+This accepts the historical form as well as a new top prompt without an intro. It validates
+the original guide, inserts the supplied paragraph, moves the prompt/header below it, shortens
+the EOF note and rebinds the digest. Before writing, it asserts that the new body equals the
+old body with only that paragraph and its two separator LFs inserted after the H1, that no
+other metadata changed, and that the result validates and renders identically. It refuses
+legacy front matter, an existing intro before a top prompt, and invalid or stale inputs.
+Review the intro for meaning; paragraph syntax cannot establish human authorship or accuracy.
+Rebuild both bundles and run the gates for each migration batch. This tooling change delivers
+no guide edits.
+
Schema 1 requires exactly these top-level fields:
- `schema`: integer `1`.
@@ -196,10 +226,10 @@ Schema 1 requires exactly these top-level fields:
- `body_sha256`: lowercase SHA-256 of the UTF-8 body returned by `without_summary()` after `split()`
in `tools/version_basis.py`. Legacy front matter and the summary are excluded. For the new
layout, the EOF section and exactly two separator LFs are removed, and validation recomputes
- and strips only the exact generated header. Human-body bytes, including trailing whitespace,
+ and strips only the exact generated top block (or historical header). Human-body bytes, including trailing whitespace,
retain their old meaning. Do not hash the complete Markdown file. A changed or stale header
fails even with `--write` or `--rebind`; after changing metadata that affects the header,
- replace the header with the exact output of `header(data)` before regenerating. Do not rebind
+ replace the block with `top_block(data, guide)` (historical form: `header(data)`) before regenerating. Do not rebind
to excuse a header mismatch.
- `components`: nonempty object keyed by IDs matching `[a-z][a-z0-9-]*`. Each component has exactly
`name`, `basis` and `sources`. Use `unknown` when no version is recorded, but a source URL
diff --git a/tools/check_verify_marking.py b/tools/check_verify_marking.py
index 5e5a3263c..ca0d5e5b0 100644
--- a/tools/check_verify_marking.py
+++ b/tools/check_verify_marking.py
@@ -399,6 +399,9 @@ def guide_context(text, in_verify=False, in_quote=False, *, enrolled=False):
# Only a document's leading, strictly parsed metadata is opaque. Recursive
# quote scans must never gain a front-matter or generated-summary exemption.
if not in_quote:
+ if enrolled:
+ from version_basis import mask_top
+ text = mask_top(text)
text = text.replace("\r\n", "\n").replace("\r", "\n")
from version_basis import split_front_matter
data, body = split_front_matter(text)
@@ -722,7 +725,9 @@ def scan_units(text, *, enrolled=False, parser=None):
legacy_ranges = set(verify_ranges(legacy_heads))
# Front matter is strict JSON metadata, not Markdown. Preserve maps
# while masking it; no declaration or heading is selected from source.
- from version_basis import split_front_matter, START, END, NOTE_TITLE
+ from version_basis import split_front_matter, START, END, NOTE_TITLE, mask_top
+ if enrolled:
+ text = mask_top(text)
text = text.replace("\r\n", "\n").replace("\r", "\n")
try:
data, body = split_front_matter(text)
diff --git a/tools/check_verify_safety.py b/tools/check_verify_safety.py
index 48e7fc495..ab5737733 100644
--- a/tools/check_verify_safety.py
+++ b/tools/check_verify_safety.py
@@ -301,6 +301,7 @@ def logical_lines(text, guide="GUIDE.md"):
try:
section = vb.eof_section(text)
if vb.updated(text, guide) == text:
+ text = vb.mask_top(text)
text = text[:text.index(section)]
except (OSError, ValueError, TypeError, KeyError):
pass
diff --git a/tools/run_all_checks.sh b/tools/run_all_checks.sh
index d55c119b4..bcd65295e 100755
--- a/tools/run_all_checks.sh
+++ b/tools/run_all_checks.sh
@@ -643,6 +643,13 @@ heading_slugs() {
import re, sys
text = open(sys.argv[1], encoding="utf-8").read()
+sys.path.insert(0, "tools")
+from version_basis import PROMPT_ANCHOR, mask_top
+try:
+ if mask_top(text, sys.argv[1]) != text:
+ print(PROMPT_ANCHOR)
+except (ValueError, TypeError, KeyError):
+ pass # Invalid generated content cannot supply the prompt anchor.
text = re.sub(r"", "", text, flags=re.S)
in_fence = False
diff --git a/tools/test_verify_marking.py b/tools/test_verify_marking.py
index 55647796e..63c64c051 100644
--- a/tools/test_verify_marking.py
+++ b/tools/test_verify_marking.py
@@ -290,7 +290,7 @@ def run(body, expected, name, message="", *, unit_expected=None):
# The EOF exception admits only the fixed H2 and the two data fences.
from version_basis import NOTE_TITLE, note
- ai = (START + '\n' + note('guide.md') + '\n\n**Version basis**\n\n'
+ ai = (START + '\n' + note('guide.md', historical=True) + '\n\n**Version basis**\n\n'
'````json\n{}\n````\n' + END + '\n')
bottom = marked_pilot + '\n\n' + ai
run(bottom, 0, 'EOF note ends Verify without adding units')
diff --git a/tools/test_version_basis.py b/tools/test_version_basis.py
index e66d546d4..0f2453ec6 100644
--- a/tools/test_version_basis.py
+++ b/tools/test_version_basis.py
@@ -1923,8 +1923,8 @@ def test_migration_refuses_digest_change(self):
vb.migrated(self.legacy(BODY.replace('# Fixture\n\n', '# Fixture\nIntro\n\n')),
'fixture.md')
renderer = vb.two_section
- def changed(data, body, guide):
- return renderer(data, body + 'Extra prose.\n', guide)
+ def changed(data, body, guide, **options):
+ return renderer(data, body + 'Extra prose.\n', guide, **options)
with patch.object(vb, 'two_section', side_effect=changed):
with self.assertRaisesRegex(ValueError, 'migration changed digested body'):
vb.migrated(self.legacy(), 'fixture.md')
@@ -2020,5 +2020,170 @@ def test_shape_urls_and_verify_fingerprints(self):
self.assertEqual(list(findings), [f'fixture.md:{line}: bare URL contains __: https://example.com/a__b'])
+class TopPromptTests(unittest.TestCase):
+ INTRO = 'TEST TEXT: Product is a service; this guide helps your AI assistant check its security.'
+
+ def historical(self):
+ return vb.migrated(vb.updated(document(fixture()), 'fixture.md'), 'fixture.md')
+
+ def test_positions_digest_prompt_and_note(self):
+ old = self.historical()
+ introduced = vb.introduced(old, 'fixture.md', self.INTRO)
+ expected = BODY.replace('# Fixture\n\n', '# Fixture\n\n' + self.INTRO + '\n\n', 1)
+ for position, body in (('title', BODY), ('intro', expected)):
+ with self.subTest(position=position):
+ data = fixture()
+ data['body_sha256'] = vb.digest(body)
+ text = vb.two_section(data, body, 'fixture.md', position=position)
+ self.assertEqual(vb.updated(text, 'fixture.md'), text)
+ self.assertEqual(vb.without_summary(text), body)
+ self.assertEqual(vb.split(text)[0]['body_sha256'], vb.digest(body))
+ start = len('# Fixture\n\n') + (len(self.INTRO) + 2 if position == 'intro' else 0)
+ self.assertEqual(text[start:start + len(vb.top_block(data, 'fixture.md'))],
+ vb.top_block(data, 'fixture.md'))
+ self.assertEqual(text.count(vb.prompt('fixture.md')), 1)
+ self.assertIn(vb.prompt('fixture.md'), old)
+ self.assertIn('', text)
+ self.assertIn('[the prompt at the top](#ai-assistant-prompt)', vb.eof_section(text))
+ self.assertNotIn('```text', vb.eof_section(text))
+ self.assertEqual(vb.render(data, 'fixture.md').split('**Version basis**', 1)[1],
+ vb.render(data, 'fixture.md', historical=True).split('**Version basis**', 1)[1])
+ if position == 'intro':
+ self.assertEqual(text, introduced)
+ else:
+ self.assertEqual(vb.introduced(text, 'fixture.md', self.INTRO), introduced)
+ with self.assertRaisesRegex(ValueError, 'body changed'):
+ vb.updated(introduced.replace(self.INTRO, self.INTRO + ' Changed.'), 'fixture.md')
+ self.assertEqual(vb.updated(old, 'fixture.md'), old)
+
+ def test_edited_duplicate_misplaced_and_wrong_guide_fail_closed(self):
+ for position in ('title', 'intro'):
+ text = vb.two_section(fixture(), BODY, 'fixture.md', position=position)
+ block = vb.top_block(fixture(), 'fixture.md') + '\n\n'
+ stripped = text.replace(block, '', 1)
+ mutations = [text.replace('give it the prompt below:', 'give it this prompt:', 1),
+ text.replace('/main/fixture.md', '/main/other.md', 1),
+ text.replace('ai-assistant-prompt', 'changed-anchor', 1),
+ text.replace('Written against Product', 'Written against Other', 1),
+ text.replace('```text', '```bash', 1),
+ text.replace(block, block + block, 1), stripped,
+ stripped.replace('## Verify', 'Extra paragraph.\n\n' + block + '## Verify', 1),
+ stripped.replace('## Sources', block + '## Sources', 1),
+ text.replace(block, '\n' + block, 1),
+ text.replace(block, block.replace('\n', '\r\n'), 1)]
+ for changed in mutations:
+ for operation in (vb.updated, vb.rebound):
+ with self.subTest(position=position, operation=operation, changed=changed[:150]):
+ with self.assertRaises(ValueError):
+ operation(changed, 'fixture.md')
+ historical = self.historical()
+ with self.assertRaises(ValueError):
+ vb.updated(historical.replace(vb.header(fixture()), vb.top_block(fixture(), 'fixture.md'), 1),
+ 'fixture.md')
+
+ def test_intro_grammar_and_body_difference_assertion(self):
+ old = self.historical()
+ for intro in ('', ' one', 'one\n\ntwo', '# Heading', '- item', '> quote',
+ '```text\nx\n```', 'HTML', 'one\n---', 'one\rwrapped',
+ 'one\n\n', 'one\n indented', '[ref]: https://example.com'):
+ with self.subTest(intro=intro), self.assertRaises(ValueError):
+ vb.introduced(old, 'fixture.md', intro)
+ wrapped = self.INTRO + '\nA second line of the same paragraph.'
+ result = vb.introduced(old, 'fixture.md', wrapped)
+ self.assertIn(wrapped, vb.without_summary(result))
+ with self.assertRaisesRegex(ValueError, 'already has an intro'):
+ vb.introduced(result, 'fixture.md', self.INTRO)
+ renderer = vb.two_section
+ def changed(data, body, guide, **options):
+ if options.get('position') == 'historical':
+ return renderer(data, body, guide, **options)
+ return renderer(data, body.replace('A documented control.', 'Changed control.'), guide, **options)
+ with patch.object(vb, 'two_section', side_effect=changed):
+ with self.assertRaisesRegex(ValueError, 'beyond inserted intro'):
+ vb.introduced(old, 'fixture.md', self.INTRO)
+ def changed_metadata(data, body, guide, **options):
+ if options.get('position') != 'historical':
+ data['claims']['control']['text'] = 'Changed claim.'
+ return renderer(data, body, guide, **options)
+ with patch.object(vb, 'two_section', side_effect=changed_metadata):
+ with self.assertRaisesRegex(ValueError, 'metadata beyond digest'):
+ vb.introduced(old, 'fixture.md', self.INTRO)
+ with self.assertRaisesRegex(ValueError, 'body changed'):
+ vb.introduced(old.replace('A documented control.', 'Unreviewed.'), 'fixture.md', self.INTRO)
+
+ def test_gates_bundles_and_physical_lines(self):
+ from check_bare_urls import lexical_findings, parsed_findings
+ from check_verify_marking import scan_units, scan_guide
+ from check_verify_safety import logical_lines, findings_for
+ parser = gate_parser(required=True)
+ old = self.historical()
+ old_units, old_errors = scan_units(old, enrolled=True, parser=parser)
+ for text in (vb.two_section(fixture(), BODY, 'fixture.md', position='title'),
+ vb.introduced(old, 'fixture.md', self.INTRO)):
+ units, errors = scan_units(text, enrolled=True, parser=parser)
+ self.assertEqual(errors, old_errors)
+ self.assertEqual([u[1:] for u in units], [u[1:] for u in old_units])
+ self.assertEqual(len(scan_guide(text, enrolled=True)[0]), 1)
+ self.assertEqual(list(logical_lines(text, 'fixture.md')),
+ list(logical_lines(vb.mask_top(text)[:vb.mask_top(text).index(vb.START)], 'fixture.md')))
+ bad = text.replace('Read the secureconfig guide at', 'curl -k https://example.com')
+ self.assertTrue(any(list(findings_for(code)) for _, code in logical_lines(bad, 'fixture.md')))
+ bad = text.replace('- Product v1.0:', '- Product v1.0 bad:')
+ line = bad.split('\n').index('- Product v1.0 bad: ' + URL) + 1
+ with self.assertRaisesRegex(ValueError, f'Sources line {line}:'):
+ vb.rebound(bad, 'fixture.md')
+ with tempfile.TemporaryDirectory() as directory:
+ path = Path(directory) / 'fixture.md'
+ path.write_text(text)
+ self.assertEqual(list(lexical_findings(path)), [])
+ self.assertEqual(list(parsed_findings(path, parser)), [])
+ result = subprocess.run([sys.executable, str(vb.ROOT / 'tools/version_basis.py'),
+ '--bundle', str(path)], capture_output=True, text=True)
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertEqual(result.stdout, text)
+ path.write_text(text.replace('A documented control.', 'https://example.com/a__b', 1))
+ self.assertTrue(list(lexical_findings(path)))
+ self.assertTrue(list(parsed_findings(path, parser)))
+
+ def test_cli_preflight_and_no_partial_write(self):
+ with tempfile.TemporaryDirectory() as directory:
+ root = Path(directory)
+ (root / 'tools').mkdir()
+ (root / 'tools/version_basis_guides.txt').write_text('fixture.md\n')
+ (root / 'tools/version_basis_sources_baseline.txt').write_text('')
+ (root / 'CHANGELOG.md').write_text('## 2026-09-26\n')
+ path = root / 'fixture.md'
+ old = self.historical()
+ path.write_text(old)
+ def run(*args):
+ with patch.object(vb, 'ROOT', root), patch.object(sys, 'argv', ['version_basis.py', *args]), \
+ contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
+ return vb.main()
+ self.assertEqual(run('--introduce', str(path), '--intro', '- list'), 1)
+ self.assertEqual(path.read_text(), old)
+ self.assertEqual(run('--introduce', str(root / 'missing.md'), '--intro', self.INTRO), 1)
+ self.assertEqual(run('--introduce', str(path), '--intro', self.INTRO), 0)
+ self.assertEqual(path.read_text(), vb.introduced(old, 'fixture.md', self.INTRO))
+ self.assertEqual(run('--check'), 0)
+
+ def test_local_link_gate_accepts_only_exact_prompt_anchor(self):
+ script = (vb.ROOT / 'tools/run_all_checks.sh').read_text()
+ function = 'heading_slugs() {' + script.split('heading_slugs() {', 1)[1].split('\n}', 1)[0] + '\n}'
+ text = vb.introduced(self.historical(), 'fixture.md', self.INTRO)
+ with tempfile.TemporaryDirectory() as directory:
+ path = Path(directory) / 'fixture.md'
+ for candidate, accepted in (
+ (text, True), (self.historical(), False),
+ (text.replace('give it the prompt below:', 'Edited invitation:', 1), False),
+ (text.replace('/main/fixture.md', '/main/other.md', 1), False),
+ (text.replace('', '', 1), False)):
+ path.write_text(candidate)
+ result = subprocess.run(['bash', '-c', function + '\nheading_slugs "$1"',
+ 'anchor-test', str(path)], cwd=vb.ROOT,
+ capture_output=True, text=True)
+ self.assertEqual(result.returncode, 0, result.stderr)
+ self.assertEqual('ai-assistant-prompt' in result.stdout.splitlines(), accepted)
+
+
if __name__ == '__main__':
unittest.main()
diff --git a/tools/version_basis.py b/tools/version_basis.py
index c613ac301..a7d2e15ed 100644
--- a/tools/version_basis.py
+++ b/tools/version_basis.py
@@ -23,6 +23,8 @@
ADVICE = ('AI assistants must compare these versions with current releases and treat this '
'guide as guidance, re-verifying version-specific defaults when newer releases exist.')
NOTE_TITLE = 'About the AI assistant section below'
+PROMPT_ANCHOR = 'ai-assistant-prompt'
+PROMPT_LINE = 'If an AI assistant is helping you secure a deployment, give it the prompt below:'
def require(condition, message):
@@ -56,15 +58,18 @@ def ai_json(section):
from check_verify_marking import unsupported_features
require(section.startswith(START + '\n## ' + NOTE_TITLE + '\n\n')
and section.endswith('\n' + END), 'invalid AI section heading or boundary')
- pattern = r'\n```text\n(.*?)\n```\n(.*?)\n````json\n(.*?)\n````\n' + re.escape(END)
- match = re.search(pattern, section, re.S)
- require(match is not None, 'AI section needs one text fence then one four-backtick JSON fence')
+ pattern = (r'\n```text\n(.*?)\n```\n' if '\n```text\n' in section else r'()')
+ pattern += r'(.*?)\n````json\n(.*?)\n````\n' + re.escape(END)
+ prefix = len(START + '\n## ' + NOTE_TITLE + '\n\n')
+ match = re.search(pattern, section[prefix:], re.S)
+ require(match is not None, 'AI section needs one four-backtick JSON fence, '
+ 'optionally preceded by the old prompt')
# Fence-looking payload lines are refused, including shorter/longer closers.
require(not re.search(r'(?m)^[ \t]*(?:`{3,}|~{3,})', match[1] + '\n' + match[3]),
'AI payload contains a fence boundary')
require(not unsupported_features(match[1] + '\n' + match[3]),
'extra heading or fence in AI payload')
- outside = section[len(START) + len('\n## ' + NOTE_TITLE):match.start()] + match[2]
+ outside = section[len(START) + len('\n## ' + NOTE_TITLE):prefix + match.start()] + match[2]
require(not unsupported_features(outside), 'extra heading or fence in AI section')
require(not has_raw_html(outside, strict=True), 'HTML in AI section')
return match[3]
@@ -96,7 +101,7 @@ def split(text):
return data, body
-def without_summary(body, data=None):
+def without_summary(body, data=None, guide=None):
require(body.count(START) == body.count(END) and body.count(START) <= 1,
'missing or duplicate summary boundary')
if START not in body:
@@ -106,7 +111,8 @@ def without_summary(body, data=None):
if body[start:].startswith(START + '\n## ' + NOTE_TITLE + '\n'):
stored, _ = split(body)
# The EOF renderer adds exactly two LFs, independent of the body's ending.
- return without_header(stored if data is None else data, body[:start - 2])
+ return without_header(stored if data is None else data, body[:start - 2], guide,
+ historical='\n```text\n' in eof_section(body))
require(body[:start].endswith('\n\n') and body[end + len(END):].startswith('\n\n'),
'summary needs blank lines')
return body[:start] + body[end + len(END) + 2:]
@@ -305,7 +311,7 @@ def definitions(body, line_offset=0):
reference link, however its label or an invalid inline destination is spelled.
"""
from check_guide_shape import scan
- errors = [f'line {index + 1 + line_offset}: reference definition; '
+ errors = [f'line {source_line(index + 1, line_offset)}: reference definition; '
'enrolled guides cite with inline links only'
for index, line in sorted(scan(body)[1].items()) if _DEFINITION.match(line)]
require(not errors, '; '.join(errors))
@@ -375,7 +381,7 @@ def sources_grammar(body, heads, line_offset=0):
for index in range(token.start, token.end)}
for index, line in enumerate(lines):
if index not in code:
- errors.extend(f'Sources line {start + index + 1 + line_offset}: {construct}; '
+ errors.extend(f'Sources line {source_line(start + index + 1, line_offset)}: {construct}; '
'cite URLs as bare URLs, or inline [text](url) links'
for construct in grammar_violations(line, raw[start + index]))
require(not errors, '; '.join(errors))
@@ -413,7 +419,7 @@ def guide_violations(body, heads, line_offset=0):
for index, line in enumerate(lines):
if index in code:
continue
- where = f'line {index + 1 + line_offset}: '
+ where = f'line {source_line(index + 1, line_offset)}: '
if _HTML_BLOCK.match(line) and not _LINE_START_AUTOLINK.match(line):
errors.append(where + 'raw HTML block')
errors.extend(where + f'raw HTML <{match[1]}> element' for match in _TAG.finditer(line)
@@ -430,9 +436,9 @@ def guide_violations(body, heads, line_offset=0):
if SOURCES_RE.match(title) and not _SOURCES_HEADING.fullmatch(title):
construct = ('text after the Sources heading' if title[SOURCES_RE.match(title).end():]
else 'text other than words before Sources in its heading')
- errors.append(f'line {index + 1 + line_offset}: {construct}')
+ errors.append(f'line {source_line(index + 1, line_offset)}: {construct}')
elif SOURCES_RE.match(title) and not ATX_RE.match(lines[index]):
- errors.append(f'line {index + 1 + line_offset}: Setext Sources heading')
+ errors.append(f'line {source_line(index + 1, line_offset)}: Setext Sources heading')
require(not errors, '; '.join(errors))
@@ -701,7 +707,7 @@ def validate(data, body, guide='GUIDE.md', baseline=None, line_offset=0, parser=
'documentation_checked must be YYYY-MM (preserve recorded precision)')
documentation = datetime.date.fromisoformat(data['documentation_checked'] + '-01')
require(documentation <= checked, 'documentation check follows metadata review')
- body = without_summary(body, data) # Strip only the header recomputed from this metadata.
+ body = without_summary(body, data, guide) # Strip only the exact generated top block.
string(data['body_sha256'])
require(re.fullmatch(r'[0-9a-f]{64}', data['body_sha256']), 'bad body digest')
expected_digest = digest(body)
@@ -933,22 +939,76 @@ def header(data):
f'[the section for AI assistants](#{anchor}) at the end of this guide.')
-def without_header(data, body):
+def intro_end(body):
+ """Character offset after H1, one unindented CommonMark paragraph and two LFs."""
+ from check_verify_marking import gate_parser, ParserUnavailable
title, separator, rest = body.partition('\n\n')
- require(title.startswith('# ') and '\n' not in title, 'header must immediately follow the H1')
- expected = header(data) + '\n\n'
- require(separator and rest.startswith(expected), 'missing or stale generated header')
- return title + separator + rest[len(expected):]
-
-
-def note(guide):
+ require(title.startswith('# ') and '\n' not in title and separator,
+ 'generated block must immediately follow the H1 or its intro')
+ intro, separator, _ = rest.partition('\n\n')
+ require(separator and intro and all(line and not line[0].isspace()
+ for line in intro.split('\n')) and not any(ord(c) < 32 and c != '\n' for c in intro),
+ 'intro must be one unindented prose paragraph')
+ try:
+ tokens = gate_parser(required=True).parse(intro)
+ except ParserUnavailable as exc:
+ raise ValueError('intro recognition requires the pinned parser: ' + str(exc)) from exc
+ require([token.type for token in tokens] == ['paragraph_open', 'inline', 'paragraph_close']
+ and tokens[0].map == [0, len(intro.split('\n'))]
+ and not has_raw_html(intro, strict=True),
+ 'intro must be one unindented prose paragraph')
+ return len(title) + 2 + len(intro) + 2
+
+
+def top_span(data, body, guide=None, historical=False):
+ """Recognize only the exact generated bytes at either permitted boundary."""
+ title, separator, _ = body.partition('\n\n')
+ require(title.startswith('# ') and '\n' not in title and separator,
+ 'generated block must immediately follow the H1 or its intro')
+ start = len(title) + 2
+ if historical:
+ expected = header(data) + '\n\n'
+ position = 'historical'
+ else:
+ anchor = f'\n\n'
+ position = 'title'
+ if not body[start:].startswith(anchor):
+ start = intro_end(body)
+ position = 'intro'
+ candidate = body[start:]
+ name = re.search(r'\nhttps://raw\.githubusercontent\.com/jposluns/secureconfig/main/'
+ r'([A-Za-z][A-Za-z0-9-]*\.md)\nin full,', candidate)
+ require(name is not None, 'missing or stale generated top block')
+ expected = top_block(data, name[1] if guide is None else guide) + '\n\n'
+ require(body[start:].startswith(expected), 'missing or stale generated header or top block')
+ remaining = body[:start] + body[start + len(expected):]
+ require(PROMPT_ANCHOR not in remaining and PROMPT_LINE not in remaining
+ and header(data) not in remaining,
+ 'duplicate or misplaced generated block')
+ return start, start + len(expected), position
+
+
+def without_header(data, body, guide=None, historical=False):
+ start, end, _ = top_span(data, body, guide, historical)
+ return body[:start] + body[end:]
+
+
+def mask_top(text, guide=None):
+ """Blank an exact new top block for gates, preserving physical line numbers."""
+ if text.startswith('---\n') or START not in text or '\n## ' + NOTE_TITLE + '\n' not in text:
+ return text
+ section = eof_section(text)
+ if '\n```text\n' in section:
+ return text
+ data, _ = split(text)
+ start, end, _ = top_span(data, text[:text.index(START) - 2], guide)
+ return text[:start] + '\n' * text[start:end].count('\n') + text[end:]
+
+
+def prompt(guide):
name = Path(guide).name
require(re.fullmatch(r'[A-Za-z][A-Za-z0-9-]*\.md', name), 'bad guide name for AI prompt')
- return f"""## {NOTE_TITLE}
-
-The rest of this file is generated by the repository's tooling and written for AI assistants rather than for people. It records the versions this guide was written against, the guide's claims with a status for each (demonstrated by a check that ran, or reasoned from the cited sources), and the mapping from each claim to its sources. You are welcome to read it, but it is dense and can be hard to take in.
-
-If an AI assistant is helping you secure a deployment, give it the prompt below:
+ return f"""{PROMPT_LINE}
```text
Read the secureconfig guide at
@@ -960,16 +1020,34 @@ def note(guide):
guidance a newer release may have changed; and run or adapt the guide's
Verify steps where you can. List separately every item you could not
verify instead of reporting it as passed.
-```
+```"""
+
+
+def top_block(data, guide):
+ return f'\n\n' + prompt(guide) + '\n\n' + header(data)
+
+
+def note(guide, historical=False):
+ if historical:
+ return f"""## {NOTE_TITLE}
+
+The rest of this file is generated by the repository's tooling and written for AI assistants rather than for people. It records the versions this guide was written against, the guide's claims with a status for each (demonstrated by a check that ran, or reasoned from the cited sources), and the mapping from each claim to its sources. You are welcome to read it, but it is dense and can be hard to take in.
+
+{prompt(guide)}
This guide is guidance to verify against your own deployment, not a certification, an audit, or a guarantee of compliance with any standard."""
+ return (f'## {NOTE_TITLE}\n\n'
+ 'This generated section holds the recorded versions, claim statuses and source mappings '
+ f'for AI assistants; use [the prompt at the top](#{PROMPT_ANCHOR}). '
+ 'This guide is guidance to verify against your own deployment, not a certification, '
+ 'an audit, or a guarantee of compliance with any standard.')
-def render(data, guide=None):
+def render(data, guide=None, historical=False):
dates = (f"Metadata reviewed {data['checked']}; documentation checked {data['documentation_checked']} "
'(exact day unknown). ') if guide is None else ''
advice = ADVICE if guide is None else ADVICE.replace('these versions', 'the versions listed below')
- lines = [START] + ([note(guide), ''] if guide is not None else [])
+ lines = [START] + ([note(guide, historical), ''] if guide is not None else [])
lines += ['**Version basis**', '', advice, '',
dates + 'DEMONSTRATED refers to historical evidence in this guide; '
'REASONED records source reasoning, not a live demonstration. Unknown means no version recorded.', '',
@@ -1017,15 +1095,24 @@ def front_matter(data):
return '\n'.join(lines + [' }', '}', '---', ''])
+def source_line(line, offset):
+ if isinstance(offset, tuple):
+ boundary, count = offset
+ return line + (count if line > boundary else 0)
+ return line + offset
+
+
def body_line_offset(text, full_body):
"""Removed lines before human content below the H1, including Sources.
- EOF generated content contributes no offset; only its top header does.
+ EOF content contributes no offset. A top block shifts only following lines.
"""
if not text.startswith('---\n') and START in full_body:
data, _ = split(text)
if data is not None:
- return (header(data) + '\n\n').count('\n')
+ start, end, _ = top_span(data, full_body[:full_body.index(START) - 2],
+ historical='\n```text\n' in eof_section(full_body))
+ return full_body[:start].count('\n'), full_body[start:end].count('\n')
removed = full_body[full_body.index(START):full_body.index(END) + len(END) + 2] \
if START in full_body else ''
return text[:len(text) - len(full_body)].count('\n') + removed.count('\n')
@@ -1035,7 +1122,7 @@ def rebound(text, guide='GUIDE.md', baseline=None, parser=None):
"""Validate the reviewed inventory and replace only the top-level digest token."""
data, full_body = split(text)
require(data is not None, 'missing version_basis metadata')
- body = without_summary(full_body)
+ body = without_summary(full_body, guide=guide)
string(data['body_sha256'])
require(re.fullmatch(r'[0-9a-f]{64}', data['body_sha256']), 'bad body digest')
data['body_sha256'] = digest(body)
@@ -1060,19 +1147,52 @@ def rebound(text, guide='GUIDE.md', baseline=None, parser=None):
def updated(text, guide='GUIDE.md', baseline=None, parser=None):
data, full_body = split(text)
require(data is not None, 'missing version_basis metadata')
- body = without_summary(full_body)
+ body = without_summary(full_body, guide=guide)
validate(data, full_body, guide, baseline, body_line_offset(text, full_body), parser)
if not text.startswith('---\n'):
- return two_section(data, body, guide)
+ _, _, position = top_span(data, full_body[:full_body.index(START) - 2], guide,
+ historical='\n```text\n' in eof_section(full_body))
+ return two_section(data, body, guide, position=position)
title, rest = body.split('\n\n', 1)
prefix = front_matter(data)
return prefix + title + '\n\n' + render(data) + '\n\n' + rest
-def two_section(data, body, guide):
- title, rest = body.split('\n\n', 1)
- require(title.startswith('# ') and '\n' not in title, 'header must immediately follow the H1')
- return title + '\n\n' + header(data) + '\n\n' + rest + '\n\n' + render(data, guide) + '\n'
+def two_section(data, body, guide, *, position='intro'):
+ require(position in ('historical', 'title', 'intro'), 'unknown top-block position')
+ title, separator, _ = body.partition('\n\n')
+ require(title.startswith('# ') and '\n' not in title and separator,
+ 'generated block must immediately follow the H1 or its intro')
+ start = intro_end(body) if position == 'intro' else len(title) + 2
+ historical = position == 'historical'
+ block = header(data) if historical else top_block(data, guide)
+ return (body[:start] + block + '\n\n' + body[start:] + '\n\n'
+ + render(data, guide, historical=historical) + '\n')
+
+
+def introduced(text, guide, intro, baseline=None, parser=None):
+ """Insert only the supplied paragraph; validate and assert the body delta."""
+ require(not text.startswith('---\n'), '--introduce requires a two-section guide')
+ require(updated(text, guide, baseline, parser) == text, 'stale summary; refresh before migration')
+ data, full = split(text)
+ _, _, position = top_span(data, full[:full.index(START) - 2], guide,
+ historical='\n```text\n' in eof_section(full))
+ require(position in ('historical', 'title'), 'guide already has an intro before the top block')
+ old_body = without_summary(full)
+ title, rest = old_body.split('\n\n', 1)
+ expected = title + '\n\n' + intro + '\n\n' + rest
+ require(intro_end(expected) == len(title) + 2 + len(intro) + 2,
+ 'intro must be exactly one paragraph without surrounding blank lines')
+ original = json.loads(json.dumps(data))
+ data['body_sha256'] = digest(expected)
+ result = two_section(data, expected, guide)
+ new_data, new_full = split(result)
+ require(without_summary(new_full) == expected,
+ 'migration changed body beyond inserted intro; refusing')
+ require(new_data == dict(original, body_sha256=digest(expected)),
+ 'migration changed metadata beyond digest; refusing')
+ require(updated(result, guide, baseline, parser) == result, 'migration is not canonical')
+ return result
def migrated(text, guide, baseline=None, parser=None):
@@ -1080,8 +1200,8 @@ def migrated(text, guide, baseline=None, parser=None):
require(text.startswith('---\n'), '--migrate requires a legacy guide')
require(updated(text, guide, baseline, parser) == text, 'stale summary; refresh before migration')
data, full_body = split(text)
- old_body = without_summary(full_body)
- result = two_section(data, old_body, guide)
+ old_body = without_summary(full_body, guide=guide)
+ result = two_section(data, old_body, guide, position='historical')
new_data, new_full = split(result)
require(old_body == without_summary(new_full), 'migration changed digested body; refusing')
require(new_data == data, 'migration changed metadata; refusing')
@@ -1113,6 +1233,9 @@ def main():
modes.add_argument('--bundle', type=Path)
modes.add_argument('--migrate', nargs='+', type=Path, metavar='GUIDE',
help='convert named legacy guides, preserving the digested body exactly')
+ modes.add_argument('--introduce', type=Path, metavar='GUIDE',
+ help='move a two-section prompt to the top and insert --intro text')
+ parser.add_argument('--intro', help='one human-written paragraph for --introduce')
modes.add_argument('--rebind', type=Path, metavar='GUIDE',
help='assert the claim inventory was reviewed; update only its digest')
parser.add_argument('guide', nargs='?', type=Path,
@@ -1122,6 +1245,8 @@ def main():
parser.add_argument('--no-advisory', action='store_true',
help='suppress the missing-parser advisory when a caller has already printed it')
args = parser.parse_args()
+ if (args.intro is not None) != (args.introduce is not None):
+ parser.error('--intro and --introduce require each other')
if args.no_advisory and args.bundle is None:
parser.error('--no-advisory requires --bundle')
try:
@@ -1152,6 +1277,15 @@ def main():
data, _ = split(expected)
print(f" ok {path.name}: migrated; body bytes equal; body_sha256={data['body_sha256']}")
return 0
+ if args.introduce:
+ path = args.introduce.resolve()
+ require(path in selected, '--introduce requires an enrolled guide')
+ expected = introduced(path.read_bytes().decode('utf-8'), path.name, args.intro,
+ baseline, source_parser)
+ path.write_bytes(expected.encode('utf-8'))
+ print(f' ok {path.name}: introduced; body differs only by inserted paragraph; '
+ f"body_sha256={split(expected)[0]['body_sha256']}")
+ return 0
if args.rebind:
path = args.rebind.resolve()
require(path in selected, '--rebind requires an enrolled guide')