Notebook header `{raw}` blocks render as literal HTML source under lecture page titles

[mmcky]

Part of #147 (parity study 2026-09-11 — content correctness).

Revised in place 2026-09-11 after the maintainer review. The fix no longer lands in the theme: the lecture sources drop their {raw} blocks at cutover, and the notebook header moves into the ipynb export (QuantEcon/mystmd#108). The original theme-renderer options are kept below under "Options considered". The revision is recorded in the comments.

Problem

Most QuantEcon lecture sources open with a notebook-only logo header, <div id="qe-notebook-header">, in a {raw} jupyter or a {raw} html block. On mystmd neither form is dropped or rendered as HTML. The block's source appears as escaped text in a paragraph directly under the page title, so markup is the first thing a reader sees. Under Sphinx the {raw} jupyter form never reached the HTML page, and the book theme hid the {raw} html form with #qe-notebook-header { display: none; } (_quantecon-defaults.scss#L9-L11).

Both forms show today (checked 2026-09-11). Each page below serves <p>&lt;div id=&quot;qe-notebook-header&quot; ... immediately after its <h1>:

Page	Header form	Source
lecture-wasm, live: https://quantecon.github.io/lecture-wasm/short-path/	{raw} html	lectures/short_path.md#L15-L21
lecture-python-programming jb2 preview: /need-for-speed	{raw} jupyter	lectures/need_for_speed.md#L13-L19
lecture-python-programming jb2 preview: /polars	{raw} jupyter	lectures/polars.md#L15-L21

This repo's PR previews show the same text. They build lecture-python-programming's main branch with stock mystmd (preview.yml#L62-L88); an example is the PR #200 preview's /numpy.

The notebooks are broken too. The QuantEcon mystmd fork's ipynb export drops both forms and logs Unsupported node type: raw for each, so notebooks built on the fork have no header (QuantEcon/mystmd#108).

Why it happens

• Every {raw} directive becomes a raw node that keeps its lang and its source in value (myst-directives raw.ts#L15-L29).
• The per-page raw transform parses only latex and tex. Every other raw node gets a paragraph > text child holding the source verbatim, and any lang other than text logs unknown format for raw content: <lang> (myst-cli transforms/raw.ts#L14-L27). Upstream jupyter-book/mystmd has the same code, and its request for raw HTML support is still open (Allow raw HTML / script / style blocks in MyST documents jupyter-book/mystmd#1040).
• Nothing renders raw nodes. The theme's renderer overrides cover only list and output (app/renderers.tsx), and myst-to-react 1.3.0 has no raw entry. The default renderer wraps the escaped text in a <div>.
• A CSS rule alone cannot hide it: the page has no #qe-notebook-header element, only a <p> holding the escaped source.

What the lecture sources contain

Every {raw} block in the Markdown sources of 18 lecture repositories, counted on each default branch on 2026-09-11. 549 of the 554 blocks are the same header; no source uses a raw role.

Repository	Commit	Header, {raw} jupyter	Header, {raw} html	Other {raw} blocks
lecture-python.myst	3b5fde64	107	2	1 (opt_transport.md table)
lecture-python.zh-cn	3f423564	107	2	1 (opt_transport.md table)
lecture-python-advanced.myst	7f42c5e2	3	52	0
lecture-dp	da83d88d	21	19	0
lecture-python-programming	5e574c17	23	0	0
lecture-python-programming.fa, .fr, .zh-cn	1ef279e7, f0f6ce0a, 90336d28	23 each	0	0
lecture-python-programming.ml	4a33f615	2	0	0
lecture-julia.myst	18877b5d	0	60	0
lecture-dle	975b7a43	0	25	0
lecture-tools-techniques	e1d01560	0	14	1 (opt_transport.md table)
lecture-eqm	5c574371	0	12	0
lecture-stats	54114b26	0	11	0
lecture-python-intro	c8b8a71c	2	6	1 (simple_linear_regression.md iframe)
lecture-intro.zh-cn	156fe455	2	6	1 (simple_linear_regression.md iframe)
lecture-jax	24ef0c8a	0	3	0
lecture-wasm	149182bd	0	1	0
Total		336	213	5

What each kind of block does under Sphinx today:

Block	Web page	Published notebook
Header, {raw} jupyter	not emitted	the logo cell (cell 0 of lecture-python-programming.notebooks numpy.ipynb, cell 1 of lecture-python.notebooks kalman.ipynb)
Header, {raw} html	hidden by CSS	absent (lecture-python-advanced.notebooks arma.ipynb, lqramsey.ipynb)
opt_transport.md factory table	an HTML table the lecture text refers to	absent (lecture-python.notebooks and lecture-tools-techniques.notebooks opt_transport.ipynb)
Our World in Data <iframe>	an embedded chart	absent; the cell keeps only the link (lecture-python-intro.notebooks simple_linear_regression.ipynb)

The {raw} html header does nothing in either output: its 213 blocks are dead content.

Options considered

The first version of this issue proposed a theme-side fix. The review weighed four approaches:

Option	What it does	Why it was not chosen
A, broad (theme)	Render nothing for raw nodes other than latex, tex and text	Hides the web symptom only. It also deletes the opt_transport.md table, and leaves the notebooks without a header.
A, narrow (theme)	Render nothing for raw nodes containing qe-notebook-header	Hides the header only. The table still shows as literal markup, and the notebooks still have no header.
B, pass through (theme)	Drop jupyter nodes, inject html nodes as HTML, and hide the header with CSS	Puts unsanitised author HTML into the server-rendered page, and still leaves the notebooks without a header.
C, format-aware {raw} in the fork	Teach mystmd {raw} jupyter (notebook only) and {raw} html	Builds permanent Sphinx compatibility into the fork for a builder name upstream does not use, to keep 549 copies of branding in lecture files.

Decision (2026-09-11)

1. Sources become mystmd-native; there is no Sphinx compatibility layer. The migration is all-or-nothing, so no lecture repo builds with both Sphinx and mystmd after cutover. The sources must stay valid for Sphinx until then, so the rewrite is scripted and runs at cutover.

2. The notebook header is generated by the ipynb export from a project setting, settings.myst_to_ipynb.header, in ipynb export has no project-level notebook header, so the QuantEcon logo is copied into 549 lecture source files as raw blocks mystmd#108. That makes one edit point per repository, or one across all repositories through a shared config pulled in with extends:. Nothing about the header reaches the web page, so the theme needs no renderer and no CSS rule.

3. A rewrite script removes every {raw} block from the lecture sources:

   Block	Rewrite	Verified
   Header, either form (549)	Delete	—
   Our World in Data iframe (2)	The native {iframe} directive	With myst v1.10.1 (qe-v10): renders an iframe node
   opt_transport.md table (3)	Remove the {raw} html fence and keep the <table> as HTML in the Markdown	With myst v1.10.1 (qe-v10): mystmd's HTML transform gives a real table with colspan: 3 and rowspan: 2
   Any other {raw} block	The script stops and names the file and line	—

4. Sites already building on mystmd take the rewrite now: lecture-wasm (one header, short_path.md, reported on short-path renders its qe-notebook-header raw HTML block as literal visible text lecture-wasm#66) and lecture-python-programming's jb2 branch ([Experimental] Build the lectures with QuantEcon/mystmd instead of jupyter-book lecture-python-programming#363).

5. This repo's PR preview runs the same script after myst init (preview.yml#L62-L88). Previews then show post-cutover sources, and every theme PR exercises the script.

6. General output targeting belongs elsewhere. The design for content targeted by output (Sphinx {only} style: one way to mark a node's outputs, applied where each output's pipeline splits off) is on Sphinx {only} blocks render as red Unknown Directive callouts because mystmd has no {only} directive mystmd#104, with upstream Tag content for optional inclusion in build outputs jupyter-book/mystmd#1280.

Open: where the rewrite script lives. The proposal is scripts/ in this repo, because preview.yml is its first user; it moves to the lecture migration tooling if that has a home before cutover.

Work items

Item	Repository	Tracked in
myst_to_ipynb.header setting in the ipynb export	QuantEcon/mystmd	QuantEcon/mystmd#108
Rewrite script	this repo (proposed)	this issue
PR preview runs the rewrite	this repo	this issue
Source inventory across the lecture repos, and the cutover rollout (one PR per repo)	QuantEcon/workspace-lectures	QuantEcon/workspace-lectures#64
Remove the header from short_path.md	QuantEcon/lecture-wasm	QuantEcon/lecture-wasm#66
Remove the headers on jb2 and set the header in myst.yml	QuantEcon/lecture-python-programming	QuantEcon/lecture-python-programming#363
Migration checklist entries for the script and the setting	this repo	#209
Output targeting design	QuantEcon/mystmd	QuantEcon/mystmd#104

Acceptance criteria

• The rewrite script deletes both header forms, converts the two iframes to {iframe} and the three tables to HTML in the Markdown, leaves a file unchanged on a second run, and exits non-zero naming any other {raw} block.
• The PR preview workflow runs the script, and a PR preview's /need-for-speed and /numpy show no header text.
• https://quantecon.github.io/lecture-wasm/short-path/ shows no header text, and the result is reported on short-path renders its qe-notebook-header raw HTML block as literal visible text lecture-wasm#66.
• The jb2 build of /need-for-speed shows no header text.
• ipynb export has no project-level notebook header, so the QuantEcon logo is copied into 549 lecture source files as raw blocks mystmd#108 is merged.
• The migration checklist (Lecture repos moving off the Sphinx theme have no checklist of what myst.yml and CI must now set #209) lists running the rewrite script and setting settings.myst_to_ipynb.header.

Export gaps found while testing

The test build for the decision also showed three ipynb export gaps in the fork, now filed:

• ipynb export writes tables with colspan or rowspan as pipe tables, which cannot express merged cells, so the columns misalign mystmd#109: a table written as HTML in the Markdown with colspan/rowspan renders correctly on the site, but is written into the notebook as a pipe table, so its columns misalign. After the rewrite, the opt_transport notebooks would carry that misaligned table where Sphinx shipped none.
• ipynb export with markdown: commonmark writes {iframe} as MyST directive syntax, which Jupyter and Colab show as a literal code block mystmd#110: under markdown: commonmark, an {iframe} is written as MyST directive syntax, which Jupyter and Colab show as a literal code block. An earlier version of this section said the iframe was dropped; it is not. Sphinx's notebook kept only a link.
• ipynb export drops every raw node, including {raw} text and {raw} latex content, and logs an error that --strict does not count mystmd#111: every raw node is dropped from the notebook, with an Unsupported node type: raw error that --strict does not count. The rewrite removes every raw node from the lecture sources, so this matters mainly to other projects.

Context

• Source: the 2026-09-11 book-theme parity study, row X02 (verdict missing_untracked, relevance high).
• Scope history: the first version of this issue counted 456 header files in 15 repositories (122 {raw} html, 334 {raw} jupyter). The revision's audit covers 18 repositories, adding lecture-julia.myst, lecture-dle and lecture-intro.zh-cn, and counts blocks: 549 headers. The study's addendum had once limited the defect to the {raw} html form, from a preview URL that returned 404; both forms leak.
• Consumer report: short-path renders its qe-notebook-header raw HTML block as literal visible text lecture-wasm#66 (open since 2026-08-12), raised during the QuantEcon/workspace-lectures#37 validation.
• Earlier sighting: a PR #155 comment (2026-09-02) noted the literal header on that PR's preview.
• Review record: the licence-footer decision (Decide how the CC BY-SA licence footer survives the cutover: a theme default or a per-repo footer.md #203) came from the same review of the Book-theme parity cutover — tracking #147 gate; the decision above is stamped in this issue's comments.

[mmcky]

Revision — 2026-09-11

Revised in place after the review of the #147 gate. Decision: the fix leaves the theme.

• Sources: at cutover a rewrite script removes every {raw} block from the lecture sources. The 549 header blocks are deleted, the two iframes become {iframe}, and the three opt_transport.md tables become HTML in the Markdown.
• Notebook header: the ipynb export generates it from settings.myst_to_ipynb.header, filed as ipynb export has no project-level notebook header, so the QuantEcon logo is copied into 549 lecture source files as raw blocks mystmd#108. The header has one edit point per repository, or one across all of them through extends:.
• Not doing: no raw renderer in the theme, and no Sphinx compatibility for {raw} jupyter or {raw} html in the fork. General output targeting goes to Sphinx {only} blocks render as red Unknown Directive callouts because mystmd has no {only} directive mystmd#104.
• Now: lecture-wasm and the lecture-python-programming jb2 branch take the rewrite straight away, and this repo's PR preview runs the same script.

What changed in the body:

• The audit now covers 18 repositories: 549 header blocks and 5 other raw blocks, checked against the published notebooks and a test build on the fork.
• The theme-renderer variants moved under "Options considered".
• The work items and acceptance criteria were replaced.
• One item is still open: where the rewrite script lives.

[mmcky]

Revised in place: the three ipynb export gaps from the test build are now filed as QuantEcon/mystmd#109 (span tables), QuantEcon/mystmd#110 ({iframe} under markdown: commonmark) and QuantEcon/mystmd#111 (raw nodes). Also corrected: the iframe is not dropped from the notebook, as the section first said. It is written as MyST directive syntax.

[mmcky]

Revised in place: a work item now links QuantEcon/workspace-lectures#64, which coordinates the cross-repo side of the rewrite. That issue holds the source inventory of 20 lecture repos, covering every construct mystmd does not render, not only {raw} blocks, and the per-repo rollout at cutover.