feat: link contents to .ipynb and expand LaTeX macros before conversion

- Point the "Contents" and "Exercise Answers" tables in basic_lessons/README.md
  at the generated .ipynb files instead of the .md sources, so readers land on
  the executable notebooks.
- Add convert_to_ipynb.py: a two-step .md -> intermediate.md -> .ipynb pipeline
  that first expands the project.math LaTeX macros (myvec/mymatrix/quat/dual)
  into an intermediate .md, then runs jupytext on that file. Most .ipynb
  renderers cannot draw custom macros, so the generated notebooks now contain
  only standard LaTeX and render anywhere. The intermediate file is a transient
  build artifact (deleted after conversion; --keep retains it).
  Expansion mirrors KaTeX/TeX macro semantics (braced args incl. nested
  braces, single space-separated tokens, no-arg macros) and leaves
  {code-cell} bodies untouched so Python source is never corrupted.
- Wire the script into the CI "Generate downloadable notebooks" step.
- Document the new pipeline in AGENTS.md.

Co-authored-by: openhands <openhands@all-hands.dev>

Author: openhands-agent

.github/workflows/notebook_to_html.yml

@@ -33,18 +33,16 @@ jobs:
       # Checks out your repository under $GITHUB_WORKSPACE, so your job can access it
       - uses: actions/checkout@v4
 
-      # Convert basic_lessons/*.md → basic_lessons/*.ipynb so the site provides "Download notebook" buttons
+      # Convert basic_lessons/*.md → basic_lessons/*.ipynb so the site provides "Download notebook" buttons.
+      # The .md uses custom LaTeX macros (see `project.math` in myst.yml) that most .ipynb
+      # renderers cannot draw, so convert_to_ipynb.py first expands them into an intermediate
+      # .md (which is deleted afterwards) and then runs jupytext on that.
       # https://jupytext.readthedocs.io/ — supports md:myst format natively
       # Must run BEFORE build so myst/jupyter-book can pick up the generated notebooks
       - name: Generate downloadable notebooks
         run: |
-          pip install jupytext
-          for f in basic_lessons/lesson*_tutorial.md basic_lessons/lesson*_exercise_answers.md; do
-            [ -f "$f" ] || continue
-            out="${f%.md}.ipynb"
-            python -m jupytext --from md:myst --to notebook --output "$out" "$f"
-            echo "Generated: $out"
-          done
+          pip install jupytext pyyaml
+          python convert_to_ipynb.py
 
       # Runs a set of commands using the runner's shell
       # https://mystmd.org/guide/deployment-github-pages#fn-except-custom-domains

AGENTS.md

@@ -12,6 +12,7 @@
 |------|---------|
 | `basic_lessons/` | Canonical source: MyST text notebooks (`.md` with `{code-cell}` directives) — 6 tutorials + 5 exercise answer keys |
 | `basic_lessons/.gitignore` | Excludes generated `.ipynb` files (produced at build time) |
+| `convert_to_ipynb.py` | Builds the downloadable `.ipynb` files: `.md` → intermediate `.md` (LaTeX macros expanded) → `.ipynb` |
 | `other/` | Supplementary content (e.g. `dqrobotics.md`) |
 | `myst.yml` | MyST project config (root): LaTeX macros, TOC, site options |
 | `build_html.sh` | Build script for `jupyter-book` |
@@ -59,18 +60,24 @@ x = np.array([1, 2, 3])
 
 ### Downloadable `.ipynb` from `.md` notebooks
 
-The `basic_lessons/` `.md` files are the canonical source. `.ipynb` files are generated at build time so visitors can download them:
+The `basic_lessons/` `.md` files are the canonical source. `.ipynb` files are generated at build time so visitors can download them. The conversion is a **two-step pipeline** run by `convert_to_ipynb.py`:
 
-1. **CI pipeline** (`.github/workflows/notebook_to_html.yml`) runs `jupytext --from md:myst --to notebook` before the MyST build, converting each `basic_lessons/*.md` → `basic_lessons/*.ipynb`.
-2. **`myst.yml` TOC** references the generated `.ipynb` for the lesson section — MyST renders these identically to the `.md` but provides native "Download notebook" buttons.
-3. **`basic_lessons/.gitignore`** excludes `.ipynb` so only `.md` is tracked in git.
+```
+basic_lessons/*.md  ->  <stem>_expanded.md  (LaTeX macros expanded)  ->  basic_lessons/*.ipynb
+```
+
+1. **CI pipeline** (`.github/workflows/notebook_to_html.yml`) runs `python convert_to_ipynb.py` before the MyST build.
+2. **Why the intermediate step?** The lessons use custom LaTeX macros (e.g. `\myvec{q}`) that MyST expands at build time by passing them to KaTeX. Most standalone `.ipynb` renderers (JupyterLab, VS Code, nbviewer, ...) do **not** know these macros and would show them literally. So `convert_to_ipynb.py` first expands every macro to its definition from `myst.yml` (`project.math`), writing a temporary `<stem>_expanded.md`; `jupytext` then converts that expanded file to `.ipynb`. The generated `.ipynb` therefore contains **only standard LaTeX** and renders anywhere. You keep writing the original `.md` with the convenient macros.
+3. **`myst.yml` TOC** references the generated `.ipynb` for the lesson section — MyST renders these identically to the `.md` but provides native "Download notebook" buttons.
+4. **`basic_lessons/.gitignore`** excludes `.ipynb` so only `.md` is tracked in git.
+
+The intermediate `<stem>_expanded.md` files are also build artifacts — deleted after conversion (use `--keep` to inspect them). They are not tracked and not referenced in `myst.yml`.
 
 To generate locally (e.g. for testing):
 ```bash
-pip install jupytext
-for f in basic_lessons/lesson*_tutorial.md basic_lessons/lesson*_exercise_answers.md; do
-  python -m jupytext --from md:myst --to notebook --output "${f%.md}.ipynb" "$f"
-done
+pip install jupytext pyyaml
+python convert_to_ipynb.py          # all lessons
+python convert_to_ipynb.py --keep   # keep the intermediate _expanded.md files
 ```
 
 ### Image references
@@ -101,10 +108,8 @@ pip install jupyter-book --pre
 **jupyter-book build (CI pipeline):**
 ```bash
 # Step 1: Generate .ipynb from .md (required for download buttons)
-pip install jupytext
-for f in basic_lessons/lesson*_tutorial.md basic_lessons/lesson*_exercise_answers.md; do
-  python -m jupytext --from md:myst --to notebook --output "${f%.md}.ipynb" "$f"
-done
+pip install jupytext pyyaml
+python convert_to_ipynb.py
 
 # Step 2: Build the site
 chmod +x build_html.sh
@@ -142,7 +147,7 @@ warning classes seen so far (all fixed in #11):
 ### CI/CD
 
 The GitHub Actions workflow (`.github/workflows/notebook_to_html.yml`) runs on pushes to `main` and on pull requests:
-1. Generates `.ipynb` from `.md` using jupytext
+1. Generates `.ipynb` from `.md` via `python convert_to_ipynb.py` (expanding LaTeX macros into an intermediate `.md` first)
 2. Runs `./build_html.sh` (jupyter-book pipeline)
 3. Uploads `_build/html/` as Pages artifact
 4. Deploys to GitHub Pages
@@ -174,6 +179,12 @@ Thank you! Please report it at https://github.com/MarinhoLab/OpenExecutableBooks
 - `\quat{}` for quaternions
 - `\dual{}` for dual numbers
 
+These are defined in `myst.yml` under `project.math` (KaTeX `#1` substitutions) and
+are expanded by MyST when building the site. Use them freely in the `.md` source —
+`convert_to_ipynb.py` expands them into the downloadable `.ipynb` so the notebooks
+also render in standalone viewers. **Code cells are left untouched** (a macro name
+in a Python comment/variable is never expanded).
+
 ### Image references:
 - Use relative paths: `![alt](Lesson4.png)` (relative to `basic_lessons/`)
 - Images live alongside the lesson files in `basic_lessons/`.
@@ -193,6 +204,7 @@ Thank you! Please report it at https://github.com/MarinhoLab/OpenExecutableBooks
 
 ### Files excluded from version control:
 - `basic_lessons/*.ipynb` — Generated at build time from `.md` files (via `basic_lessons/.gitignore`)
+- `basic_lessons/*_expanded.md` — transient intermediate files from `convert_to_ipynb.py` (deleted after conversion; `--keep` retains them for inspection)
 
 There is intentionally no root `.gitignore`. If you create local `venv/` or `_build/`
 directories, keep them out of commits (e.g. via `.git/info/exclude`).
@@ -207,6 +219,6 @@ directories, keep them out of commits (e.g. via `.git/info/exclude`).
 4. Use `{code-cell}` directives for Python code blocks
 5. Use `%%capture` on `%pip install` cells to suppress output
 6. Update `myst.yml` — add new file(s) to `project.toc` list as `.ipynb` (generated at build time)
-7. Update `basic_lessons/README.md` — add the new lesson to the contents table
-8. Test: `./build_html.sh` from the repository root
+7. Update `basic_lessons/README.md` — add the new lesson to the contents table, linking to the generated `.ipynb` (not the `.md`)
+8. Test: run `python convert_to_ipynb.py` to regenerate the `.ipynb`, then `./build_html.sh` from the repository root
 9. Open PR with descriptive title and body

basic_lessons/README.md

@@ -15,19 +15,19 @@ The reader is expected to follow it sequentially.
 
 | Number | Title and Link               | Content                                                                                                                                                                                                                                    |
 |--------|------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| 0      | [](./lesson0_tutorial.md)    | Setting up the virtual environment and installing all required dependencies.                                                                                                                                                               |
-| 1      | [](./lesson1_tutorial.md)    | Basic operations in Python and `numpy`                                                                                                                                                                                                     |
-| 2      | [](./lesson2_tutorial.md)    | Learn about elements and operations in $\mathbb{R}^n$, $SO(n)$, and $SE(n)$ with $n\in{\{2,3\}}$ related to positions, orientations, and poses, respectively.                                                                              |
-| 3      | [](./lesson3_tutorial.md)    | Learn about the composition of rigid body motion in series to obtain the forward kinematics model of a robotic manipulator, mapping their configuration space $\myvec{q}\in\mathbb{R}^n$ into their task space $\myvec{x}\in\mathbb{R}^m$. |
-| 4      | [](./lesson4_tutorial.md)    | Learn about the first-order differential mapping $\dot{\myvec{x}}=\mymatrix{J}\dot{\myvec{q}}$ between joint space and task space velocities through the calculation of the Jacobian $\mymatrix{J}$.                                       |
-| 5      | [](./lesson5_tutorial.md)    | Employ the previous knowledge in all previous lessons to employ a Lyapunov-stable control law to move a manipulator   in task space using configuration-space signals.                                                                     |
+| 0      | [](./lesson0_tutorial.ipynb) | Setting up the virtual environment and installing all required dependencies.                                                                                                                                                               |
+| 1      | [](./lesson1_tutorial.ipynb) | Basic operations in Python and `numpy`                                                                                                                                                                                                     |
+| 2      | [](./lesson2_tutorial.ipynb) | Learn about elements and operations in $\mathbb{R}^n$, $SO(n)$, and $SE(n)$ with $n\in{\{2,3\}}$ related to positions, orientations, and poses, respectively.                                                                              |
+| 3      | [](./lesson3_tutorial.ipynb) | Learn about the composition of rigid body motion in series to obtain the forward kinematics model of a robotic manipulator, mapping their configuration space $\myvec{q}\in\mathbb{R}^n$ into their task space $\myvec{x}\in\mathbb{R}^m$. |
+| 4      | [](./lesson4_tutorial.ipynb) | Learn about the first-order differential mapping $\dot{\myvec{x}}=\mymatrix{J}\dot{\myvec{q}}$ between joint space and task space velocities through the calculation of the Jacobian $\mymatrix{J}$.                                       |
+| 5      | [](./lesson5_tutorial.ipynb) | Employ the previous knowledge in all previous lessons to employ a Lyapunov-stable control law to move a manipulator   in task space using configuration-space signals.                                                                     |
 
 ### Exercise Answers
 
 | Lesson | Link |
 |--------|------|
-| L1 | [](./lesson1_exercise_answers.md) |
-| L2 | [](./lesson2_exercise_answers.md) |
-| L3 | [](./lesson3_exercise_answers.md) |
-| L4 | [](./lesson4_exercise_answers.md) |
-| L5 | [](./lesson5_exercise_answers.md) |
+| L1 | [](./lesson1_exercise_answers.ipynb) |
+| L2 | [](./lesson2_exercise_answers.ipynb) |
+| L3 | [](./lesson3_exercise_answers.ipynb) |
+| L4 | [](./lesson4_exercise_answers.ipynb) |
+| L5 | [](./lesson5_exercise_answers.ipynb) |