Skip to content

feat: link contents to .ipynb and expand LaTeX macros before conversion #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mmmarinho merged 1 commit into from
Aug 19, 2026
Merged

feat: link contents to .ipynb and expand LaTeX macros before conversion #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 6 additions & 8 deletions .github/workflows/notebook_to_html.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,18 +33,16 @@ jobs:
# Checks out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v4

# Convert basic_lessons/*.md → basic_lessons/*.ipynb so the site provides "Download notebook" buttons
# Convert basic_lessons/*.md → basic_lessons/*.ipynb so the site provides "Download notebook" buttons.
# The .md uses custom LaTeX macros (see `project.math` in myst.yml) that most .ipynb
# renderers cannot draw, so convert_to_ipynb.py first expands them into an intermediate
# .md (which is deleted afterwards) and then runs jupytext on that.
# https://jupytext.readthedocs.io/ — supports md:myst format natively
# Must run BEFORE build so myst/jupyter-book can pick up the generated notebooks
- name: Generate downloadable notebooks
run: |
pip install jupytext
for f in basic_lessons/lesson*_tutorial.md basic_lessons/lesson*_exercise_answers.md; do
[ -f "$f" ] || continue
out="${f%.md}.ipynb"
python -m jupytext --from md:myst --to notebook --output "$out" "$f"
echo "Generated: $out"
done
pip install jupytext pyyaml
python convert_to_ipynb.py

# Runs a set of commands using the runner's shell
# https://mystmd.org/guide/deployment-github-pages#fn-except-custom-domains
Expand Down
42 changes: 27 additions & 15 deletions AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
|------|---------|
| `basic_lessons/` | Canonical source: MyST text notebooks (`.md` with `{code-cell}` directives) — 6 tutorials + 5 exercise answer keys |
| `basic_lessons/.gitignore` | Excludes generated `.ipynb` files (produced at build time) |
| `convert_to_ipynb.py` | Builds the downloadable `.ipynb` files: `.md` → intermediate `.md` (LaTeX macros expanded) → `.ipynb` |
| `other/` | Supplementary content (e.g. `dqrobotics.md`) |
| `myst.yml` | MyST project config (root): LaTeX macros, TOC, site options |
| `build_html.sh` | Build script for `jupyter-book` |
Expand Down Expand Up @@ -59,18 +60,24 @@ x = np.array([1, 2, 3])

### Downloadable `.ipynb` from `.md` notebooks

The `basic_lessons/` `.md` files are the canonical source. `.ipynb` files are generated at build time so visitors can download them:
The `basic_lessons/` `.md` files are the canonical source. `.ipynb` files are generated at build time so visitors can download them. The conversion is a **two-step pipeline** run by `convert_to_ipynb.py`:

1. **CI pipeline** (`.github/workflows/notebook_to_html.yml`) runs `jupytext --from md:myst --to notebook` before the MyST build, converting each `basic_lessons/*.md` → `basic_lessons/*.ipynb`.
2. **`myst.yml` TOC** references the generated `.ipynb` for the lesson section — MyST renders these identically to the `.md` but provides native "Download notebook" buttons.
3. **`basic_lessons/.gitignore`** excludes `.ipynb` so only `.md` is tracked in git.
```
basic_lessons/*.md -> <stem>_expanded.md (LaTeX macros expanded) -> basic_lessons/*.ipynb
```

1. **CI pipeline** (`.github/workflows/notebook_to_html.yml`) runs `python convert_to_ipynb.py` before the MyST build.
2. **Why the intermediate step?** The lessons use custom LaTeX macros (e.g. `\myvec{q}`) that MyST expands at build time by passing them to KaTeX. Most standalone `.ipynb` renderers (JupyterLab, VS Code, nbviewer, ...) do **not** know these macros and would show them literally. So `convert_to_ipynb.py` first expands every macro to its definition from `myst.yml` (`project.math`), writing a temporary `<stem>_expanded.md`; `jupytext` then converts that expanded file to `.ipynb`. The generated `.ipynb` therefore contains **only standard LaTeX** and renders anywhere. You keep writing the original `.md` with the convenient macros.
3. **`myst.yml` TOC** references the generated `.ipynb` for the lesson section — MyST renders these identically to the `.md` but provides native "Download notebook" buttons.
4. **`basic_lessons/.gitignore`** excludes `.ipynb` so only `.md` is tracked in git.

The intermediate `<stem>_expanded.md` files are also build artifacts — deleted after conversion (use `--keep` to inspect them). They are not tracked and not referenced in `myst.yml`.

To generate locally (e.g. for testing):
```bash
pip install jupytext
for f in basic_lessons/lesson*_tutorial.md basic_lessons/lesson*_exercise_answers.md; do
python -m jupytext --from md:myst --to notebook --output "${f%.md}.ipynb" "$f"
done
pip install jupytext pyyaml
python convert_to_ipynb.py # all lessons
python convert_to_ipynb.py --keep # keep the intermediate _expanded.md files
```

### Image references
Expand Down Expand Up @@ -101,10 +108,8 @@ pip install jupyter-book --pre
**jupyter-book build (CI pipeline):**
```bash
# Step 1: Generate .ipynb from .md (required for download buttons)
pip install jupytext
for f in basic_lessons/lesson*_tutorial.md basic_lessons/lesson*_exercise_answers.md; do
python -m jupytext --from md:myst --to notebook --output "${f%.md}.ipynb" "$f"
done
pip install jupytext pyyaml
python convert_to_ipynb.py

# Step 2: Build the site
chmod +x build_html.sh
Expand Down Expand Up @@ -142,7 +147,7 @@ warning classes seen so far (all fixed in #11):
### CI/CD

The GitHub Actions workflow (`.github/workflows/notebook_to_html.yml`) runs on pushes to `main` and on pull requests:
1. Generates `.ipynb` from `.md` using jupytext
1. Generates `.ipynb` from `.md` via `python convert_to_ipynb.py` (expanding LaTeX macros into an intermediate `.md` first)
2. Runs `./build_html.sh` (jupyter-book pipeline)
3. Uploads `_build/html/` as Pages artifact
4. Deploys to GitHub Pages
Expand Down Expand Up @@ -174,6 +179,12 @@ Thank you! Please report it at https://github.com/MarinhoLab/OpenExecutableBooks
- `\quat{}` for quaternions
- `\dual{}` for dual numbers

These are defined in `myst.yml` under `project.math` (KaTeX `#1` substitutions) and
are expanded by MyST when building the site. Use them freely in the `.md` source —
`convert_to_ipynb.py` expands them into the downloadable `.ipynb` so the notebooks
also render in standalone viewers. **Code cells are left untouched** (a macro name
in a Python comment/variable is never expanded).

### Image references:
- Use relative paths: `![alt](Lesson4.png)` (relative to `basic_lessons/`)
- Images live alongside the lesson files in `basic_lessons/`.
Expand All @@ -193,6 +204,7 @@ Thank you! Please report it at https://github.com/MarinhoLab/OpenExecutableBooks

### Files excluded from version control:
- `basic_lessons/*.ipynb` — Generated at build time from `.md` files (via `basic_lessons/.gitignore`)
- `basic_lessons/*_expanded.md` — transient intermediate files from `convert_to_ipynb.py` (deleted after conversion; `--keep` retains them for inspection)

There is intentionally no root `.gitignore`. If you create local `venv/` or `_build/`
directories, keep them out of commits (e.g. via `.git/info/exclude`).
Expand All @@ -207,6 +219,6 @@ directories, keep them out of commits (e.g. via `.git/info/exclude`).
4. Use `{code-cell}` directives for Python code blocks
5. Use `%%capture` on `%pip install` cells to suppress output
6. Update `myst.yml` — add new file(s) to `project.toc` list as `.ipynb` (generated at build time)
7. Update `basic_lessons/README.md` — add the new lesson to the contents table
8. Test: `./build_html.sh` from the repository root
7. Update `basic_lessons/README.md` — add the new lesson to the contents table, linking to the generated `.ipynb` (not the `.md`)
8. Test: run `python convert_to_ipynb.py` to regenerate the `.ipynb`, then `./build_html.sh` from the repository root
9. Open PR with descriptive title and body
22 changes: 11 additions & 11 deletions basic_lessons/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,19 @@ The reader is expected to follow it sequentially.

| Number | Title and Link | Content |
|--------|------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0 | [](./lesson0_tutorial.md) | Setting up the virtual environment and installing all required dependencies. |
| 1 | [](./lesson1_tutorial.md) | Basic operations in Python and `numpy` |
| 2 | [](./lesson2_tutorial.md) | Learn about elements and operations in $\mathbb{R}^n$, $SO(n)$, and $SE(n)$ with $n\in{\{2,3\}}$ related to positions, orientations, and poses, respectively. |
| 3 | [](./lesson3_tutorial.md) | Learn about the composition of rigid body motion in series to obtain the forward kinematics model of a robotic manipulator, mapping their configuration space $\myvec{q}\in\mathbb{R}^n$ into their task space $\myvec{x}\in\mathbb{R}^m$. |
| 4 | [](./lesson4_tutorial.md) | Learn about the first-order differential mapping $\dot{\myvec{x}}=\mymatrix{J}\dot{\myvec{q}}$ between joint space and task space velocities through the calculation of the Jacobian $\mymatrix{J}$. |
| 5 | [](./lesson5_tutorial.md) | Employ the previous knowledge in all previous lessons to employ a Lyapunov-stable control law to move a manipulator in task space using configuration-space signals. |
| 0 | [](./lesson0_tutorial.ipynb) | Setting up the virtual environment and installing all required dependencies. |
| 1 | [](./lesson1_tutorial.ipynb) | Basic operations in Python and `numpy` |
| 2 | [](./lesson2_tutorial.ipynb) | Learn about elements and operations in $\mathbb{R}^n$, $SO(n)$, and $SE(n)$ with $n\in{\{2,3\}}$ related to positions, orientations, and poses, respectively. |
| 3 | [](./lesson3_tutorial.ipynb) | Learn about the composition of rigid body motion in series to obtain the forward kinematics model of a robotic manipulator, mapping their configuration space $\myvec{q}\in\mathbb{R}^n$ into their task space $\myvec{x}\in\mathbb{R}^m$. |
| 4 | [](./lesson4_tutorial.ipynb) | Learn about the first-order differential mapping $\dot{\myvec{x}}=\mymatrix{J}\dot{\myvec{q}}$ between joint space and task space velocities through the calculation of the Jacobian $\mymatrix{J}$. |
| 5 | [](./lesson5_tutorial.ipynb) | Employ the previous knowledge in all previous lessons to employ a Lyapunov-stable control law to move a manipulator in task space using configuration-space signals. |

### Exercise Answers

| Lesson | Link |
|--------|------|
| L1 | [](./lesson1_exercise_answers.md) |
| L2 | [](./lesson2_exercise_answers.md) |
| L3 | [](./lesson3_exercise_answers.md) |
| L4 | [](./lesson4_exercise_answers.md) |
| L5 | [](./lesson5_exercise_answers.md) |
| L1 | [](./lesson1_exercise_answers.ipynb) |
| L2 | [](./lesson2_exercise_answers.ipynb) |
| L3 | [](./lesson3_exercise_answers.ipynb) |
| L4 | [](./lesson4_exercise_answers.ipynb) |
| L5 | [](./lesson5_exercise_answers.ipynb) |
Loading
Loading