Add OPD training example

[atoniolo76]

Checklist

• Example is documented with comments throughout, in a Literate Programming style.
• Example does not require third-party dependencies to be installed locally
• Example pins its dependencies
  • Example pins container images to a stable tag, not a dynamic tag like latest
  • Example specifies a python_version for the base image, if it is used
  • Example pins all dependencies to at least minor version, ~=x.y.z or ==x.y
  • Example dependencies with version < 1 are pinned to patch version, ==0.y.z

Outside contributors

You're great! Thanks for your contribution.

[devin-ai-integration]

Review

Nice tutorial — the OPD walkthrough is a solid addition. A few things to address before merge:

Bug

Source docstring says 004_ but file is 003_

"""Tutorial source for `004_on_policy_distillation` — parsed by generate_tutorial.py."""

Should be 003_on_policy_distillation.

Typos (in source, propagated to all generated files)

Typo	Fix	Occurrences
determnistic	deterministic	2 (dataset intro, reward section)
paramter	parameter	1 (next steps)
models vocabularies	models' vocabularies	1 (next steps)
log ()P(x) / Q(x))	log(P(x) / Q(x))	1 (reward section — stray open paren)

Conceptual: KL explanation contradicts itself

The reward function markdown says:

we use the "reverse" KL divergence where P is the student's probability distribution, allowing us to apply a teacher penalty to modes only valued by the student model.

Then immediately:

In the "reverse" case, where P is the teacher's probability distribution, we apply a penalty to modes that the teacher thinks is valuable but may be out of distribution for the student model.

These say opposite things about which distribution is P in the "reverse" case. The second sentence reads like a description of the forward KL, not the reverse. Worth clarifying — the advantage formula in the intro (log π_student − log π_teacher) is KL(student ‖ teacher), which is the conventional "reverse" KL when the teacher is treated as the reference. One clean sentence would do.

Nit

Missing newline at end of tutorials/tutorial_generator/rl/003_on_policy_distillation.py.

PR checklist

The checklist items are all unchecked — might want to tick the ones that apply (literate-programming style, dependency pinning, etc.).

[joyliu-q]

Typo: 003

[joyliu-q]

Devin found some typos but otherwise this lgtm!