Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 10 additions & 108 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,124 +2,26 @@

Live training curves in Jupyter, Colab and the VS Code / Cursor interactive window, with a tqdm bar underneath, at (almost) no cost to the training loop.

[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ARENA-education/liveplot/blob/demo/examples/demo.ipynb) [![tests](https://github.com/ARENA-education/liveplot/actions/workflows/tests.yml/badge.svg)](https://github.com/ARENA-education/liveplot/actions/workflows/tests.yml)

Documentation: **[arena-education.github.io/liveplot](https://arena-education.github.io/liveplot/)** (this README, the API, and the examples; built from the `.md` files on every push).

Try it in Colab with the badge above: that notebook is [`examples/demo.py`](examples/demo.py), a cell-by-cell tour of the features, which CI converts with jupytext and publishes to the `demo` branch on every push to `main`.
[![docs](https://img.shields.io/badge/docs-arena--education.github.io%2Fliveplot-blue)](https://arena-education.github.io/liveplot/) [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ARENA-education/liveplot/blob/demo/examples/demo.ipynb) [![tests](https://github.com/ARENA-education/liveplot/actions/workflows/tests.yml/badge.svg)](https://github.com/ARENA-education/liveplot/actions/workflows/tests.yml)

![training loss every step, eval loss and accuracy every 50 steps, on one panel with two y-axes](docs/demo.gif)

The GIF is cell 2 of [`examples/demo.py`](examples/demo.py), a tour of the features in a cell-separated file: open it in the VS Code / Cursor interactive window and run it cell by cell. That cell trains a tiny numpy classifier, logging the training loss every step and the eval loss and accuracy every 50 steps, laid out as `"loss | eval_loss eval_acc"`. The GIF itself was recorded by the library: `examples/make_gif.py` runs the same loop as a plain script with `record="docs/demo.gif"`.

```python
from liveplot import LivePlot

plot = LivePlot(range(num_steps)) # a tqdm bar + a live plot
for step in plot:
loss, acc = train_step()
plot.log(loss=loss, acc=acc) # step is implicit; the bar's postfix shows the latest values
```

Metrics are discovered from what you log. With no layout given they all share one panel, with a legend. To split them up, give panel strings:

```python
plot = LivePlot(range(num_steps), "loss", "return | entropy", "lossD lossG | acc")
```

Each string is one panel. Names separated by spaces share the left y-axis; names after a `|` go on a right-hand y-axis. Anything you log that no string mentions gets a panel of its own. Every panel has a legend.

For nested loops, create the plot once and wrap the inner loop with `plot(...)`; the count, and so the x-axis, continues across epochs and you get one tqdm bar per epoch, as with tqdm:

```python
plot = LivePlot("loss", "acc", total=epochs * len(loader))
for epoch in range(epochs):
for imgs, labels in plot(loader, desc=f"epoch {epoch}"): # kwargs go to tqdm
plot.log(loss=train_step(imgs, labels))
plot.log(acc=evaluate()) # metrics can have different cadences
plot.finish() # or wrap the whole thing in `with LivePlot(...) as plot:`
```

If you'd rather supply the x values yourself, use the context manager and pass the step to `log`:

```python
with LivePlot("loss", "acc", total=num_steps) as plot:
for step in range(num_steps):
plot.log(step, loss=train_step())
```

Already have a tqdm bar? Pass it as the iterable and it is reused instead of wrapped: `LivePlot(tqdm(loader), "loss")`.

## Metrics logged at different rates

Nothing special is needed. Every metric keeps its own list of `(x, value)` points, and each `log` call stamps its metrics with the current x. A metric logged every step gets a point per step; one logged every 50 steps, or once per epoch after the inner loop, gets a point wherever the count was at the time and is drawn as a line through those points. So in `"loss | eval_loss eval_acc"` the left axis fills in continuously while the right axis grows a point every 50 steps, all against the same x.

## The x-axis

It follows tqdm: the plot counts items consumed and never looks at their values.

x = initial + n * unit_scale

`n` is 0 inside the loop body for the first item, like `for step in range(N)`. `initial` shifts the start (`initial=1` for 1-based, `initial=10` to plot `range(10, 20)` at its values). `unit` and `unit_scale` relabel and rescale it: `unit="examples", unit_scale=batch_size` plots against examples seen, and the tqdm bar shows the same numbers. `total`, in items like tqdm's, fixes the x range; the single-loop form takes it from `len(iterable)`. An explicit `plot.log(step, ...)` uses that x for that call only, like wandb's `step=`.

## Install

```
pip install git+https://github.com/ARENA-education/liveplot.git
```

Only `matplotlib` is required. `ipython` is needed for the live display and `tqdm` for the bar; any notebook has both, and without them the plot silently just collects `plot.data`.

## How it works

The training thread only appends numbers (about 40 µs per `log`). A separate render process owns the matplotlib figure, redraws it at most once per `refresh_seconds` (default 1; `0` means on every arrival), and sends back PNG bytes that get swapped into a fixed output cell. The output is a plain image, so it behaves identically in Jupyter, Colab, VS Code and Cursor: no widgets, no JavaScript, no CDN.

Interrupting the cell is safe. Jupyter sends its interrupt to every process the kernel started; the render process ignores it, so you get a frozen plot with `plot.data` intact. A plot that is dropped without `finish()` shuts its process down when garbage collected, and the process exits by itself if the notebook kernel dies.

## Options

| | |
|---|---|
| `total`, `initial`, `unit`, `unit_scale` | tqdm's arguments, with tqdm's meaning; they define the x-axis (see above) |
| `refresh_seconds` | minimum time between redraws (default 1.0). Points arriving in between are batched into the next frame. `0` redraws whenever new data arrives, as fast as rendering allows (roughly 0.15 s per frame at a few thousand points), and costs nothing while idle. |
| `max_cols`, `rows`, `cols` | grid shape; `max_cols=None` gives a near-square grid |
| `progress`, `desc` | disable the bundled tqdm bars, or give the single-loop form's bar a description |
| `cell_size`, `dpi` | size of each panel in inches, and PNG resolution |
| `record` | `True` keeps every rendered frame in `plot.frames`; a path such as `"run.gif"` also writes an animated GIF at `finish()`. `plot.save_gif(path, speedup=1.0)` does it on demand. Works outside a notebook too. |

For labels or fixed ranges, use a dict instead of a string for that panel:

```python
{"metrics": ["acc"], "ylim": (0, 1), "ylabel": "test accuracy", "xlabel": "epoch"}
```

Allowed keys: `title`, `metrics`, `secondary`, `xlabel`, `ylabel`, `ylabel2`, `xlim`, `ylim`, `ylim2`, `axhlines`, `axhlines2`, `smooth`, `yscale`, `yscale2`. Where a key has a matplotlib counterpart it uses matplotlib's name (`ax.set(title=..., xlabel=..., xlim=..., yscale=...)`); the `2` suffix means the right-hand axis.

## Reference lines

The names follow matplotlib. A dashed horizontal line with a legend entry, for the level a curve should reach or beat:

```python
LivePlot(range(N), {"metrics": ["loss"], "axhlines": {"uniform": math.log(d_vocab), "unigram": 7.35}})
plot.axhline(500, "solved", metric="return") # at run time; goes on the axis of that metric's panel
plot.axvline(label="lr drop") # a dotted vertical line on every panel at the current x
plot.axhline(0.9, "target", metric="acc", color="red", linestyle="-") # extra kwargs go to the matplotlib artist
```

In a panel dict, `axhlines` takes `{label: y}`, a list of values, or a list of `axhline` kwargs such as `dict(y=0.9, label="target", color="red")`; `axhlines2` is the same for the right-hand axis.

## Smoothing and log axes

Per-step losses are noisy. `smooth=0.9` on a panel draws each of its curves through wandb's default smoothing, the time-weighted exponential moving average, with the same 0 to 1 weight as wandb's smoothing slider and the raw values faded behind. `LivePlot(..., smooth=0.9)` makes that the default for every panel, and `"smooth": 0` on a panel opts out. `yscale="log"` (and `yscale2` for the right axis) gives a log axis.
from liveplot import LivePlot

```python
LivePlot(range(N), {"metrics": ["loss"], "smooth": 0.9, "yscale": "log"}, "acc")
plot = LivePlot(loader, "loss | acc", "lr") # wrap the loop like tqdm; "|" puts acc on a right-hand axis
plot["acc"].set_ylim(0, 1) # configure like matplotlib
for batch in plot:
loss, acc = train_step(batch)
plot.log(loss=loss, acc=acc, lr=lr) # log like wandb
```

`plot.figure()` returns a matplotlib Figure of the plot as it stands, built independently of the live renderer, for `fig.savefig(...)`, a title, or any other tweak.

`plot.log` accepts keywords, an explicit step (`plot.log(step, loss=...)`), or a dict (`plot.log(step, {"loss": ...})`). Values can be anything `float()` accepts, including one-element tensors. `plot.data` holds the full history as `{metric: (steps, values)}` and `plot.latest` the most recent value of each.
It wraps iterables like tqdm, logs like wandb and is configured like matplotlib, so the names are ones you already know. Rendering happens in a separate process and the output is a plain image, so it behaves the same everywhere with no widgets or JavaScript, and interrupting a cell is safe.

## Credits
**[Documentation](https://arena-education.github.io/liveplot/)**: the [guide](https://arena-education.github.io/liveplot/guide/) (nested loops, the x-axis, reference lines, smoothing, recording GIFs), the [API](https://arena-education.github.io/liveplot/api/), and the [examples](https://arena-education.github.io/liveplot/examples/). The tour in [`examples/demo.py`](examples/demo.py) runs cell by cell in the interactive window or [in Colab](https://colab.research.google.com/github/ARENA-education/liveplot/blob/demo/examples/demo.ipynb).

The per-panel label/limit options and the grid-shape rule are adapted from Tyler Lum's [live_plotter](https://github.com/tylerlum/live_plotter) (MIT); see `THIRD_PARTY_LICENSES.md`.
MIT. Per-panel option names and the grid rule are adapted from Tyler Lum's [live_plotter](https://github.com/tylerlum/live_plotter); see `THIRD_PARTY_LICENSES.md`.
Binary file added docs/axes_api.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
117 changes: 117 additions & 0 deletions docs/guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Guide

The one-liner first, then everything it can grow into.

```python
from liveplot import LivePlot

plot = LivePlot(range(num_steps)) # a tqdm bar + a live plot
for step in plot:
loss, acc = train_step()
plot.log(loss=loss, acc=acc) # step is implicit; the bar's postfix shows the latest values
```

Metrics are discovered from what you log. With no layout given they all share one panel, with a legend. To split them up, give panel strings:

```python
plot = LivePlot(range(num_steps), "loss", "return | entropy", "lossD lossG | acc")
```

Each string is one panel. Names separated by spaces share the left y-axis; names after a `|` go on a right-hand y-axis. Anything you log that no string mentions gets a panel of its own. Every panel has a legend.

For nested loops, create the plot once and wrap the inner loop with `plot(...)`; the count, and so the x-axis, continues across epochs and you get one tqdm bar per epoch, as with tqdm:

```python
plot = LivePlot("loss", "acc", total=epochs * len(loader))
for epoch in range(epochs):
for imgs, labels in plot(loader, desc=f"epoch {epoch}"): # kwargs go to tqdm
plot.log(loss=train_step(imgs, labels))
plot.log(acc=evaluate()) # metrics can have different cadences
plot.finish() # or wrap the whole thing in `with LivePlot(...) as plot:`
```

If you'd rather supply the x values yourself, use the context manager and pass the step to `log`:

```python
with LivePlot("loss", "acc", total=num_steps) as plot:
for step in range(num_steps):
plot.log(step, loss=train_step())
```

Already have a tqdm bar? Pass it as the iterable and it is reused instead of wrapped: `LivePlot(tqdm(loader), "loss")`.

## Metrics logged at different rates

Nothing special is needed. Every metric keeps its own list of `(x, value)` points, and each `log` call stamps its metrics with the current x. A metric logged every step gets a point per step; one logged every 50 steps, or once per epoch after the inner loop, gets a point wherever the count was at the time and is drawn as a line through those points. So in `"loss | eval_loss eval_acc"` the left axis fills in continuously while the right axis grows a point every 50 steps, all against the same x.

## The x-axis

It follows tqdm: the plot counts items consumed and never looks at their values.

x = initial + n * unit_scale

`n` is 0 inside the loop body for the first item, like `for step in range(N)`. `initial` shifts the start (`initial=1` for 1-based, `initial=10` to plot `range(10, 20)` at its values). `unit` and `unit_scale` relabel and rescale it: `unit="examples", unit_scale=batch_size` plots against examples seen, and the tqdm bar shows the same numbers. `total`, in items like tqdm's, fixes the x range; the single-loop form takes it from `len(iterable)`. An explicit `plot.log(step, ...)` uses that x for that call only, like wandb's `step=`.

## How it works

The training thread only appends numbers (about 40 µs per `log`). A separate render process owns the matplotlib figure, redraws it at most once per `refresh_seconds` (default 1; `0` means on every arrival), and sends back PNG bytes that get swapped into a fixed output cell. The output is a plain image, so it behaves identically in Jupyter, Colab, VS Code and Cursor: no widgets, no JavaScript, no CDN.

Interrupting the cell is safe. Jupyter sends its interrupt to every process the kernel started; the render process ignores it, so you get a frozen plot with `plot.data` intact. A plot that is dropped without `finish()` shuts its process down when garbage collected, and the process exits by itself if the notebook kernel dies.

## Options

| | |
|---|---|
| `total`, `initial`, `unit`, `unit_scale` | tqdm's arguments, with tqdm's meaning; they define the x-axis (see above) |
| `refresh_seconds` | minimum time between redraws (default 1.0). Points arriving in between are batched into the next frame. `0` redraws whenever new data arrives, as fast as rendering allows (roughly 0.15 s per frame at a few thousand points), and costs nothing while idle. |
| `max_cols`, `rows`, `cols` | grid shape; `max_cols=None` gives a near-square grid |
| `progress`, `desc` | disable the bundled tqdm bars, or give the single-loop form's bar a description |
| `cell_size`, `dpi` | size of each panel in inches, and PNG resolution |
| `record` | `True` keeps every rendered frame in `plot.frames`; a path such as `"run.gif"` also writes an animated GIF at `finish()`. `plot.save_gif(path, speedup=1.0)` does it on demand. Works outside a notebook too. |

## Titles, labels, limits: matplotlib's names

Panels and axes are addressed and configured with the setters you already know from matplotlib, before or during the loop:

```python
plot = LivePlot(loader, "loss | acc", "lr")

plot["acc"].set_ylim(0, 1) # plot[metric] is the y-axis holding that metric (left or right)
plot["acc"].set_ylabel("test accuracy")
plot["loss"].axhline(0.1, label="target", color="red") # extra kwargs go to the matplotlib artist

plot.panels[0].set_title("training") # a panel: set_title, set_xlabel, set_xlim, set_smooth, axvline
plot.panels[0].set_xlim(0, 10_000)
plot.panels[0].axvline(label="lr drop") # at the current x

plot.set(xlabel="examples", smooth=0.9) # on the plot itself: every panel, like Axes.set(**kwargs)
```

`plot[metric]` finds the right-hand axis too, so there is no separate spelling for it; `plot.panels[i].left` / `.right` name the two axes explicitly. Everything on an axis takes matplotlib's name and signature: `set_ylabel`, `set_ylim` (two arguments or a tuple), `set_yscale("log")`, `axhline`, `set(**kwargs)`, and the panel's own setters are reachable from it as they would be on an `Axes`. A setter called mid-run just triggers a re-layout on the next frame.

The dict form, `{"metrics": ["acc"], "ylim": (0, 1), ...}`, is still accepted in the layout list but the setters are the intended way.

`plot.figure()` returns a matplotlib Figure of the plot as it stands, built independently of the live renderer, for `fig.savefig(...)`, a title, or any other tweak.

`plot.log` accepts keywords, an explicit step (`plot.log(step, loss=...)`), or a dict (`plot.log(step, {"loss": ...})`). Values can be anything `float()` accepts, including one-element tensors. `plot.data` holds the full history as `{metric: (steps, values)}` and `plot.latest` the most recent value of each.

## Reference lines

Matplotlib's `axhline` / `axvline`, on an axis, a panel, or the whole plot. Horizontal lines get a legend entry; extra kwargs go to the artist.

```python
plot["loss"].axhline(math.log(d_vocab), "uniform") # the level a curve should beat
plot["return"].axhline(500, "solved", color="green")
plot.axvline(label="lr drop") # every panel, at the current x
plot.panels[1].axvline(2000, "checkpoint", linestyle="-") # one panel, at a given x
```

## Smoothing and log axes

Per-step losses are noisy. `set_smooth(0.9)` draws each curve of a panel through wandb's default smoothing, the [time-weighted exponential moving average](https://docs.wandb.ai/models/app/features/panels/line-plot/smoothing), with the same 0 to 1 weight as wandb's smoothing slider and the raw values faded behind. On the plot it applies to every panel; `set_smooth(0)` turns it off. `set_yscale("log")` on an axis, a panel, or the plot gives log axes.

```python
plot.set_smooth(0.9)
plot["loss"].set_yscale("log")
```

11 changes: 9 additions & 2 deletions examples/demo.py
Original file line number Diff line number Diff line change
Expand Up @@ -101,11 +101,15 @@ def train_classifier(steps=600, batch_size=64, lr=0.5, eval_every=50, seed=0, pa
# 3. Nested loops (epochs x batches). Create the plot once and wrap the INNER loop with plot(...):
# the count, and so the x-axis, continues across epochs and you get one tqdm bar per epoch,
# exactly as with tqdm. `total` is in items, like tqdm's, and fixes the x range up front.
# A dict instead of a string pins the accuracy axis to [0, 1] and labels it.
# Titles, labels and limits use matplotlib's own setter names, addressed by metric or by panel.

if MAIN:
epochs, loader = 3, [None] * 60
plot = LivePlot("loss", {"metrics": ["acc"], "ylim": (0, 1), "ylabel": "test accuracy"}, total=epochs * len(loader))
plot = LivePlot("loss", "acc", total=epochs * len(loader))
plot["acc"].set_ylim(0, 1) # plot[metric] is the y-axis holding that metric
plot["acc"].set_ylabel("test accuracy")
plot.panels[0].set_title("training loss")
plot.set(xlabel="batches") # every panel, like Axes.set(**kwargs)
for epoch in range(epochs):
for batch in plot(loader, desc=f"epoch {epoch}"):
plot.log(loss=math.exp(-plot.n / 60) + 0.05 * random.random())
Expand All @@ -126,6 +130,9 @@ def train_classifier(steps=600, batch_size=64, lr=0.5, eval_every=50, seed=0, pa
plot.log(step, acc=min(1.0, step / 300))
if step >= 200:
plot.log(step, lr=1e-3 * (400 - step) / 200)
if step == 200:
plot.axvline(label="lr decay starts") # a vertical line on every panel, at the current x
plot["acc"].axhline(0.9, "target") # a horizontal reference line on acc's axis
slow()

# %%
Expand Down
Loading
Loading