Write the note at the length the README promises

CONTRIBUTING.md

@@ -45,6 +45,34 @@ editing one file rather than two.
 `mkdocs build --strict` will not catch that on its own: a page outside the
 navigation is an INFO line it prints before building the site anyway.
 
+## How long a note is
+
+Long enough to be read instead of the paper, not long enough to be read instead
+of thinking.
+
+The README promises a note is readable without the original open. That rules
+out the abstract-with-links shape, which is the one that comes out if you write
+quickly: a claim, a gesture at the evidence, a conclusion. It is a bookmark
+with paragraph breaks.
+
+A note that keeps the promise usually has to:
+
+- **explain the mechanism, not only the finding.** If the result depends on an
+  algorithm, the algorithm gets stated - the equation if there is one, and what
+  each term means. A reader should not have to already know it;
+- **carry the numbers and the conditions.** How many subjects, how many seeds,
+  which comparison, what size of effect. A result quoted without its conditions
+  cannot be argued with, which means it cannot be checked;
+- **take the objections one at a time.** Later work that complicates the
+  finding gets its own subsection and its own mechanism, not a sentence in a
+  list. Two papers disagreeing is the interesting part, not an aside;
+- **say what would falsify it, concretely.** "More work is needed" is not that.
+  Name the result that would undo each section.
+
+There is no word count, because a target invites padding. The test is whether a
+reader who has not opened the paper can follow the argument and find where it
+would break.
+
 ## Translations
 
 A translation sits beside its original as `name.ru.md`, not in a parallel tree,

docs/notes/dopamine-reward-prediction-error.md

@@ -6,8 +6,14 @@ topics: [neuroscience, reinforcement-learning, dopamine]
 sources:
   - title: "Schultz, Dayan & Montague (1997). A Neural Substrate of Prediction and Reward. Science 275(5306), 1593–1599"
     url: https://doi.org/10.1126/science.275.5306.1593
+  - title: "Sutton (1988). Learning to predict by the methods of temporal differences. Machine Learning 3, 9–44"
+    url: https://doi.org/10.1007/BF00115009
   - title: "Matsumoto & Hikosaka (2009). Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459, 837–841"
     url: https://doi.org/10.1038/nature08028
+  - title: "Howe et al. (2013). Prolonged dopamine signalling in striatum signals proximity and value of distant rewards. Nature 500, 575–579"
+    url: https://doi.org/10.1038/nature12475
+  - title: "Steinberg et al. (2013). A causal link between prediction errors, dopamine neurons and learning. Nature Neuroscience 16, 966–973"
+    url: https://doi.org/10.1038/nn.3413
   - title: "Dabney et al. (2020). A distributional code for value in dopamine-based reinforcement learning. Nature 577, 671–675"
     url: https://doi.org/10.1038/s41586-019-1924-6
 ---
@@ -16,70 +22,207 @@ sources:
 
 Midbrain dopamine neurons in the macaque fire in a pattern that looks like the
 error term of temporal-difference learning. The correspondence is close enough
-that an algorithm written in 1988 predicted what an electrode would record in
-1997. That is the reason this result matters, and also the reason it gets
-overstated.
+that an algorithm published in 1988 predicted, in outline, what an electrode
+would record in 1997. That is why the result matters, and it is also why it gets
+overstated: a striking match between a theory and a measurement invites the
+reader to collapse the two, and much of the work since has been about why they
+should not be collapsed.
 
-## What the paper claims
+This note is about what the 1997 recordings established, what they left open,
+and which later results actually move the conclusion rather than decorate it.
 
-Schultz, Dayan and Montague report three response patterns in dopamine neurons
-of monkeys learning a conditioned task:
+## The algorithm came first
 
-- An unexpected reward produces a burst.
-- Once a cue reliably predicts the reward, the burst moves to the cue, and
-  delivery of the reward itself produces nothing.
-- Expected reward withheld produces a dip below baseline, timed to the moment
-  the reward should have arrived.
+Temporal-difference learning was not derived from neuroscience. Sutton
+introduced it for a purely computational problem: how to learn to predict a
+delayed outcome without waiting for the outcome every time.
 
-Those three are exactly the signs of the TD error δ = r + γV(s′) − V(s):
-positive on surprise, zero once predicted, negative on omission. The paper's
-claim is that dopamine broadcasts this error, and that it is the teaching signal
-downstream structures learn from.
+The setup is an agent moving through states. It keeps an estimate `V(s)` of the
+total future reward available from state `s`. On each step it observes the
+immediate reward `r` and the next state `s′`, and forms the error
 
-## What the evidence shows
+```
+δ = r + γV(s′) − V(s)
+```
 
-The recordings support the shape of the correspondence: sign and timing move as
-the theory says they should. That is a strong result and it has replicated
-widely.
+where `γ` discounts future reward. The term `r + γV(s′)` is a fresh estimate of
+what `s` was worth, built from one step of real experience plus the current
+guess about everything after it. `V(s)` is the old estimate. Their difference is
+the surprise, and the algorithm nudges `V(s)` toward the fresh estimate in
+proportion to it.
 
-What the recordings do not establish is the equation. A signal that behaves like
-an RPE under these conditions is consistent with dopamine carrying an RPE; it
-does not rule out its carrying something correlated with one. The distinction
-survives because the manipulations here are correlational — neurons are recorded
-during learning, not silenced to see what learning does without them.
+Three properties of `δ` matter for what follows, and all three fall out of the
+definition rather than being bolted on:
 
-## What would change the reading
+**Positive on unexpected reward.** If reward arrives where none was predicted,
+`r` is large and `V(s)` is not, so `δ` is positive.
 
-Two later results already have.
+**Zero once prediction is accurate.** Once `V` has learned, the fresh estimate
+agrees with the old one and the difference vanishes. A perfectly predicted
+reward produces no error at all — the signal is about the *unpredicted* part,
+not about the reward.
 
-**Dopamine neurons are not one population.** Matsumoto and Hikosaka found
-neurons excited by aversive as well as appetitive stimuli, sitting anatomically
-apart from the value-coding ones. A single scalar teaching signal cannot account
-for both groups; at least one of them is coding something closer to salience.
+**It moves backwards in time.** When a cue reliably precedes reward, the cue's
+own value rises and the error appears at the cue instead of at the reward. This
+is not a separate rule; it follows from `γV(s′)` carrying value one step back on
+each visit.
 
-**The variability is not noise.** Dabney and colleagues showed that dopamine
-neurons differ systematically in how optimistically or pessimistically they
-respond, and that the spread across the population encodes a distribution over
-future reward rather than its mean. Under that reading, the 1997 result recorded
-the average of a population code and named it the code.
+## What the recordings showed
 
-Neither overturns the original observation. Both change what it is evidence for:
-the finding is that dopamine carries prediction-error-like information, not that
-dopamine is the RPE.
+Schultz, Dayan and Montague recorded single units in the midbrain of macaques
+learning conditioned tasks, and reported three patterns.
+
+**An unpredicted reward produces a burst.** Juice delivered without warning
+drives a short, sharp increase in firing above baseline.
+
+**Once a cue predicts the reward, the burst moves to the cue.** After
+conditioning, the same juice at the expected moment produces nothing. The burst
+has transferred to the earliest reliable predictor.
+
+**Reward withheld produces a dip.** If the cue appears and the reward does not,
+firing drops below baseline — and it drops at the moment the reward was due, not
+when the trial ends.
+
+Set those against the three properties above and they match one for one,
+including the detail hardest to get by accident: the timing of the dip. A neuron
+that simply reported "something good happened" has no reason to do anything at a
+moment when nothing happens. A neuron reporting `δ` has to, because at that
+moment `V(s)` predicts reward and the observation contradicts it.
+
+## What that establishes, and what it does not
+
+The correspondence is strong evidence that dopamine carries
+prediction-error-like information. It is not evidence that dopamine *is* the
+prediction error, and the distance between those two statements is where the
+later work lives.
+
+### The original design is correlational
+
+Neurons are recorded while an animal learns. Nothing is manipulated, so the
+design cannot separate a signal that *drives* learning from one that merely
+accompanies it. A signal correlated with `δ` is consistent with dopamine
+broadcasting the teaching signal, and equally consistent with dopamine reporting
+something that co-varies with it.
+
+That gap was closed later, and it is worth being precise about how, because this
+is the step that upgrades the story rather than repeating it.
+
+Steinberg and colleagues used a **blocking** paradigm. In blocking, a second cue
+introduced alongside an already-predictive first cue is normally *not* learned
+about: the first cue already predicts the outcome, so there is no error left to
+drive learning about the second. The theory's explanation for blocking is
+exactly that `δ` is zero.
+
+They then activated dopamine neurons optogenetically at the moment of reward —
+precisely where the theory says the error should be zero — and animals learned
+about the blocked cue anyway. An artificial `δ`, inserted where none existed,
+produced the learning that a real `δ` would have.
+
+That is causal, and it is a sharper test than it first appears: the manipulation
+does not make the animal generally more attentive or the reward more pleasant.
+It supplies one specific quantity at one specific moment, and the behaviour that
+follows is the one the equation predicts.
+
+### The population is treated as one signal
+
+The 1997 account speaks of dopamine neurons as if they report a single scalar,
+broadcast widely. Two later findings complicate that in different directions.
+
+## What changes the reading
+
+### The population is not homogeneous
+
+Matsumoto and Hikosaka found two groups of dopamine neurons that behave
+differently and sit in different places.
+
+One group is excited by reward-predicting cues and inhibited by aversive ones —
+the value-coding behaviour the RPE account expects. The other is excited by
+*both* appetitive and aversive events, which is not value at all. It looks
+closer to salience: "something important is happening", regardless of sign.
+
+The two groups are anatomically separable, with value-coding neurons
+concentrated more ventromedially and salience-like ones more dorsolaterally.
+
+This does not overturn the 1997 result — the value-coding group behaves as
+described. What it removes is the licence to speak of *the* dopamine signal. At
+least one substantial population is doing something else, and an experiment that
+samples the midbrain without regard to recording site will average across both
+and report the blend as if it were one thing.
+
+### The variability is not noise
+
+Dabney and colleagues asked what the *spread* across dopamine neurons means. On
+a scalar account, individual neurons are noisy copies of one quantity and the
+differences between them are measurement error, to be averaged away.
+
+The distributional account says otherwise. Neurons differ systematically in how
+asymmetrically they treat positive and negative errors: some amplify positive
+surprise relative to negative, others do the reverse. A neuron with that
+asymmetry does not converge on the mean of future reward. It converges on a
+**quantile** of the reward distribution, and which quantile is set by how
+strongly it weights each direction.
+
+A population with a spread of asymmetries therefore encodes a spread of
+quantiles — a representation of the whole distribution over future reward rather
+than of its expectation. The paper reports the asymmetry across recorded neurons
+and decodes the implied distribution from it.
+
+Under that reading, the 1997 experiments recorded the average of a population
+code and named the average the code. The original observation survives; what it
+was evidence *for* narrows.
+
+### Ramping is still awkward
+
+Howe and colleagues measured dopamine in the striatum with fast-scan cyclic
+voltammetry while rats ran a maze for reward, and found slow ramps: dopamine
+rising gradually as the animal approached the goal, over seconds.
+
+A pure RPE should not do that. Once the path is well learned, each step is
+predicted and the error should sit near zero throughout. A signal climbing
+steadily with proximity looks more like value itself than like an error about
+value.
+
+Reconciliations exist — ramps as the derivative of a value function under
+particular assumptions about state representation and discounting, or as a
+distinct signal carried on a slower timescale by the same transmitter. What
+matters here is that the ramp is a measurement in tension with the simple
+version of the story, taken with a different method on a different timescale,
+and it has not been dissolved by restating the theory.
+
+## What would change this note
+
+A demonstration that the value-coding and salience-coding populations are
+separable by their **downstream targets**, not only by recording site, would
+tighten the account considerably: it would turn an anatomical correlation into a
+claim about what each signal is *for*.
+
+A result showing the asymmetric scaling Dabney reports is an artefact of
+recording or of the fitting procedure would return the population to a scalar
+code and undo that section entirely.
+
+A resolution of ramping that predicts *new* measurements, rather than
+accommodating the existing ones, would close the loose end this note is
+currently obliged to leave open.
 
 ## Why this sits in a machine learning repository
 
-The traffic runs both ways, and it is easy to mistake which way it is running.
-TD learning was not derived from neuroscience — it came out of Sutton's work on
+The traffic runs in both directions, and it is easy to mistake which way it is
+running at any given moment.
+
+TD learning was not copied from the brain. It came out of Sutton's work on
 prediction, and the biology arrived afterwards as confirmation. Distributional
-RL then went the same direction: an algorithmic idea first, a neural signature
-found second.
+RL repeated the pattern in the same order: an algorithmic idea developed for
+reasons internal to reinforcement learning, followed by a search for a neural
+signature, which was then found.
 
-So the honest summary is narrower than the popular one. Reinforcement learning
-did not copy the brain. It produced a theory precise enough to be checked
-against one.
+So the honest summary is narrower than the popular one, and more interesting for
+being narrower. Reinforcement learning did not learn its algorithms by looking
+at brains. It produced theories precise enough that a brain could be used to
+check them — which is a harder thing to do, and a better reason to care.
 
 ## Where it connects
 
-Nothing yet. When there is a note on distributional RL as an algorithm rather
-than a finding, it belongs here.
+Nothing yet. A note on distributional RL as an algorithm — quantile regression,
+the asymmetric loss that produces the effect, and what predicting a distribution
+buys over predicting a mean — belongs beside this one, and would let this note
+stop explaining the algorithmic half in passing.