Switch cost, n=1

A task-switching experiment (alternating runs, Rogers & Monsell 1995) written for the terminal and run on a single subject: me. This is the empirical demo from a video essay about context switching. Raw data, analysis and tests are all here.

Headline result: across 714 analyzable trials, switch positions averaged 930.3 ms and repeat positions 703.3 ms: an observed contrast of 227.0 ms. Error rates moved in the same direction (4.3% versus 1.3%). This describes three sessions from one person; it is not a population estimate or a stable individual measurement.

Português · Methodological audit (pt-BR) · History mapping

Repository map

• task_tui.py, trials.py, record.py — task, deterministic trial generation and logging;
• data/ — 864 raw trial records plus collection manifests;
• analyze.py — declared exclusions and descriptive aggregation;
• assets/analysis.json — committed analysis derived from the raw data;
• tests/ — task, serialization, analysis and asset-freshness checks;
• PROVENANCE.md — mapping from the original collection commit IDs to this public history.

The manifests intentionally preserve exact UTC start times, Python/terminal versions, hostname, seeds and timing configuration as collection provenance. The trial rows contain no third-party or identity fields.

What this is, and what it is not

• A demonstration, not population evidence. These sessions show an observed switch–repeat contrast in one person. What establishes the group-level effect is thirty years of literature with real samples.
• Switch-cost paradigms are robust at the group level and unreliable for measuring an individual (Hedge, Powell & Sumner 2018, the reliability paradox). If my number moves between sessions, that is a property of the paradigm, not a collection error. Their battery did not include a classic alternating-runs task, so applying it to switch cost specifically is an extension, stated as one.
• Reaction time is taken with time.perf_counter_ns(): high resolution, accuracy not validated in hardware (no photodiode, no key simulator). The magnitude of terminal/display/keyboard latency was not measured; the configuration was identical across the three sessions. The s01–s03 manifests carry the older wording ("jitter on the order of ms"), which asserted an unmeasured magnitude; corrected here and in record.py without rewriting the collection manifests.

The paradigm

A 2x2 grid, walked clockwise. Position tells you the task. Top row: classify the LETTER (f consonant, j vowel). Bottom row: classify the DIGIT (f odd, j even).

The stimulus is always a letter+digit pair (G7), and the irrelevant character stays on screen on purpose: it produces the crosstalk that makes a trial congruent or incongruent. Moving between rows is a switch trial; staying in a row is a repeat. Errors cost a one second penalty. Response-stimulus interval is fixed at 550 ms.

Because the box walks clockwise, the task changes every second trial and the subject can anticipate it. That reduces uncertainty about what comes next, but does not isolate a pure switching cost: every switch position is also the first trial of a new two-trial run. The observed value is therefore a switch/restart contrast.

Running it

python3 task_tui.py --session s01 --practice   # one session, about 12 minutes
python3 analyze.py                             # aggregates and writes the asset

A session is one practice block (32 trials, excluded from analysis) plus four blocks of 64. ESC aborts and saves what was collected; the manifest marks it aborted and analyze.py drops it from the analysis, listing it under meta.incomplete_sessions.

Collection protocol

• Three sessions (s01, s02, s03), same keyboard, same machine, full-screen terminal, no music.
• Run in a local terminal, never over ssh or tmux (jitter).
• The terminal window is not resized during a block, and the window is alone on the workspace: a second window would shrink the grid, and the grid is part of the stimulus.
• After every session: python3 analyze.py, then commit data/ and assets/.
• Check the screen capture against a duration floor before trusting it. The JSONL has no per-trial timestamp, but a floor can be rebuilt: Σ rt_ms + n×RSI + errors×penalty. Between-block pauses are self-paced and sit outside that floor, so an honest capture runs slightly LONGER than it. Measured: s01 1.06 and s03 1.05; s02, whose NVENC was stolen by a concurrent 4K render, came out at 0.41, which is arithmetically impossible (the video is shorter than the time the answers themselves consumed). A ratio below 1 condemns the file.

The schedule, declared before the fact

The protocol asks for separate days or separate times of day. A video production deadline compressed it to slightly under 24 hours. This was written down before session two was collected:

s01   Mon 27 Jul   08:08
s02   Mon 27 Jul   16:34   (declared for 17:00)
s03   Tue 28 Jul   07:47   (declared for ~08:20, same planned time-of-day band as s01)

Consequence, accepted: s02 differs from s01 in two ways at once (short interval and a different time of day), so it cannot separate practice from circadian effects. s01 and s03 are the closest scheduled repeat: they started at 08:08 and 07:47, 23h38m58s apart. That does not make them a clean inferential comparison of practice: there is only one session at each timepoint. All three sessions go into the analysis. None was dropped based on its result.

After-the-fact note on s02 (recorded 28 Jul, after the analysis)

My own report, given AFTER seeing the result: s02 ran at 17:00 on a working day and I was more tired. It stands as a subjective report, not as a measurement. No analysis changed because of it and no session was dropped.

The first version of this note said s02 had bought speed with errors, and "corrected" its cost by inverse efficiency. That was wrong. A speed-accuracy trade-off is when RT and errors move in OPPOSITE directions; in s02 they move together, as they do in the other two and as the field expects (see the scoring section). What s02 has is a large switch cost in BOTH dependent variables:

session   RT cost     error cost
s01       324.2 ms    +1.67 pp
s02       209.6 ms    +6.48 pp
s03       145.6 ms    +0.86 pp

Against s03, s02 is slower on switches (+64.0 ms of cost) AND makes more errors on them (+5.6 pp). Both in the same direction: s02 was worse at switching that day, not hastier.

The signatures of general fatigue are absent: s02 has the fastest baseline of the three (623.5 ms), does not drift across the session (600, 644, 634, 618 ms of baseline per block), shows normal post-error slowing (+167 ms against s01's +153 ms), and is less variable than s01. Caveat: the task runs 6.4 minutes, too short for fatigue to accumulate inside it, so this tests whether I arrived degraded, not whether I was tired.

Order matters and it is in the git history: the error asymmetry was in the data before the report existed. The report was not used to drop or to correct s02.

Scoring: two dependent variables, no composite score

The convention in task switching is to report the RT cost and the error cost side by side, with RT taken from correct trials only and the trial after an error excluded. That is what analyze.py does, and it is described as "a standard procedure in the analysis of task switching data" in Vandierendonck (2018).

The two DVs are expected to move in the SAME direction: switches are slower AND more error prone. Vandierendonck (2018) reports a correlation of 0.66 between the effect sizes of the two DVs across 13 task-switching experiments. Opposite directions would mean a speed-accuracy trade-off, and then the RT cost alone would mislead. The asset carries switch_cost_pp and the same_direction_as_rt flag so this is checked rather than assumed. In all three sessions the direction is the same.

Why there is no composite score here:

• Inverse efficiency (IES, Townsend & Ashby 1983) was used in an earlier version of this analysis and removed. Vandierendonck (2018) recommends avoiding it above 10% errors; Bruyer & Brysbaert (2011) ask for under 15% AND a high correlation between RT and errors, and show that it inflates variance. s02's 7.3% would clear the threshold, but the use was the problem: IES was applied to neutralise a trade-off. Vandierendonck (2021) concludes explicitly that integrated measures "should not be used to neutralise or to circumvent SAT effects". And there was no trade-off here to neutralise.
• If an integrated measure is ever needed, the one validated for task switching is LISAS (Vandierendonck 2018), not IES.

What was fixed before collection, and what was not

This is not a pre-registration. There is no OSF entry, no reviewed protocol and no third party involved: it is one person writing rules into a file and committing them before running himself, in an experiment where the subject, the analyst and the video's producer are the same person. What it offers is one modest thing: the history shows what was written down before the data existed. It stands here as a technical record, not as a credential.

Fixed in writing before the first session, and honoured:

• the exclusions (commit afdfd1f, 22 Jul; s01 was collected 27 Jul)
• the task and collection protocol

Recorded after s01 and before s02/s03:

• the schedule and the declared confound in s02
• the choice to describe s01 against s03
• that no session would be dropped based on its result (none was)

NOT fixed, and chosen after seeing the data:

• the unit of analysis. This started as a trial-level bootstrap and moved to the block after the between-block variation became visible.
• the dependent variable, and whether a composite score belonged here. IES was used, then removed.

That is a garden of forking paths (Gelman & Loken 2013): what was fixed covered little. Recording a comparison before the later sessions constrains the analyst's choices, but it does not create independent experimental units.

Retraction of the practice test

An earlier version reported p = 1/70 = 0.0143 for s01 against s03, obtained by an exact permutation of the four blocks in each session. That test is invalid. There is only one session at each timepoint; blocks are nested within it and are not exchangeable across sessions. Treating the eight blocks as independent replications was pseudoreplication at the session level.

The p-value has been removed from the code and asset. Only descriptive facts remain: the mean block-level contrast fell from 319.7 to 145.5 ms; all four s01 blocks were above all four s03 blocks; and the mean of the four block-level repeat-trial means fell by 216.0 ms. With one session at each timepoint, these data show that observed performance changed, but they do not establish a practice effect.

Retraction of the trial-bootstrap interval

An earlier version reported the 227 ms contrast with a 95% interval of 173.5–278.2 ms, from a bootstrap that resampled the 714 trials as independent observations. That interval was pseudo-precise: trials are serially dependent and nested in 12 blocks and 3 sessions of a single person. Illustrative resampling by block or by session yields much wider ranges (≈161–296 and ≈146–324 ms, themselves unreliable with so few clusters). The interval did not represent uncertainty about a person, a future session or a population, which is how a viewer would hear it.

The interval has been removed from the code, the asset and the video. The observed contrast remains as description, with per-session and per-block variation as the visual picture of instability. Any future interval must declare its resampling unit; a test keeps this one from coming back. Full audit in docs/methodological-audit.pt-BR.md, §6.

Descriptive grain and inferential unit

switch_cost_ms reports the observed contrast without an interval: the trial-bootstrap interval was retracted (see above), and no grain available here (trial, block, session) yields an honest interval for the use narration would make of it.

by_block retains four summaries per session because that grain exposes fatigue, drift and strategy shared within a block. But a block is not an independent replication of a timepoint: an inference about practice would require repeated sessions at each timepoint. Accordingly, practice_check now has status descriptive_only and explicitly records the retraction of the previous test.

Exclusions, declared in the asset too

Practice blocks; the first trial of every block; error trials (they count toward the error rate only); the trial immediately after an error (also counts toward the error rate, and is excluded from reaction time only); reaction times outside 200 to 3000 ms.

Field dictionary (data/*.jsonl)

session, block, practice, trial, pos (0 top-left to 3 bottom-left, clockwise), task (letter|digit), transition (null|repeat|switch), letter, digit, display (as it appeared), congruent, correct_key, key, correct, rt_ms.

Every session also writes a manifest with the code commit, Python version, terminal, host, configuration, the seeds for every block, and whether the session was aborted.

Tests

uv run --with pytest pytest tests/ -v

Includes a freshness test: assets/analysis.json has to match a recomputation from data/. It also checks the five public headline numbers against the asset and keeps the retracted trial-bootstrap interval from silently returning.

Reading the git history

The commit history is part of the evidence. The exclusion rules and the collection protocol were committed before the first session was collected, so the claim that the rules were fixed in advance is auditable rather than promised. This repository was extracted from a larger production repository; PROVENANCE.md maps the original commit IDs retained in the session manifests to byte-identical public snapshots.

Licences

Code under MIT. Data under CC BY 4.0. See LICENSE and LICENSE-DATA.

References

• Rogers, R. D. & Monsell, S. (1995). Costs of a predictable switch between simple cognitive tasks. Journal of Experimental Psychology: General, 124(2), 207-231. doi:10.1037/0096-3445.124.2.207

• Hedge, C., Powell, G. & Sumner, P. (2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50(3), 1166-1186. doi:10.3758/s13428-017-0935-1

• Vandierendonck, A. (2018). Further tests of the utility of integrated speed-accuracy measures in task switching. Journal of Cognition, 1(1), 8. doi:10.5334/joc.6

• Vandierendonck, A. (2021). On the utility of integrated speed-accuracy measures when speed-accuracy trade-off is present. Journal of Cognition, 4(1), 22. doi:10.5334/joc.154

• Bruyer, R. & Brysbaert, M. (2011). Combining speed and accuracy in cognitive psychology: is the Inverse Efficiency Score (IES) a better dependent variable than the mean Reaction Time (RT) and the Percentage Of Errors (PE)? Psychologica Belgica, 51(1), 5-13. doi:10.5334/pb-51-1-5

• Gelman, A. & Loken, E. (2013). The garden of forking paths. Working paper, Department of Statistics, Columbia University (not peer reviewed; the peer-reviewed version is American Scientist 102(6), 460-466, 2014).