Skip to content

Before heartbeat.authorChips is ever on: measure the prompt line on real renders + an adversarial eval of the chip guard #384

Description

@kpmmmurphy

heartbeat.authorChips (PR #382) ships off, with no Settings row, on purpose. This issue is the gate for ever turning it on. Two things are owed, and each needs the Mac's MLX slot to itself.

1 · Measure the prompt line on real renders (Lil first, then Big)

Asking for ASK: lines changes the heartbeat prompt, and every sentence that prompt has gained got performed (#349's date line, the small-talk RULE). The instrument is already in: one .notice line per render —

chips: written=N admitted=M stored=K

Run ≥ 20 real pulses with -heartbeat.authorChips YES and record: how often ASK lines appear at all; written → admitted → stored; whether the narrative itself changes (length, the opener rule, NarrativeGuard rejection rate vs the flag-off baseline, TODO: behaviour when both writes are on); and whether ASK: ever leaks into the note body or the model writes the literal <question>. The wording in HeartbeatPrompt is a first draft to measure, not a finding.

2 · An adversarial eval of PulseAskLine.admit

The guard is a denylist. Three review passes in a row each found another way to spell a refused word (a zero-width space, a hyphen, three tokens), and a fourth (a Cyrillic "о") turned up while folding the third. The rules now close whole classes — control/format scalars, every adjacent-token window, mixed-script words, whitespace-squeezed link fragments — and a twenty-chip panel pins that ordinary questions still pass. The next spelling still exists.

What bounds the risk is structural: a chip is one short question (≤ 44 chars), shown to the user, and sent only because they read it and tapped it; nothing auto-sends. Before the flag is a default, that argument deserves a real test:

  • a generated adversarial set (leetspeak, enclosed alphanumerics, bidi, homoglyph tables, other-language instruction verbs) run through admit, with the pass rate recorded;
  • an end-to-end injection attempt: a hostile MCP visitor name / chat title in the digest → does Lil ever author a chip that carries it, and does the guard stop it;
  • a decision on taint: a tapped chip is an ordinary user turn today. Should a chip-originated turn be marked (e.g. kept out of memory distillation), the way recent_activity carries DistillationTaint?

Not before both are done

No Settings row, no default-on, no mention on the site.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions