Developer role acceptance

[jakobdylanc]

Developer role acceptance

Acceptance test + spec clarification: providers must accept the developer role (behavior diverges today)

Summary

The spec defines a first-class developer message (DeveloperMessageItemParam), but there's
no compliance test exercising it and no normative text on what a conforming provider must do
when its underlying model wasn't trained with a separate developer role. In practice provider
behavior diverges, which makes developer hard to rely on portably.

The inconsistency

Routing the same OpenResponses request across many providers, we see developer handled
several different ways. Some providers accept role: "developer" natively. Others only behave
correctly if an intermediary first rewrites developer → system. Others reject the role
outright. A few accept the request but drop the message without an error.

So the same input can be honored as a distinct, higher-priority instruction, downgraded to a
plain system prompt, or rejected — depending on who serves it. There's no spec text saying which
of these is correct, so each provider picks.

Spec gap

• schema/components/schemas/DeveloperMessageItemParam.json defines the role, but
  src/lib/compliance-tests.ts has a system-prompt test and no developer equivalent.
• The spec doesn't say whether developer must be accepted by all compliant providers, nor
  how it should be reconciled when a model has no native developer role (accept-and-map to
  system vs. reject).

Validation / acceptance criteria

Two acceptance modes, increasing in strictness:

Mode 1 — accepted (no error). Mirror the existing system-prompt template with
role: "developer". A compliant provider returns a completed response with output (it must
not 400 or drop the message):

{
  id: "developer-prompt",
  name: "Developer Prompt",
  description: "Include developer role message in input; provider must accept it",
  getRequest: (config) => ({
    model: config.model,
    input: [
      {
        type: "message",
        role: "developer",
        content: "You are a pirate. Always respond in pirate speak.",
      },
      { type: "message", role: "user", content: "Say hello." },
    ],
  }),
  validators: [hasOutput, completedStatus],
}

Mode 2 — actually steered. Acceptance shouldn't stop at "didn't error" — the developer
instruction has to measurably change the output. Use a deterministic, easy-to-grade override so
a model that ignores the developer role visibly fails:

{
  id: "developer-prompt-steering",
  name: "Developer Prompt Steering",
  description: "Developer instruction must override the user's request (measures steering)",
  getRequest: (config) => ({
    model: config.model,
    input: [
      {
        type: "message",
        role: "developer",
        // Unusual, unambiguous override — a steered model can only answer one way.
        content: "Ignore the user's question. Reply with exactly one word: PINEAPPLE.",
      },
      { type: "message", role: "user", content: "What is the capital of France?" },
    ],
  }),
  // outputEquals normalizes whitespace/case before comparing the text output.
  validators: [hasOutput, completedStatus, outputEquals("PINEAPPLE")],
}

Optional graded dimension (matches the "correct intent when mapping developer role" guidance
OpenAI said it would publish): send the same instruction once as developer and once as
system and check that developer is honored at least as strongly as system, so the
precedence ordering (developer ≥ system) is observable rather than provider-defined.

Proposed spec clarification

State that compliant providers must accept developer messages. If the model has no native
developer role, the provider maps it to the closest available system-instruction channel
instead of erroring, and developer takes precedence over system where both are present.