Testing Strategy

Testing Strategy

Cross-cutting reference. Applies to the reference prototype (prototype/) and test suite (tests/) tracked in Roadmap and Status.

Testing Philosophy

Because several pillars make judgment calls that directly affect a founder's relationships and revenue — what gets surfaced, what a draft says, what a negotiation brief recommends — testing here is not limited to standard functional correctness. It also has to verify the constraints from Design Philosophy actually hold, not just that the code runs without errors.

Test Pyramid

flowchart TD
    A[Unit Tests<br/>individual functions: scoring math, feature extraction] --> B[Component Tests<br/>each pillar in isolation against fixed schemas]
    B --> C[Contract Tests<br/>schema conformance between pillars]
    C --> D[Constraint Tests<br/>Design Philosophy invariants]
    D --> E[Scenario Tests<br/>end-to-end pipeline against realistic message sets]

Loading

Unit Tests

Cover deterministic logic directly: urgency signal weighting math (Cortex Scoring Model), feature extraction functions (Persona Stylometry Engine), leverage score computation (Negotiation Intelligence Algorithms), and fatigue decay functions (Energy Shield Decision Fatigue Model). These are fast, numerous, and run on every change.

Component Tests

Each pillar is tested in isolation against fixed input/output fixtures conforming to the schemas in Data Schemas Reference — for example, feeding Persona Mirror Drafting a fixed AnalysisResult and StyleProfile and verifying the draft's style-consistency score meets threshold, independent of whether Cortex or Stylometry actually produced those inputs correctly.

Contract Tests

Verify that what one component actually produces validates against what the next component actually expects — catching schema drift between, say, a NormalizedMessage producer and consumer before it causes a runtime failure. See Data Schemas Reference for the schema evolution policy these tests enforce.

flowchart LR
    A[Ingestion output] -->|validate against| B[NormalizedMessage schema]
    C[Cortex output] -->|validate against| D[AnalysisResult schema]
    E[Persona output] -->|validate against| F[DraftResponse schema]
    G[Negotiation output] -->|validate against| H[NegotiationBrief schema]

Loading

Constraint Tests

This is the category unique to this project's risk profile: automated checks that the Design Philosophy invariants hold in practice, not just in documentation.

Constraint	Test approach
No fabricated urgency	Feed messages with no real deadline evidence; assert urgency factors never include a fabricated deadline
No fabricated scarcity in negotiation briefs	Feed threads with no real capacity constraint; assert scarcity_signal is null
SLA tiers are sentiment-independent	Feed identical relationship/inquiry-type messages with varying sentiment; assert tier assignment is unchanged
No third-party voice modeling	Attempt to construct a StyleProfile from non-founder-authored content; assert rejection
Draft factual-claim check	Feed a draft-generation request where the model is prompted to reference a nonexistent action; assert the post-generation check flags it

sequenceDiagram
    participant Test as Constraint Test
    participant Sys as Pipeline Component
    participant Check as Post-hoc Verifier

    Test->>Sys: adversarial or edge-case input
    Sys->>Sys: process normally
    Sys->>Check: output
    Check->>Check: verify against Design Philosophy invariant
    Check-->>Test: pass / fail, with specific violated constraint if failed

Loading

Scenario Tests

End-to-end runs of realistic message sequences — a multi-message negotiation thread, a frustrated-client escalation, a burst of messages during a high-fatigue period — verifying the full pipeline behaves sensibly across pillars, not just correctly within any single one.

Regression Coverage for Prompts

Every prompt specification in Prompt Specification Standard ships with its own test_cases, run automatically whenever a prompt version changes, before the new version can be activated.

Current Status

Per Roadmap and Status, the test suite (tests/) is a Phase 4 deliverable, developed alongside the reference prototype. This page defines the target strategy the suite should be built against.

See Also

• Data Schemas Reference
• Design Philosophy
• Prompt Specification Standard