Skip to content

Improve alpha testing coverage for MCP output, host compatibility, and npm smoke tests #1

Description

@shreyash-chonkie

Context

The current alpha test suite is credible: it covers compiler discovery, MCP runtime methods, auth, pagination, widget bundling, platform-specific files, diagnostics, package metadata, and npm install/create smoke tests. Before treating Sidecar as stable, we should raise the bar around generated artifacts and real host compatibility.

Current coverage

  • Unit/runtime tests for tools, resources, prompts, auth, proxy, pagination, schema enforcement, and the MCP Apps bridge.
  • Compiler tests for reserved files, folder-derived ids/URIs, widget resources, generated clients, platform-specific file selection, and diagnostics.
  • E2E build tests for plain MCP, ChatGPT target, Claude plugin output, widget URI cache busting, path containment, and plugin filename/frontmatter sanitization.
  • Packaging checks for exports/bins and smoke tests for npm install sidecar-ai and npm create sidecar-app@latest.

Work to add

  • Add golden artifact snapshots for every generated manifest/plugin file.
  • Validate emitted MCP/plugin artifacts against official JSON Schemas where available.
  • Add live host acceptance tests for ChatGPT and Claude where automation is possible.
  • Add browser visual regression tests for native components across light/dark mode and host recipes.
  • Add CI smoke tests that install from npm, scaffold an app, run sidecar check, and run sidecar build.
  • Add more adversarial/fuzz tests for JSON-RPC request handling, auth, proxy middleware, and path handling.

Acceptance criteria

  • CI has a job that validates generated artifacts structurally, not only through hand-written expectations.
  • CI has a published-package smoke test path that exercises sidecar-ai and create-sidecar-app from npm.
  • We have a documented distinction between fast unit tests, build artifact tests, visual tests, and live host acceptance tests.
  • Failures identify whether the breakage is compiler output, MCP protocol behavior, host-specific behavior, packaging, or visual/native UI parity.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions