Skip to content

[codex] Add OpenAI compatibility guardrail report#987

Open
mimeding wants to merge 1 commit intoosaurus-ai:mainfrom
mimeding:codex/api-compat-guardrails
Open

[codex] Add OpenAI compatibility guardrail report#987
mimeding wants to merge 1 commit intoosaurus-ai:mainfrom
mimeding:codex/api-compat-guardrails

Conversation

@mimeding
Copy link
Copy Markdown
Contributor

@mimeding mimeding commented Apr 30, 2026

Business rationale

OpenAI-compatible clients are part of the Osaurus multi-provider harness, and users notice immediately when familiar Chat Completions response shapes, streaming frames, tool-call deltas, or validation errors drift. This PR gives maintainers a repeatable local report with saved artifacts, so API compatibility changes can be reviewed against concrete evidence instead of manual curl snippets.

Coding rationale

The guardrail stays as a script because it needs to exercise a running Osaurus server as a black-box API contract, not import app internals. The Makefile target gives contributors a stable entry point while still allowing HOST, MODEL, OUT_DIR, and REQUIRE_TOOL_CALLS overrides. The script records request, status, headers, body, and streaming chunk artifacts so failures are inspectable after the run. The branch was rebuilt as one clean commit on current origin/main, dropping unrelated stale stack history from earlier PR dependencies.

What changed

  • Expanded scripts/openai_compat_report.sh beyond streaming-only checks.
  • Added non-streaming Chat Completions response-shape validation.
  • Added SSE framing and streaming chunk-shape validation.
  • Added OpenAI-style streaming tool-call delta validation with optional REQUIRE_TOOL_CALLS=1.
  • Added unsupported request validation/error-shape coverage for n > 1.
  • Wrote detailed artifacts under results/openai_compat/ and a Markdown summary at results/openai_compat_report.md by default.
  • Added make compat-openai and documented it in docs/DEVELOPER_TOOLS.md.

Validation

  • bash -n scripts/openai_compat_report.sh
  • shellcheck scripts/openai_compat_report.sh
  • make -n compat-openai HOST=http://localhost:1337 MODEL=foundation
  • git diff --check HEAD~1..HEAD
  • swift build --package-path Packages/OsaurusCore
  • swift build --package-path Packages/OsaurusCore -c release
  • swift test --package-path Packages/OsaurusCore — initial run hit one unrelated suite issue; immediate rerun passed with 1436 tests in 192 suites.
  • Swift-format/swiftlint touched-file checks are not applicable: this PR touches no Swift files.
  • CLI tests skipped: this PR touches no CLI package files.

Non-scope

  • Does not add a GitHub Actions job for live API compatibility because it requires a running Osaurus server and model.
  • Does not change server request/response behavior.
  • Does not add generated report artifacts to the repository.

Residual risks

  • Tool-call validation can only pass when the selected model actually emits tool calls; by default the script records that as a warning unless REQUIRE_TOOL_CALLS=1 is set.
  • The report is a local maintainer guardrail, so future CI automation would need a separate controlled server fixture.

@mimeding mimeding force-pushed the codex/api-compat-guardrails branch from 89d100e to bd1f58f Compare April 30, 2026 18:28
@mimeding mimeding marked this pull request as ready for review May 1, 2026 02:52
@mimeding
Copy link
Copy Markdown
Contributor Author

mimeding commented May 1, 2026

Author-side cleanup complete; this PR is now ready for review.

Verification:

  • GitHub checks are green: test-core, test-cli, swiftlint, shellcheck, update_release_draft, and pr-clean-gate.
  • scripts/ci/check-pr-clean.sh osaurus-ai/osaurus 987 reports clean.

Historical failures were from stale DerivedData module resolution and stale FluidAudio locks; current head has the PR cold-DerivedData path and FluidAudio 0.14.3 lock alignment.

@mimeding mimeding force-pushed the codex/api-compat-guardrails branch 2 times, most recently from 3db134b to ed0e77b Compare May 1, 2026 16:26
Business rationale: OpenAI-compatible clients are a key multi-provider harness surface, and maintainers need a repeatable local report that catches response-shape, streaming, tool-call, and validation regressions before users hit them.

Coding rationale: the guardrail stays in scripts rather than app code because it exercises a running server as a black-box compatibility contract; Makefile and docs entries make it discoverable, while artifacts under results/openai_compat keep request and response evidence available for review.

Co-authored-by: Michael Meding <mmeding@Michaels-Mac-Studio.local>
@mimeding mimeding force-pushed the codex/api-compat-guardrails branch from ed0e77b to 137754e Compare May 3, 2026 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant