Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .agent/packages/webhooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,14 +37,16 @@
- `WebhookDispatcher` _(final)_
- `WebhookHandler` _(final)_
- `WebhookMessage` _(final)_
- `WebhookReplayCommand` _(final)_ — implements `SignalableCommandInterface`
- `WebhookShowFailedCommand` _(final)_ — implements `SignalableCommandInterface`
- `WebhookReplayCommand` _(final)_
- `WebhookShowFailedCommand` _(final)_
- `WebhookVerifyMiddleware` _(final)_ — implements `MiddlewareInterface`
- `WebhooksConfiguration` _(final)_ — implements `ConfigurationInterface`

## Tests as documentation

- `tests/Webhooks/Cli/WebhookReplayCommandTest.php`
- `tests/Webhooks/Cli/WebhookShowFailedCommandTest.php`
- `tests/Webhooks/Configuration/WebhooksConfigurationTest.php`
- `tests/Webhooks/Dispatcher/RetryPolicyTest.php`
- `tests/Webhooks/Dispatcher/WebhookDispatcherTest.php`
- `tests/Webhooks/Dispatcher/WebhookHandlerTest.php`
Expand Down
56 changes: 56 additions & 0 deletions benchmarks/tokens-to-ship/fixtures/posts-webhook-in.openapi.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
openapi: 3.1.0
info:
title: Stripe webhook receiver (inbound)
version: 1.0.0
paths:
/webhooks/stripe:
post:
operationId: receiveStripe
summary: Receive Stripe webhook events
# Inbound: verify HMAC-SHA256 over the raw body, enforce a 5m
# timestamp window, and dedupe by event id for 24h. Field order
# mirrors OpenApiEmitter::renderWebhook so openapi:roundtrip is clean.
x-altair-webhook:
direction: in
signing: hmac-sha256
secret_name: stripe
header: Stripe-Signature
dedupe_ttl: 24h
x-altair-domain:
class: App\Webhook\ReceiveStripe
invocation: __invoke
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [id, type]
properties:
id: { type: string }
type: { type: string }
responses:
'200':
description: Accepted (or idempotently absorbed on replay)
content:
application/json:
schema:
type: object
properties:
ok: { type: boolean }
'400':
description: Missing or expired timestamp (replay window)
content:
application/json:
schema:
type: object
properties:
error: { type: string }
'401':
description: Signature verification failed
content:
application/json:
schema:
type: object
properties:
error: { type: string }
60 changes: 60 additions & 0 deletions benchmarks/tokens-to-ship/fixtures/posts-webhook-out.openapi.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
openapi: 3.1.0
info:
title: Posts API (outbound webhook)
version: 1.0.0
paths:
/posts:
post:
operationId: createPost
summary: Create a post and emit a signed post.created webhook
# Outbound: after a successful create, sign + dispatch a
# post.created webhook with retry + dead-letter. Field order mirrors
# OpenApiEmitter::renderWebhook so openapi:roundtrip is clean (only
# non-default retry values are carried, otherwise they would be
# dropped on re-emit and the gate would flag drift).
x-altair-webhook:
direction: out
signing: hmac-sha256
retry:
max_attempts: 8
backoff: linear
base_delay: 60s
dead_letter: webhook.deadletter
x-altair-domain:
class: App\Post\CreatePost
invocation: __invoke
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [title, body]
properties:
title: { type: string }
body: { type: string }
responses:
'201':
description: Created (a post.created webhook is dispatched)
content:
application/json:
schema:
$ref: '#/components/schemas/Post'
'422':
description: Validation failed
content:
application/json:
schema:
type: object
properties:
errors:
type: object
components:
schemas:
Post:
type: object
required: [id, title, body]
properties:
id: { type: string }
title: { type: string }
body: { type: string }
125 changes: 125 additions & 0 deletions benchmarks/tokens-to-ship/task-webhooks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
# Frozen task variant — webhooks (inbound + outbound)

> Companion to [`task.md`](./task.md), [`task-import.md`](./task-import.md)
> and [`task-idempotency.md`](./task-idempotency.md). Measures what it
> costs each arm to deliver a *correctly signed and idempotent inbound
> webhook receiver* and a *retried and dead-letterable outbound
> dispatcher* — the canonical realistic third-party-integration task in
> 2026.

Inbound and outbound are **two separate fixtures and two separate
acceptance sections**. They measure different things; lumping them into
one number would average a misleading aggregate. Two rows in the
published table is the honest framing.

## Prompt given to the agent (verbatim)

### Inbound

> Add a **Stripe webhook receiver** at `POST /webhooks/stripe` that:
>
> - Verifies an HMAC-SHA256 signature carried in the `Stripe-Signature`
> header against a shared secret. A tampered or absent signature is
> rejected with `401`.
> - Reads a signed timestamp and rejects requests outside a 5-minute
> window with `400` (replay protection).
> - Dedupes by event id for 24 hours: a redelivery of an already-seen
> event returns `200` without re-processing.
>
> Stop when the acceptance suite passes.

### Outbound

> After a successful `POST /posts`, emit a signed `post.created`
> webhook to a configured subscriber. Sign with HMAC-SHA256, retry on
> `5xx` / network failure with backoff, and dead-letter after the
> configured maximum attempts. Provide a way to re-send a
> dead-lettered delivery.
>
> Stop when the acceptance suite passes.

## What changes between arms

- **Arm A — Altair.** Agent adds a `webhook:` block to the relevant
spec (`direction: in` on the receiver, `direction: out` on
`POST /posts`), re-runs `bin/altair spec:scaffold`. Inbound wires
`ActionAwareWebhookVerifyMiddleware` from the generated Action's
`webhook()` accessor; outbound wires a `WebhookDispatcher` binding.
Storage adapters come from `WebhooksConfiguration` (`InMemory*` for
the test run; Redis in a production deploy);
`bin/altair webhook:show-failed` / `webhook:replay` cover the
dead-letter loop. Done.
- **Arm B — Baseline.** Agent picks (or hand-rolls) a signing
middleware, wires the timestamp window, writes the dedupe storage,
builds the outbound dispatcher with retry + DLQ, wires replay
tooling, and writes the tests. Reasonable baselines exist for the
signing part (`paragonie/halite`, hand-rolled HMAC); the
*integration cost* — wiring verification, dedupe, retry curves,
dead-letter and replay into one coherent surface — is what is being
measured.

## Fixtures

- [`fixtures/posts-webhook-in.openapi.yaml`](./fixtures/posts-webhook-in.openapi.yaml)
— inbound: `POST /webhooks/stripe` carrying
`x-altair-webhook: { direction: in, signing: hmac-sha256,
secret_name: stripe, header: Stripe-Signature, dedupe_ttl: 24h }`.
- [`fixtures/posts-webhook-out.openapi.yaml`](./fixtures/posts-webhook-out.openapi.yaml)
— outbound: `POST /posts` carrying
`x-altair-webhook: { direction: out, signing: hmac-sha256,
retry: { max_attempts: 8, backoff: linear, base_delay: 60s },
dead_letter: webhook.deadletter }`.

Both fixtures round-trip clean through `bin/altair openapi:roundtrip`
and scaffold a working project through
`bin/altair openapi:import --scaffold`. The frozen documents carry only
non-default field values, matching the v1 wire contract: `direction`
and `signing` always travel; other fields round-trip only when they
differ from their default (see
[docs/openapi/extensions.md](../../docs/openapi/extensions.md)).

## Acceptance criteria

### Inbound (delta on the receiver)

- [ ] `POST /webhooks/stripe` with a valid HMAC + fresh event id →
`200`.
- [ ] Replay with the same event id → `200` with
`Webhook-Replayed: true`.
- [ ] Tampered signature → `401`.
- [ ] Timestamp outside the 5-minute window → `400`.

### Outbound (delta on `POST /posts`)

- [ ] Creating a post dispatches a signed POST to the configured
subscriber (verified by a test fixture endpoint), carrying
`X-Signature` / `X-Timestamp` / `X-Event-Id` / `X-Delivery-Id`.
- [ ] A failing subscriber (`5xx`) is retried with the configured
backoff and eventually dead-lettered after `max_attempts`.
- [ ] `bin/altair webhook:replay <delivery-id>` re-dispatches a
dead-lettered delivery.

## Reporting

Report this variant **separately** — and as **two rows**, inbound and
outbound. The forward task answers "given a prose spec, what does CRUD
cost?"; the import variant answers "what does translating an OpenAPI
contract cost?"; the idempotency variant answers "what does correct
idempotency cost?"; this variant answers **"what does a correct,
production-grade webhook integration cost — in each direction?"**

| Variant | Median tokens (Arm A / Arm B) | Median wallclock | pass@1 |
|---|---|---|---|
| [task.md](./task.md) | … / … | … / … | … |
| [task-import.md](./task-import.md) | … / … | … / … | … |
| [task-idempotency.md](./task-idempotency.md) | … / … | … / … | … |
| **task-webhooks.md — inbound** *(this)* | … / … | … / … | … |
| **task-webhooks.md — outbound** *(this)* | … / … | … / … | … |

The honest framing: "integrate with a third party's webhooks" is the
canonical realistic API task in 2026 — every payments / SaaS / commerce
platform ships both directions, and no PHP framework ships a native
primitive for either. The Univeros bet is that a spec block + built-in
signing / dedupe / dispatch + a round-trip-safe OpenAPI extension beats
hand-rolled-from-scratch and beats find-and-wire-three-packages, *and*
keeps beating them on the next change.
1 change: 1 addition & 0 deletions bin/altair
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ $builtIn = [
__DIR__ . '/../src/Altair/Scaffold/Cli',
__DIR__ . '/../src/Altair/Suggest/Cli',
__DIR__ . '/../src/Altair/Tinker/Cli',
__DIR__ . '/../src/Altair/Webhooks/Cli',
];
foreach ($builtIn as $entry) {
if (is_dir($entry)) {
Expand Down
2 changes: 2 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ The request/response lifecycle and everything that runs inside it.
- [Session](./packages/session.md) — server-side session storage with File / Mongo / PDO / Predis handlers, paired with the cookie envelope and HTTP cache limiters.
- [Sanitation](./packages/sanitation.md) — sixteen input filters (Alpha, Boolean, Integer, Regex, …) that normalise raw values into safe canonical forms before validation runs.
- [Validation](./packages/validation.md) — eighteen rule-based input validators (Email, IBAN, ZipCode, …) composable into rule collections and runnable through a `Validator`.
- [Idempotency](./packages/idempotency.md) — Stripe-style `Idempotency-Key` primitive: a PSR-15 middleware that hashes the body, claims the key in a pluggable store (InMemory / APCu / Redis), replays the captured response on retry, and refuses 409 on payload drift. Spec-driven via an `idempotency:` block and an `x-altair-idempotency` OpenAPI extension that round-trips.
- [Webhooks](./packages/webhooks.md) — first-class webhook primitive, both directions: inbound signature verification (HMAC-SHA256/512, Ed25519) + timestamp window + event-id dedupe as PSR-15 middleware, and an outbound signed dispatcher over Symfony Messenger with retry, dead-letter, and `bin/altair webhook:replay`. Spec-driven via a `webhook:` block and an `x-altair-webhook` OpenAPI extension that round-trips.

### Application core

Expand Down
Loading