Skip to content

fix(observability): make Datadog optional without production outage - #554

Merged
jaywedgeworth22 merged 3 commits into
mainfrom
instinct/datadog-optional-default
Nov 2, 2026
Merged

jaywedgeworth22 merged 3 commits into
mainfrom
instinct/datadog-optional-default

Conversation

@jaywedgeworth22

@jaywedgeworth22 jaywedgeworth22 commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Makes Datadog observability opt-in by default so production deployments without Datadog credentials continue serving traffic instead of returning 503.
  • Adds an explicit DD_REQUIRE=1 mode that restores the previous fail-closed behavior: requests are rejected with 503 when DD_API_KEY or DATADOG_API_KEY is missing, and the application handler is not invoked.
  • Preserves existing behavior for missing RUM tokens, which remain inactive without disrupting requests.
  • Keeps application errors visible by rethrowing them after middleware processing.

Test Coverage

  • Added configuration tests covering the default dark state, explicit enforcement, and valid Datadog configuration.
  • Added middleware tests verifying that dark mode preserves responses without making Datadog requests, enforced mode returns 503, and application errors are propagated.

Outstanding Review Findings

  • An existing script test still expects production configuration to throw when Datadog credentials are missing, causing npm test to fail until that expectation is updated.
  • The new middleware test imports a .ts path that TypeScript rejects without allowImportingTsExtensions; the import must be adjusted for the project’s compiler configuration.

Summary

  • Keeps Datadog observability optional in production when DD_* credentials are not configured, preventing missing configuration from disrupting application traffic.
  • Ensures disabled Datadog middleware makes no telemetry requests and passes application responses and errors through unchanged.
  • Preserves explicit enforcement with DD_REQUIRE=1: production requests receive HTTP 503 without invoking the application.
  • Updates observability configuration and middleware regression tests to cover the opt-in default and required-mode behavior.

Summary

Makes Datadog an optional production dependency by default while preserving strict enforcement for environments that explicitly opt in.

Changes

  • Updates production configuration validation so a missing Datadog server API key only fails when DD_REQUIRE=1 is configured.
  • Verifies that the default “dark” middleware path:
    • Makes no Datadog network requests.
    • Returns the application response unchanged.
    • Preserves application errors.
  • Retains fail-closed behavior for explicitly required Datadog: a missing DD_API_KEY returns HTTP 503 without invoking the application handler.
  • Normalizes related test formatting and imports.

Impact

Production deployments without Datadog credentials can continue serving traffic instead of failing because observability is not configured. Deployments that deliberately set DD_REQUIRE=1 continue to enforce Datadog availability and return a 503 when the required server credentials are absent.


Summary

Makes Datadog observability optional by default in production to prevent outages when server keys are missing.

Fail-closed behavior (reporting missing DD_API_KEY) now only applies when explicitly required via DD_REQUIRE=1, instead of automatically failing in all production environments.

Changes

  • Updated observability config expectations: production with no DD_* keys stays dark instead of throwing / blocking.
  • Updated server middleware expectations: when unconfigured in production, requests pass through unchanged with no outbound fetch, preserving response status and surfacing app errors as-is.
  • Preserved fail-closed guard for explicit opt-in: with VERCEL_ENV=production + DD_REQUIRE=1 and no API key, missing keys are still reported and middleware returns 503 without invoking the app.
  • Formatting / import cleanup in config.test.ts and middleware.test.ts with no behavior change.

Impact

Production deploys without Datadog credentials continue to serve traffic normally with observability disabled, while teams can still enforce required observability by setting DD_REQUIRE=1.

@kody-ai

This comment has been minimized.

@swu-echo

swu-echo Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

@kody start-review

Comment thread src/lib/observability/config.ts Resolved
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor
Autofix run https://github.com/Simple-With-Us/DealDex/actions/runs/41366841684 started for `sentry[bot]`'s tagged request. Owner/sentry request: exempt from the per-bot cap (owner rule 2026-10-20).

Comment thread src/lib/observability/config.ts Resolved
@kody-ai

This comment has been minimized.

@swu-echo

swu-echo Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

@kody start-review

@kody-ai

This comment has been minimized.

@swu-echo

swu-echo Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

@kody start-review

@kody-ai

kody-ai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Kody Review Complete

Great news! 🎉
No issues were found that match your current review configurations.

Keep up the excellent work! 🚀

⚠️ 1 Kody Rule(s) were not evaluated

These rules declare that they need repository context beyond this diff, and the review could not retrieve it, so they were not judged on this pull request:

  • Match CI env var names exactly to the script that reads them
Kody Guide: Usage and Configuration
Interacting with Kody
  • Request a Review: Ask Kody to review your PR manually by adding a comment with the @kody start-review command at the root of your PR.

  • Validate Business Logic: Ask Kody to validate your code against business rules by adding a comment with the @kody -v business-logic command.

  • Provide Feedback: Help Kody learn and improve by reacting to its comments with a 👍 for helpful suggestions or a 👎 if improvements are needed.

Current Kody Configuration
Review Options

The following review options are enabled or disabled:

Options Enabled
Bug ✅
Performance ✅
Security ✅
Business Logic ✅

Access your configuration settings here.

​

@jaywedgeworth22
jaywedgeworth22 merged commit c530941 into main Nov 2, 2026
19 checks passed
@jaywedgeworth22
jaywedgeworth22 deleted the instinct/datadog-optional-default branch November 2, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant