Summary
infra/*/src/telemetry.ts silently falls back to the console exporters — with the unbatched Simple*Processor — whenever OTEL_EXPORTER_OTLP_ENDPOINT is unset. In a container that is not dev-safe: it turns every span, every metric datapoint and every log record into a line on stdout, which the cluster log agent then harvests and ships. We hit this in a deployment and log volume went up roughly 4x, with about 99.8% of it landing as severity=info.
Detail
From infra/overlay-server/src/telemetry.ts (the same shape is in all seven components listed in infra/OBSERVABILITY.md):
const useOtlp = typeof otlpEndpoint === 'string' && otlpEndpoint.length > 0
const traceExporter = useOtlp ? new OTLPTraceExporter() : new ConsoleSpanExporter()
const metricExporter = useOtlp ? new OTLPMetricExporter() : new ConsoleMetricExporter()
const logExporter = useOtlp ? new OTLPLogExporter() : new ConsoleLogRecordExporter()
const logRecordProcessor = useOtlp
? new BatchLogRecordProcessor({ exporter: logExporter })
: new SimpleLogRecordProcessor({ exporter: logExporter })and the span processor likewise uses SimpleSpanProcessor in the fallback.
The docstring calls the fallback "dev-safe", and it is — locally. The problem is that nothing distinguishes "developer ran this with no collector" from "this is in production and the endpoint was never wired". The second case is the dangerous one and it is the silent one:
- Full auto-instrumentation is on.
getNodeAutoInstrumentations() plus RuntimeNodeInstrumentation() means http, express, mongodb, mysql2 and dns spans, and nodejs.eventloop.* / v8js.memory.heap.* / GC metrics, all rendered to stdout.
Simple*Processor means one write per record, no batching.- Console output carries no severity, so a log pipeline classifies the whole flood as info — it is indistinguishable from application logging without parsing it.
console.* is double-written. The bridge emits each call to the OTel Logs API and calls the original, so in the fallback path the same message goes to stdout twice: once as the raw console.* line and once as a ConsoleLogRecordExporter record.- The image
CMD preloads telemetry.js, so this is the default behaviour of the published image, not something a user opts into.
Suggested fixes
Any of these would have prevented it; the first two feel right together.
- Do not export at all when no endpoint is configured unless explicitly asked, e.g.
OTEL_CONSOLE_EXPORTERS=true. A process with nowhere to send telemetry should be quiet, not verbose.
- Gate the fallback on environment.
DEPLOY_ENV / NODE_ENV is already read a few lines above for deployment.environment; refusing console exporters when it is production costs nothing.
- If the fallback stays, batch it —
BatchSpanProcessor / BatchLogRecordProcessor even for console, and a much longer exportIntervalMillis for the metric reader.
- Warn loudly once at startup when falling back, naming the variable that is missing. Right now the only signal is the volume itself.
Happy to send a PR for whichever shape you prefer.
Summary
infra/*/src/telemetry.tssilently falls back to the console exporters — with the unbatchedSimple*Processor— wheneverOTEL_EXPORTER_OTLP_ENDPOINTis unset. In a container that is not dev-safe: it turns every span, every metric datapoint and every log record into a line on stdout, which the cluster log agent then harvests and ships. We hit this in a deployment and log volume went up roughly 4x, with about 99.8% of it landing asseverity=info.Detail
From
infra/overlay-server/src/telemetry.ts(the same shape is in all seven components listed ininfra/OBSERVABILITY.md):and the span processor likewise uses
SimpleSpanProcessorin the fallback.The docstring calls the fallback "dev-safe", and it is — locally. The problem is that nothing distinguishes "developer ran this with no collector" from "this is in production and the endpoint was never wired". The second case is the dangerous one and it is the silent one:
getNodeAutoInstrumentations()plusRuntimeNodeInstrumentation()means http, express, mongodb, mysql2 and dns spans, andnodejs.eventloop.*/v8js.memory.heap.*/ GC metrics, all rendered to stdout.Simple*Processormeans one write per record, no batching.console.*is double-written. The bridge emits each call to the OTel Logs API and calls the original, so in the fallback path the same message goes to stdout twice: once as the rawconsole.*line and once as aConsoleLogRecordExporterrecord.CMDpreloadstelemetry.js, so this is the default behaviour of the published image, not something a user opts into.Suggested fixes
Any of these would have prevented it; the first two feel right together.
OTEL_CONSOLE_EXPORTERS=true. A process with nowhere to send telemetry should be quiet, not verbose.DEPLOY_ENV/NODE_ENVis already read a few lines above fordeployment.environment; refusing console exporters when it isproductioncosts nothing.BatchSpanProcessor/BatchLogRecordProcessoreven for console, and a much longerexportIntervalMillisfor the metric reader.Happy to send a PR for whichever shape you prefer.