feat: add request layer lifecycle TracingChannels#196
Conversation
Add a single `express:request` TracingChannel that emits structured
lifecycle events (start, end, asyncStart, asyncEnd, error) for every
middleware, route handler, and error handler execution.
Context shape: { req, res, layer } where layer is the Layer instance,
giving subscribers access to layer.name, layer.handle.length (for error
handler detection), and layer.route. Consumers decide how to classify
and name spans based on these properties.
Route dispatch wrapper layers (internal glue that calls route.dispatch)
are excluded from tracing to avoid duplicate events. The actual user
handlers inside the route are traced individually.
Zero overhead when no subscribers are registered. The hasSubscribers
check gates all context allocation and tracePromise wrapping.
Refs: pillarjs#96
Refs: expressjs/express#6353
Extract handlePromise and unify the traced/untraced paths into a single try/catch and .then handler.
Express TracingChannel PR opened: pillarjs/router#196
There was a problem hiding this comment.
Pull request overview
Adds request-layer lifecycle instrumentation using Node.js diagnostics_channel TracingChannel, emitting structured events for middleware, route handlers, and error handlers to support APM-style tracing with context propagation.
Changes:
- Add
express:requestTracingChannel integration inLayerrequest/error handling. - Introduce shared invocation helper to wrap handler execution and promise handling with tracing hooks.
- Add a new test suite validating emitted events, context shape, and basic ordering/behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| lib/layer.js | Adds TracingChannel wiring and refactors handler invocation through a tracing-aware helper. |
| test/tracing.js | New mocha tests covering tracing context, handler types, and lifecycle event expectations. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Qard
left a comment
There was a problem hiding this comment.
Generally LGTM. I think we might want to validate if handled errors are still captured on the span prior to the handler though.
| }) | ||
|
|
||
| describe('error handler tracing', function () { | ||
| it('should trace error handlers (fn.length === 4)', function (done) { |
There was a problem hiding this comment.
Should this one capture the error even though it is considered handled overall?
There was a problem hiding this comment.
Not entirely sure, APMs will usually have a global error handler or whatever to catch them so having them traced might be an overkill and may duplicate error emissions for those who don't dedup them.
WDYT?
There was a problem hiding this comment.
But does it having a handler here result in the error not being present on the span for the layer that lead to the error handler? Whatever the behaviour is, I think it would be useful to validate it more explicitly in this particular test. We should have a clear model of what errors happen where.
There was a problem hiding this comment.
Thanks! I added more tests to pin down various behaviors. Here is how errors flow right now:
No error handler (no app.use):
- Route throws sync ->
errorchannel publishes for the route layer. Router responds 500. - Route promise rejects ->
errorchannel publishes for the route layer. Router responds 500.
With error handler:
- Route throws sync ->
errorchannel publishes for the route, handler runs, nothing on theerrorchannel for the handler. - Route rejects async -> same,
errorchannel publishes for the route, handler clean.
With error handler that also throws/rejects:
- Route throws sync, handler throws ->
errorchannel publishes twice, once per layer. Two legitimately failed executions. - Route rejects async, handler throws -> same.
For an APM: both throw cases produce two error events on the channel — one with ctx.layer pointing at the route, one at the error handler. Each event targets its own span, so the APM just marks both spans as failed.
In the case for next(err) then the child error handler span will be marked, not the parent one. Which is fine I suppose since it technically recovered as you've said.
Tests in 1ad1b9c.
30c87c5 to
1ad1b9c
Compare
Qard
left a comment
There was a problem hiding this comment.
Hmm...this handling seems a bit incorrect. If an error is unhandled it should definitely be reported. And even if it is handled we should still probably be reporting it so we can capture cases like a too permissive error handler silencing legitimate failures.
I think probably what we should be doing is always capturing the error and just adding some handled flag to the context to mark when it has been handled somehow. That way we can still report it with the detail that the application thinks it has been handled while still allowing human judgement to look at the span and see if that assessment actually makes sense.
| }) | ||
| }) | ||
|
|
||
| it('should not emit error on the originating layer when next(err) is unhandled', function (done) { |
There was a problem hiding this comment.
This seems problematic. Unhandled errors should definitely always be observable.
There was a problem hiding this comment.
Okay agreed, this could've become a real gap and it's always easier to dedup rather than miss events.
I pushed a fix to that in 90ed7e2 and adjusted the tests to pin that behavior.
Qard
left a comment
There was a problem hiding this comment.
I think it's in a reasonable state now. Might be good to also get a review from @rochdev or @Flarna for some more perspectives from other APM vendors. The specific error semantics I think might be a bit up to interpretation how they should be handled, but I think this shape should generally provide the context that is needed for the different ways we think about route errors. 🤔
| * @private | ||
| */ | ||
|
|
||
| const requestChannel = dc.tracingChannel && dc.tracingChannel('express:request') |
There was a problem hiding this comment.
Why is express:request channel used here?
I think it would be better to use a dedicated channel for router to allow consumers to decide on what to receive and how to handle it.
I think router component can be also used without express.
There was a problem hiding this comment.
Good point, I don't think it has to be express prefixed but it needs a unique prefix for sure which is why I wanted some opinions on what it should be. My concern is router:request is too generic and can easily conflict with other names in the ecosystem.
I had this in the PR description:
The channel name right now is express:request, in the original PR it was request but I think channel names need to have a relationship with the module name otherwise we risk duplication in the ecosystem.
WDYT? Maybe pillarjs:router:request?
There was a problem hiding this comment.
There seems to be a guideline.
So I guess the string to use would be pillarjs.router.request as tracing: prefix and event name postfix is added by dc.tracingChannel().
There was a problem hiding this comment.
Doesn't seem like there is a consistent standard, docs do suggest that but others just went with lib:something:action formats.
Anyways I have no strong opinions here and I think as long as the name is unique enough, it is arbitrary. I renamed it to your suggestion 👍
There was a problem hiding this comment.
I think it should just be router.request and not pillarjs.router.request. I say this because pillarjs is more like the organization name and that could change, and we might move this package to the expressjs organization (not saying we’ll do it now, but it could happen, so we shouldn’t be tied to the GitHub organization name).
There was a problem hiding this comment.
Okay, I understand the concern. In that case, instead of using pillarjs, it’s better to use express, since it’s officially under the expressjs project.
How does Fastify handle it?
There was a problem hiding this comment.
So should we go for express.request or express.router.request? To me its arbitrary but perhaps you have a strong opinion?
Fastify uses fastify.request.handler as seen here.
There was a problem hiding this comment.
I think express.request is a bit too generic. There might be other parts in express describing the request lifecycle where the router component is not included.
e.g. express.request.router or express.router.request - tending towards express.request.router like fastify and hierarchic ordering.
There was a problem hiding this comment.
Went with express.router.request since it describes the action better "Express's router is handling a request".
But happy to change to whatever that is acceptable to get this through.
Follows the Node.js TracingChannel naming guideline (dot-separated, module-scoped) and avoids the express-specific prefix since router is usable outside express.
|
The actual data published by the tracing channel is the If I understand the router module correct Also I don't find in the docs here that In general I would love some section in the docs describing the diag channel and it's emitted data. |
I can't answer that as it usually up to maintainers to decide this but I would say it does imply a public API contract and should be subject to smever. As for the The
I think we have 3 options here:
I think that would be up to the maintainers to decide, but all options work for us. |
|
I generally suggest that event contents be considered public API surface, but I would not say that is mandatory as it is very common that diagnostics_channel events would be observing what would otherwise be internal implementation details. I would ask though that any exposure of internals be explicitly called out as internal to inform consumers that they should consume those events with defensive accesses to handle cases where shapes may change. I don't fully think calling it public API surface is quite the right characterization of the intent. |
Yes, it’s part of the public API (see Line 281 in cb2a5fc |
bjohansebas
left a comment
There was a problem hiding this comment.
Please also add documentation about this in the README.
|
@bjohansebas I added a section on diagnostic channels and their payload, and marked |
| * Check if the channel has subscribers. | ||
| */ | ||
| function shouldTrace (ch) { | ||
| return ch && ch.hasSubscribers !== false |
There was a problem hiding this comment.
Since we support Node 18.0.0, users on those versions will get true, because (undefined !== false) === true.
https://nodejs.org/api/diagnostics_channel.html#diagnostics_channeltracingchannelnameorchannels
There was a problem hiding this comment.
Yep this is a trade off I suppose. I think we should then check for all channels own subscribers in that case.
| - `req`: the incoming `http.IncomingMessage` | ||
| - `res`: the `http.ServerResponse` | ||
| - `layer`: the internal `Layer` instance being invoked (exposes `.name`, `.path`, `.handle`, etc.). Note that `Layer` is an internal implementation detail and its shape may change between releases. | ||
| - `error`: the error passed to `next(err)`, when applicable |
There was a problem hiding this comment.
If someone did something like this, I know it's a very edge case, but it's still something we should fix since it's allowed by the way the router is designed:
router.get('/via-throw', function viaThrow (req, res, next) {
throw 'route' // eslint-disable-line no-throw-literal
})
router.get('/via-throw', (req, res) => res.end('second route ran'))the error event would still be emitted, so it wouldn't be limited to just next(err). Even so, the request would still end up returning a 200.
There was a problem hiding this comment.
I added an exception for these routing signals, should be ignored now.
| if (err && err !== 'route' && err !== 'router') { | ||
| ctx.error = err | ||
| // Explicitly publish the error to the error channel | ||
| requestChannel.error.publish(ctx) |
There was a problem hiding this comment.
Okay, we have a small bug here. When there are mounted sub-routers, if an error occurs, the parent router ends up publishing a duplicate error event. For example:
const http = require('node:http')
const dc = require('node:diagnostics_channel')
const Router = require('/home/bjohansebas/Dev/oss/pillarjs/router')
const errorEvents = []
dc.tracingChannel('express.router.request').subscribe({
error (ctx) {
errorEvents.push({
layer: ctx.layer && ctx.layer.name,
handled: !!ctx.handled,
message: ctx.error && ctx.error.message
})
}
})
function report (title) {
console.log('\n%s', title)
console.log(' error events published: %d %s', errorEvents.length,
errorEvents.length === 1 ? '(OK)' : '(BUG: should be exactly 1, on the origin layer)')
for (const e of errorEvents) {
console.log(' layer=%s handled=%s message=%j', e.layer, e.handled, e.message)
}
errorEvents.length = 0
}
async function run (router, path) {
const server = http.createServer(function (req, res) {
router(req, res, function (err) {
res.statusCode = err ? 500 : 404
res.end(String(err || 'not found'))
})
})
await new Promise(r => server.listen(0, r))
await fetch('http://localhost:' + server.address().port + path)
await new Promise(r => setImmediate(r))
server.close()
}
async function main () {
// Scenario A: mounted sub-router — a single next(err) in the inner handler,
// recovered by an outer error handler.
{
const outer = new Router()
const nested = new Router()
nested.get('/bar', function innerHandler (req, res, next) {
next(new Error('boom'))
})
outer.use('/foo', nested)
outer.use(function recover (err, req, res, next) { // eslint-disable-line no-unused-vars
res.statusCode = 500
res.end(err.message)
})
await run(outer, '/foo/bar')
report('Scenario A: nested router (1 mount level)')
}
// Scenario A2: two mount levels — the error is republished on EVERY ancestor.
{
const outer = new Router()
const mid = new Router()
const deep = new Router()
deep.get('/baz', function deepHandler (req, res, next) {
next(new Error('boom'))
})
mid.use('/bar', deep)
outer.use('/foo', mid)
outer.use(function recover (err, req, res, next) { // eslint-disable-line no-unused-vars
res.statusCode = 500
res.end(err.message)
})
await run(outer, '/foo/bar/baz')
report('Scenario A2: nested router (2 mount levels)')
}
// Scenario B: error handler that forwards the error with next(err).
{
const router = new Router()
router.get('/fail', function origin (req, res, next) {
next(new Error('boom'))
})
router.use(function forwardingHandler (err, req, res, next) {
next(err) // does not handle it, delegates to the next one
})
router.use(function recover (err, req, res, next) { // eslint-disable-line no-unused-vars
res.statusCode = 500
res.end(err.message)
})
await run(router, '/fail')
report('Scenario B: error handler forwarding with next(err)')
}
}
main()In principle, the error event should only be published once.
There was a problem hiding this comment.
That's a good catch. I will dedup the errors published in relation to the request object.
|
|
||
| try { | ||
| const ret = tracing | ||
| ? requestChannel.tracePromise(function () { return handlePromise(exec(wrappedNext)) }, ctx) |
There was a problem hiding this comment.
If someone does this, or manages to trigger it somehow:
router.get('/reject', async function rejectingHandler (req, res) {
throw undefined // same as: return Promise.reject()
})the error emitted in the event would be undefined, when it should instead be Error: Rejected promise.
There was a problem hiding this comment.
Usually this is fine, but since it gets normalized later it the channel error should match. Fixing this.
…el#hasSubscribers TracingChannel#hasSubscribers was added in Node 20.13/22 and is undefined on older supported versions, where `hasSubscribers !== false` is always true and forces tracing on every request. Check the sub-channels instead, which expose hasSubscribers on all supported versions.
Move the no-subscriber decision into handleRequest/handleError so the untraced path allocates no context object per request, matching the behavior on master. invokeWithTrace now receives the context object directly instead of a factory, since it only runs when tracing is active.
An error propagating through mounted sub-routers or forwarding error handlers was published on the error channel once per layer it passed through, so a single failure produced duplicate error events. Track the errors already reported for a request and publish each only at the layer where it originated.
A handler that throws 'route' or 'router' uses the same routing signals
as next('route')/next('router') to skip the route or exit the router.
Those were reaching the error channel because tracePromise reports any
thrown value; catch them before tracePromise sees them so they aren't
published as errors, matching how the signals are handled via next().
A handler that rejects or throws a falsy value (Promise.reject(), throw
undefined) was reported on the error channel as that raw value, while the
router forwards new Error('Rejected promise') to next(). Normalize once,
before tracePromise reports it, so the channel and downstream handlers see
the same error instance.
| const ret = requestChannel.tracePromise(function () { | ||
| // Catch thrown/rejected routing signals here so tracePromise doesn't | ||
| // report them as errors; real errors propagate and are reported as usual. | ||
| // A falsy rejection is normalized to the same error the router forwards to | ||
| // next(), so the error channel and downstream handlers see one instance. | ||
| let out | ||
| try { | ||
| out = handlePromise(exec(wrappedNext)) | ||
| } catch (err) { | ||
| if (isRoutingSignal(err)) { | ||
| next(err) | ||
| return | ||
| } | ||
| throw err || new Error('Rejected promise') | ||
| } | ||
|
|
||
| // wait for returned promise | ||
| if (isPromise(ret)) { | ||
| if (!(ret instanceof Promise)) { | ||
| deprecate('handlers that are Promise-like are deprecated, use a native Promise instead') | ||
| if (out) { | ||
| return out.then(undefined, function (err) { | ||
| if (isRoutingSignal(err)) { | ||
| next(err) | ||
| return | ||
| } | ||
| throw err || new Error('Rejected promise') | ||
| }) | ||
| } | ||
| }, ctx) | ||
|
|
||
| if (ret) { | ||
| ret.then(null, function (error) { | ||
| next(error || new Error('Rejected promise')) | ||
| // tracePromise already reported the rejection on this layer (with the | ||
| // normalized error); record it so outer layers don't report it again. | ||
| recordError(ctx.req, error) | ||
| next(error) | ||
| }) | ||
| } | ||
| } catch (err) { | ||
| // tracePromise already reported the sync throw on this layer; record it so | ||
| // outer layers don't report it again. | ||
| recordError(ctx.req, err) | ||
| next(err) | ||
| } |
There was a problem hiding this comment.
Okay, I think we can simplify this further while also fixing a few other cases where I found differences compared to master. I've described those cases in the comments, and you can take a look and decide whether you think this is the best approach.
| const ret = requestChannel.tracePromise(function () { | |
| // Catch thrown/rejected routing signals here so tracePromise doesn't | |
| // report them as errors; real errors propagate and are reported as usual. | |
| // A falsy rejection is normalized to the same error the router forwards to | |
| // next(), so the error channel and downstream handlers see one instance. | |
| let out | |
| try { | |
| out = handlePromise(exec(wrappedNext)) | |
| } catch (err) { | |
| if (isRoutingSignal(err)) { | |
| next(err) | |
| return | |
| } | |
| throw err || new Error('Rejected promise') | |
| } | |
| // wait for returned promise | |
| if (isPromise(ret)) { | |
| if (!(ret instanceof Promise)) { | |
| deprecate('handlers that are Promise-like are deprecated, use a native Promise instead') | |
| if (out) { | |
| return out.then(undefined, function (err) { | |
| if (isRoutingSignal(err)) { | |
| next(err) | |
| return | |
| } | |
| throw err || new Error('Rejected promise') | |
| }) | |
| } | |
| }, ctx) | |
| if (ret) { | |
| ret.then(null, function (error) { | |
| next(error || new Error('Rejected promise')) | |
| // tracePromise already reported the rejection on this layer (with the | |
| // normalized error); record it so outer layers don't report it again. | |
| recordError(ctx.req, error) | |
| next(error) | |
| }) | |
| } | |
| } catch (err) { | |
| // tracePromise already reported the sync throw on this layer; record it so | |
| // outer layers don't report it again. | |
| recordError(ctx.req, err) | |
| next(err) | |
| } | |
| requestChannel.tracePromise(function () { | |
| let out | |
| try { | |
| out = handlePromise(exec(wrappedNext)) | |
| } catch (err) { | |
| // forwarded verbatim: a falsy throw keeps routing, like the untraced path | |
| wrappedNext(err) | |
| return | |
| } | |
| if (out) { | |
| return out.then(undefined, function (err) { | |
| wrappedNext(err || new Error('Rejected promise')) | |
| }) | |
| } | |
| }, ctx) |
There was a problem hiding this comment.
For example, when a falsy value is thrown in synchronous mode, like in the following example:
dc.tracingChannel('express.router.request').subscribe(handlers)
router.get('/falsy', function throwsFalsy(req, res) {
throw undefined // eslint-disable-line no-throw-literal
})
router.get('/falsy', function continues(req, res) {
res.statusCode = 200
res.end('continued')
})
request(server)
.get('/falsy')
.expect(200, 'continued', function (err) {
if (err) return done(err)
const errorEvents = events.filter(function (e) { return e.phase === 'error' })
assert.equal(errorEvents.length, 0,
'a falsy throw is treated as "no error" by the router — subscribing must not divert it to error handling')
done()
})Without tracing enabled, the router doesn't treat this as an error and simply continues as normal. However, once tracing is enabled, it now automatically treats it as an error. It's also interesting that we don't currently have a test covering the normal behavior. It's definitely an edge case, but enabling tracing shouldn't change the router's behavior.
Could you add a test for this case
it('should continue routing after throwing a falsy value', function (done) {
const router = new Router()
const server = createServer(router)
router.use(function handle (req, res, next) {
throw undefined // eslint-disable-line no-throw-literal
})
router.use(sawError)
router.use(helloWorld)
request(server)
.get('/')
.expect(200, 'hello, world', done)
})
``There was a problem hiding this comment.
Another case we have is when the same error is rethrown. It's technically valid—people do some crazy things 😄—but here's the scenario:
dc.tracingChannel('express.router.request').subscribe(handlers)
router.get('/fail', function origin (req, res, next) {
next(new Error('boom'))
})
router.use(function rethrowing (err, req, res, next) { // eslint-disable-line no-unused-vars
throw err
})
router.use(function recover (err, req, res, next) { // eslint-disable-line no-unused-vars
res.statusCode = 500
res.end(err.message)
})
request(server)
.get('/fail')
.expect(500, 'boom', function (err) {
if (err) return done(err)
const errorEvents = events.filter(function (e) { return e.phase === 'error' })
assert.equal(errorEvents.length, 1,
'rethrowing the same error must not re-report it — throw and next(err) are equivalent to the router')
assert.equal(errorEvents[0].ctx.layer.name, 'origin')
done()
})In this case, it currently emits the error event twice. I don't think that makes sense—we shouldn't report the same error twice. From the router's perspective, throw err and next(err) are equivalent, so a single error should only produce a single error event.
It all depends on the user's coding style.
There was a problem hiding this comment.
Yep that looks much better, I enumerated all cases again and ran what I just pushed against them, should be good now 👍
Thanks for double checking here!
Replace the repeated 'express.router.request' literal with a CHANNEL constant, and hoist the per-test byLayer/byName predicate into a single shared helper at the bottom of the file.
Send every handler outcome (return, throw, rejection) through wrappedNext instead of re-throwing into tracePromise's own error publishing. This puts signal filtering and error dedup in one place and keeps two behaviors in line with the untraced router: - a sync falsy throw is forwarded verbatim, so it keeps routing instead of being turned into an error - an error rethrown by an error handler is the same error, so it is reported once at its origin rather than twice Async rejections with a falsy reason are still normalized to the error the router forwards to next(), reported once.
Add a traced() helper for the router+server+subscribe setup, a byPhase() predicate to match the byLayer() one, and a shared recover error handler, replacing the copies repeated across the tracing tests.
This PR adds a single tracing channel that emits structured lifecycle events (start, end, asyncStart, asyncEnd, error) for every middleware, route handler, and error handler execution.
This builds on #96 but uses tracing channels instead which enables context propagation out of the box.
Context shape
The context object is pretty minimal, providing both
reqandresenables APMs to figure out how to react to that specific payload.The
layeris the Layer instance. Subscribers uselayer.name,layer.handle.length(4 for error handlers), andreq.route(set when inside a matched route) to classify and name spans however they see fit.Usage Example:
This is an example I've set up locally using a Sentry setup with this PR patched in:
This produces a waterfall view looking like this:
Note: The three spans that look the same at the bottom actually represent the three handlers executing for that route, it is up to the subscriber to decide which one is the route handler and how to name the other ones since it has access to
layer.Is tracing channels experimental?
We had this come up a few times during the initiative of pushing the adoption of dc across the ecosystem, most recently this was brought up and responded to by @Qard here:
Automattic/mongoose#16105 (comment)
Related work
Caveats
pillarjs.router.request, in the original PR it wasrequestbut I think channel names need to have a relationship with the module name otherwise we risk duplication in the ecosystem.hasSubscriberswill assume that there is a subscriber and will execute the traced path (with context object allocation). Multiple libraries likeredisandmysql2accepted this limitation.tracingChannelsupport will just not offer tracing of any kind. The fallback here is NOOP.tracingChannelwill be skipped viadescribe.skip.