test: Remove old auto instrumentation e2e tests#1722
test: Remove old auto instrumentation e2e tests#1722Stephen Belanger (Qard) wants to merge 2 commits intomainfrom
Conversation
The end-to-end auto-instrumentation tests were using fs.writeFileSync to generate fixture scripts at test time, then fs.unlinkSync to clean up. When tests failed the cleanup never ran, leaving stale untracked files. Move each fixture to a static .mjs file in the fixtures/ directory and remove the dynamic write/unlink calls from the test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Luca Forstner (lforst)
left a comment
There was a problem hiding this comment.
Do you think instead of having these tests at all we could move them to the "new" E2E tests?
In general these tests seem to operate on a weird assertion level - like call api from user perspective but assert on an internal. I don't particularly like that.
These tests called the AI SDK from a user perspective but asserted on internal SPAN_DATA output via subprocess stdout parsing — an awkward level of assertion. Real coverage for this already exists in the e2e/ scenarios (anthropic-instrumentation, openai-instrumentation, etc.). Remove the end-to-end.test.ts file and its fixtures entirely. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
I've removed those. They were also validating a channel name mismatch though, which are implementation details, but also represents a real bug which probably should be covered. Should we check that internal value in the true e2e tests, or do you feel the e2e test passing is sufficient evidence of that being correct? I generally prefer more unit tests as they can catch a lot of cases where a thing works by accident and you don't realize until much later that it's been subtly broken and just not in a way that e2e tests reflected. |
In this particular case, I think the e2e tests are enough because with the current architecture, if the channels don't match the e2e tests will scream at you very loudly.
I personally like to have both. For things that are side-effect free and have a limited scope, I like unit tests. For everything that may be influenced by external factors and/or is more complicated and/or has side-effects, I usually prefer integration/e2e tests. I always gravitate towards the latter more because the coverage you get is much broader usually and also more reflects how users are seeing our product, which is arguably the most important thing. |
Summary
fs.writeFileSync, then cleaning up withfs.unlinkSync.mjsfiles in the fixtures directoryTest plan
cd js && pnpm test— end-to-end tests reference static fixture files directly🤖 Generated with Claude Code