This repository documents the results of the Typescript version bisect on Datadog frontend monorepo and how it was done. This helped identifying which commit introduced a performance regression in Datadog CI checks after the TS update from 5.5.4 to 5.6.2.
TL;DR -
This commit in 5.6.0-dev introduced the performance regression: 22bbe867fd4b86e2187270f6c284e6cfd92a3fea (full report).
In March 2023, we noticed a significant slowdown in our CI checks, particularly in TypeScript type-checking for Datadog frontend monorepo.
For info, in our CI, we typecheck our codebase in 2 parts:
typecheck:packages: typecheck and generate declaration files for all our packages (folders with their own tsconfig.json)typecheck:everything-else: we use the previous declaration files to typecheck all the other files of the codebase that are not contained in packages yet. The second step of our typecheck had the performance regression.
typecheck:everything-else time increased from around ~6 minutes to ~10 minutes. This regression appeared to coincide with upgrading from TypeScript 5.5.4 to newer versions (5.6+).
The performance impact was substantial:
- TypeScript 5.5.4: 336.02s build time
- TypeScript 5.6.2: 630.39s build time
The introduction of transformTime in the typecheck process clearly participated in the increase in timings. But even without accounting for transformTime, we still noticed a performance degradation.
Opened issue on TS repo: microsoft/TypeScript#61406
In order to make some commits of TS 5.6.0-dev work in our yarn monorepo, we had to adapt some of our configs. This increased the memory used and the build time compared to the values in the context. We still managed to identify the culprit commit 22bbe867. The table below lists the timings for this commit and the commit before:
| Commit Hash | TS Version | Build Time (s) | Build Time Without transformTime (s) | Report |
|---|---|---|---|---|
1b867c52 |
5.6.0-dev | 375.94 | 375.94 | report |
22bbe867 |
5.6.0-dev | 898.79 | 543.33 | report |
You can get the detailed output for each commit in the ./cleaned-timings folder.
As mentioned before, all commits of TS 5.6.0-dev were not compatible with our yarn monorepo setup, so we had to set the nodeLinker of yarn to node-modules to complete the bisect. This slightly changed the stats obtained with yarn tsc --extendedDiagnostics. Here are all timings obtained during the bisect of TS, compiled in 2 graphs:
This section will detail how the current method was found for bisecting different typescript versions on Datadog frontend monorepo. This part aims to document everything to potentially help for future internal TS investigations at Datadog.
At first, we tried to use the tool provided by Typescript: every-ts. This utility can build and use any TS version, by running every-ts switch <hash> && every-ts tsc. However, we were not able to properly use it in our yarn monorepo: it just couldn't resolve our modules (workspace ones and third parties).
So instead, we tried to take the generated TS repo used by every-ts, and link it with yarn: yarn link $(every-ts dir) so that it could access to our repo context, and run like a real TS install. But yarn has an ongoing issue (Libzip Error: Malloc failure) with packages of large sizes, so we were not able to use yarn link. We even tried to patch yarn internally by increasing the maximum memory in the libzip build.
After all these attempts, we finally realized that we could just build TS ourselves and install the generated .tgz file with yarn instead.
Since we wanted to bisect recent versions of TS, we were able to use the same build steps on all analyzed commits (from azure-pipelines.release.yml) without using every-ts:
npx hereby LKG
npx hereby clean
node ./scripts/addPackageJsonGitHead.mjs package.json
npm packThen we just had to replace all mentions of "typescript": "5.8.2" in all package.json within the frontend repo with "typescript": "file:/path/to/Typescript/typescript.tgz" and run yarn install.
This method worked for a good amount of commits during the bisect, but we started having issues between the commits e370c867 and a9139bfd of the 5.6.0-dev version:
➤ YN0066: │ typescript@patch:typescript@file%3A<path-to-ts-repo>/typescript.tgz%23<path-to-ts-repo>/typescript.tgz%3A%3Ahash=a6778c&locator=<path-in-dd-repo>#optional!builtin<compat/typescript>::version=5.5.0&hash=b45daf: Cannot apply hunk #16
➤ YN0013: │ 2 packages were added to the project, and 2 were removed (- 0.46 KiB).
➤ YN0000: └ Completed in 1s 898ms
➤ YN0000: ┌ Link step
➤ YN0000: │ ESM support for PnP uses the experimental loader API and is therefore experimental
➤ YN0008: │ web-ui@workspace:. must be rebuilt because its dependency tree changed
➤ YN0000: └ Completed in 3s 835ms
➤ YN0000: ┌ Post-install validation
➤ YN0001: │ TSError: ⨯ Unable to compile TypeScript:
error TS6053: File 'web-ui/tsconfig.focus.json' not found.
For the context, resolving modules in a repo that uses yarn as the package manager is not the same as having npm and its node_modules folder. In order to make Typescript able to handle yarn modules, yarn install automatically patches any TS packages that are added to the repo. This means that without this patch, TS just can't resolve modules out of the box with yarn (full context in this issue).
This is why we also got the Unable to compile TypeScript error when doing the following with every-ts as a workaround for yarn libzip issues:
yarn unplug typescript
export TS_UNPLUG_PATH = $DD_REPO + "/.yarn/unplugged/typescript-patch-26c53754b1/node_modules/typescript"
ln -sf $(every-ts dir)/bin $TS_UNPLUG_PATH
ln -sf $(every-ts dir)/lib $TS_UNPLUG_PATH
ln -sf $(every-ts dir)/package.json $TS_UNPLUG_PATH
yarn installGoing back to our issue, there are existing patches of TS for these version ranges: >=5.5.0-beta <5.5.2, >=5.5.2 <5.6.0-beta and >=5.6.0-beta <5.6.1-rc. The first patch worked until the commit e370c867 of 5.6.0-dev and the third patch started working for the commit a9139bfd of that same version. We had to modify the version field of the package.json of TS before building the tarball to 'trick' yarn and make it apply different patches.
As for why the patch stopped working, it was mainly due to yarn not being able to apply it because conflict issues: Cannot apply hunk #16. This error means that yarn couldn't apply the 16th @@ of the .diff file used to apply the patch (example of a TS patch).
So instead of trying to debug the patch, we discovered that we could make yarn change the way it exposes modules with this CLI:
yarn config set nodeLinker node-modules
This adds node_modules folders in all the packages of the repo, providing all the dependencies used by each package. With this, we were able to run yarn tsc and properly time its performances against the DD repo. However, since we couldn't use the optimized package manager of yarn, the memory used and the build time increased a bit, compared to our previous measurements (see the results).
The code in bisector.js and install-bisector.js is not very complicated, but it was tremendously fast to write using Cursor and Claude AI. We just had to give it the steps we wanted to reproduce between each bisect and how we wanted to save the measurements and the script was done in under a minute.
Obviously, TS changed between 5.5 and 5.6, and some types were erroring in 5.6 but not 5.5, and vice versa. We had to monkey patch our codebase to make these types pass in both versions, so that we could freely run our bisector. Most of the time, we just had to explicitly type some exported symbols, or cast it as unknown as <wanted-type>. Using AI on some complex symbols to explicitly type them was also very efficient here: Cursor can automatically retrieve all the needed files to get the full context, and do multiple attempts until TS passes for the modified file.