Skip to content

Distribute native binaries via npm using prebuildify (drop node-pre-gyp/S3) - #83

Merged
mohamed-rdr merged 5 commits into
masterfrom
feature/prebuildify-npm-binaries
Jul 7, 2026
Merged

mohamed-rdr merged 5 commits into
masterfrom
feature/prebuildify-npm-binaries

Conversation

@mohamed-rdr

@mohamed-rdr mohamed-rdr commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Switches @radarlabs/s2 from node-pre-gyp + S3 binary distribution to prebuildify + node-gyp-build, so the compiled N-API addon ships inside the npm tarball instead of being downloaded from node-s2-binaries.s3.amazonaws.com at install time.

This is the follow-up to the SER-97 tar/node-pre-gyp hardening (#82): rather than keep patching the node-pre-gyp/S3 toolchain, it removes that machinery entirely.

Why

  • No S3 / AWS in the release path. Publishing becomes just npm publish — no bucket to manage, no AWS credentials, no install-time network fetch that can 404 and silently fall back to a source compile.
  • Smaller consumer footprint. The whole compile toolchain (node-gyp, node-addon-api, and previously @mapbox/node-pre-gyp) moves to devDependencies, so it no longer appears in radarlabs/server's dependency tree or npm audit. node-gyp-build is the only runtime dependency.
  • N-API is ABI-stable across Node versions, so one prebuild per platform/arch serves Node 20/22/24 — 4 binaries instead of 12.

Changes

  • index.js: load via require('node-gyp-build')(__dirname).
  • package.json: drop @mapbox/node-pre-gyp, aws-sdk, bindings and the binary block; add node-gyp-build (dep) + prebuildify (devDep); buildprebuildify --napi --strip; add a files allowlist so the tarball ships only index.js / index.d.ts / prebuilds/ / README / LICENSE (previously it shipped .circleci, docker/, src/, jest config, publish scripts).
  • binding.gyp: literal target name, remove the node-pre-gyp copy target.
  • tests: load the package via require('..') instead of the node-pre-gyp.find() incantation.
  • .circleci/config.yml + docker/*.test: build the addon with prebuildify before testing.
  • New .github/workflows/release.yml: on a published GitHub Release, build the prebuild matrix (linux x64/arm64 + macOS arm64) on native runners, smoke-test each, then stage-publish the bundled package. Intel macOS (darwin-x64) is not shipped — Apple Silicon only for macOS.
  • Remove publish-linux.sh / publish-osx.sh (superseded by the release workflow).

Tarball before → after

npm pack --dry-run now yields 6 files (LICENSE, README.md, index.d.ts, index.js, package.json, prebuilds/…) instead of shipping CI/docker/source cruft.

Verification (local, darwin-arm64)

  • npm run build produces prebuilds/darwin-arm64/@radalabs+s2.node.
  • With build/ removed, require('./index.js') loads the prebuild and returns correct tokens (new s2.CellId(new s2.LatLng(...)).token() / .parent(10)).
  • All 8 suites / 45 tests pass.
  • CircleCI (this PR) builds + tests from source across Node 20/22/24.

Release / rollout notes

  • No secrets required. Publishing uses npm OIDC trusted publishing — no NPM_TOKEN, no AWS. One-time setup: on npmjs.com, add a Trusted Publisher for @radarlabs/s2 pointing at radarlabs/s2 + .github/workflows/release.yml, and set it to stage-only permissions (optionally disallow tokens). The job runs with id-token: write and npm attaches a provenance attestation automatically.
  • Staged publishing + 2FA gate. CI runs npm stage publish, so a release is staged, not public. A maintainer must approve it with 2FAnpm stage approve @radarlabs/s2 (or approve on npmjs.com) — before it goes live. The approve/reject actions can't use OIDC (they require proof-of-presence), so the human 2FA gate is enforced by npm itself.
  • To cut a release: bump version in package.json, then publish a GitHub Release with the matching tag (e.g. v0.0.7); the workflow builds all prebuilds and stages the publish. A maintainer then approves with 2FA to make it public.
  • Requirements handled by the job: npm >= 11.5.1 and Node >= 22.14 (the publish job upgrades npm and uses Node 22).
  • The GitHub-hosted ubuntu-24.04-arm + macos-13/14 runners are free for public repos.
  • One transitive high (semver ReDoS) remains inside node-gyp, now a devDependency, so it does not reach consumers.

…yp/S3)

Replaces the node-pre-gyp + S3 binary distribution with prebuildify +
node-gyp-build, so the compiled addon ships inside the npm tarball instead
of being downloaded from node-s2-binaries.s3.amazonaws.com at install time.

Why:
- No install-time network dependency on a self-managed S3 bucket, and no AWS
  credentials needed to release — publishing is just `npm publish`.
- The whole compile toolchain (node-gyp, node-addon-api) moves to
  devDependencies, so it no longer appears in consumers' dependency tree or
  npm audit. node-gyp-build is the only runtime dependency.
- The addon is N-API, which is ABI-stable across Node versions, so one
  prebuild per platform/arch serves Node 20/22/24 (4 binaries, not 12).

Changes:
- index.js: load via `require('node-gyp-build')(__dirname)`.
- package.json: drop @mapbox/node-pre-gyp, aws-sdk, bindings and the `binary`
  block; add node-gyp-build (dep) and prebuildify (devDep); `build` now runs
  `prebuildify --napi --strip`; add a `files` allowlist so the tarball ships
  only index/types/prebuilds/README/LICENSE.
- binding.gyp: literal target name, remove the node-pre-gyp copy target.
- tests: load the package via `require('..')` instead of node-pre-gyp.find().
- .circleci + docker: build the addon with prebuildify before testing.
- Add .github/workflows/release.yml: on a published GitHub Release, build the
  prebuild matrix (linux x64/arm64 + macOS x64/arm64) on native runners,
  smoke-test each, then `npm publish` the bundled package. Needs only NPM_TOKEN.
- Remove publish-linux.sh / publish-osx.sh (superseded by the release workflow).

Verified locally on darwin-arm64: prebuildify builds the prebuild, index.js
loads it with build/ removed, and all 8 suites / 45 tests pass.
@mohamed-rdr
mohamed-rdr requested review from binhrobles and jkao July 2, 2026 17:30
Drops the long-lived NPM_TOKEN in favor of npm trusted publishing: the
publish job mints a short-lived OIDC token (permissions: id-token: write),
so no secret is stored or rotated, and npm attaches a provenance attestation
automatically. Requires npm >= 11.5.1, so the job upgrades npm before
publishing. One-time setup: configure a Trusted Publisher for the package on
npmjs.com pointing at radarlabs/s2 + .github/workflows/release.yml.
…blic)

Switches the publish job to `npm stage publish` so CI never makes a release
public on its own. With the trusted publisher configured for stage-only
permissions, every CI publish is staged; a maintainer must then run
`npm stage approve` (or approve on npmjs.com) with 2FA to promote it — the
stage approve/reject actions require proof-of-presence and cannot use OIDC,
so the human 2FA gate is enforced by npm itself. Bumps the publish job to
Node 22 (trusted publishing requires Node >= 22.14).
@mohamed-rdr
mohamed-rdr force-pushed the feature/prebuildify-npm-binaries branch from d75e34c to 68ac721 Compare July 2, 2026 18:36
The macos-13 (Intel) runner is scarce and being deprecated by GitHub, making
it the slowest and most fragile leg of the matrix. Apple Silicon is standard,
production runs on linux, and the darwin-x64 leg only served local dev on Intel
Macs. Ship linux x64/arm64 + macOS arm64 only.
@mohamed-rdr
mohamed-rdr force-pushed the feature/prebuildify-npm-binaries branch from bb12fe6 to 88d2ba9 Compare July 2, 2026 18:46
include:
- { os: ubuntu-latest, name: linux-x64 }
- { os: ubuntu-24.04-arm, name: linux-arm64 }
- { os: macos-14, name: darwin-arm64 }

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we might not need this right?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping this one: it's the Apple Silicon prebuild, and the tarball has no source fallback, so without it npm install hard-fails on Macs. Production is linux-only, so it only affects local dev — leaving the thread open in case you'd rather drop macOS support entirely.

🤖 Addressed by Claude Code

Comment thread .github/workflows/release.yml Outdated
@mohamed-rdr
mohamed-rdr merged commit 82cd2fa into master Jul 7, 2026
3 checks passed
@mohamed-rdr
mohamed-rdr deleted the feature/prebuildify-npm-binaries branch July 7, 2026 17:03
@mohamed-rdr mohamed-rdr mentioned this pull request Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants