Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 31 additions & 9 deletions .appveyor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,11 @@ init:

environment:
matrix:
# Build Node.js
# Build Node.js. The native addon is built with Bazel (there is no
# node-gyp source build anymore); use a VS2022 worker for the Bazel
# MSVC toolchain.
- nodejs_version: stable
APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2022

# Build plain C++
- nodejs_version: none
Expand All @@ -32,15 +35,36 @@ for:
install:
- ps: Install-Product node $env:nodejs_version $env:platform
- npm -g i npm@latest
# @bazel/bazelisk provides both `bazel` and `bazelisk` commands on the
# npm global bin directory, which is already on PATH (unlike choco shims
# installed mid-build).
- appveyor-retry npm -g i @bazel/bazelisk
- bazel --version
build_script:
- set npm_config_build_from_source=true&& appveyor-retry call npm install
- appveyor-retry call npm install --omit=optional
# Only the addon itself is built with Bazel here. The Bazel dictionary
# genrules run rules_python py_binary tools whose venv bootstrap needs
# symlink support that AppVeyor workers lack, so dictionaries and configs
# come from the CMake install below (also used for the jieba plugin
# packaging), where data/scripts run on the system Python directly.
- echo [opencc npm package] building opencc win32-x64 addon with Bazel
- bazel build -c opt //node:opencc
- echo [opencc npm package] building dictionaries and jieba plugin with CMake
- cmake -A%platform% -S. -Bbuild-npm-jieba -DCMAKE_INSTALL_PREFIX:PATH=%CD%\build-npm-jieba-install -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS:BOOL=OFF -DBUILD_OPENCC_JIEBA_PLUGIN:BOOL=ON -DENABLE_GTEST:BOOL=OFF -DENABLE_BENCHMARK:BOOL=OFF
- cmake --build build-npm-jieba --config Release --target install
- ps: |
New-Item -ItemType Directory -Force prebuilds\assets | Out-Null
New-Item -ItemType Directory -Force prebuilds\win32-x64 | Out-Null
Copy-Item build-npm-jieba-install\share\opencc\*.ocd2 prebuilds\assets\
Get-ChildItem build-npm-jieba-install\share\opencc\*.json |
Where-Object { $_.Name -notlike '*jieba*' -and $_.Name -notlike '*.schema.json' } |
Copy-Item -Destination prebuilds\assets\
Copy-Item bazel-bin\node\opencc.node prebuilds\win32-x64\opencc.node
test_script:
- npm test
- set PREBUILDS_ONLY=1&& npm test
after_test:
- chcp 65001
- echo [opencc npm package] building opencc win32-x64 prebuild
- if exist prebuilds rmdir /S /Q prebuilds
- npm run prebuild
- echo [opencc npm package] verifying opencc win32-x64 prebuild layout
- if exist prebuilds\darwin-arm64 exit /B 1
- if exist prebuilds\linux-x64 exit /B 1
- if exist prebuilds\linux-arm64 exit /B 1
Expand All @@ -52,9 +76,7 @@ for:
- echo [opencc npm package] packing opencc win32-x64 scoped binary package
- npm pack dist\scoped-packages\@opencc\opencc-win32-x64
- for %%F in (opencc-opencc-win32-x64-*.tgz) do set OPENCC_WIN32_TGZ=%%~fF
- echo [opencc npm package] building opencc-jieba win32-x64 package files
- cmake -A%platform% -S. -Bbuild-npm-jieba -DCMAKE_INSTALL_PREFIX:PATH=%CD%\build-npm-jieba-install -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS:BOOL=OFF -DBUILD_OPENCC_JIEBA_PLUGIN:BOOL=ON -DENABLE_GTEST:BOOL=OFF -DENABLE_BENCHMARK:BOOL=OFF
- cmake --build build-npm-jieba --config Release --target install
- echo [opencc npm package] assembling opencc-jieba win32-x64 package files
- mkdir plugins\jieba\node\prebuilds\win32-x64
- mkdir plugins\jieba\node\data
- copy build-npm-jieba-install\share\opencc\*jieba.json plugins\jieba\node\data\
Expand Down
105 changes: 97 additions & 8 deletions .github/workflows/nodejs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -98,11 +98,90 @@ jobs:
prebuilds/**
**/npm-debug.log

# Install the packed npm tarball without prebuilt binaries, so the
# node-gyp source build path is exercised exactly as it runs on platforms
# without a scoped binary package (issue #1409).
# Pack the main npm tarball plus the Bazel-built scoped binary package for
# the host platform, install both together, and smoke test — mirroring how
# a release install resolves the native addon on platforms with a prebuilt
# binary.
pack-install:
name: Install packed npm tarballs (${{ matrix.os }})
permissions:
contents: read
runs-on: ${{ matrix.os }}

strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, macos-15-intel]
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0
- name: Use Node.js
uses: actions/setup-node@v5
with:
node-version: 24
- uses: bazel-contrib/setup-bazel@0.19.0
with:
bazelisk-cache: true
repository-cache: true
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.12"
- name: Configure BuildBuddy
if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository
shell: bash
env:
BUILDBUDDY_API_KEY: ${{ secrets.BUILDBUDDY_API_KEY }}
run: |
if [ -n "${BUILDBUDDY_API_KEY}" ]; then
{
echo "build --config=buildbuddy"
echo "build --remote_header=x-buildbuddy-api-key=${BUILDBUDDY_API_KEY}"
echo "build --bes_header=x-buildbuddy-api-key=${BUILDBUDDY_API_KEY}"
} >> .bazelrc.user
echo "BuildBuddy enabled."
else
echo "BUILDBUDDY_API_KEY not set; skipping BuildBuddy config."
fi
- run: npm ci --omit=optional
- name: Build Node.js prebuild with Bazel
shell: bash
run: ./scripts/build-node-prebuild-bazel.sh
- name: Pack npm tarballs
shell: bash
run: |
npm run prepare:scoped-packages
target="$(node -p "process.platform + '-' + process.arch")"
mkdir -p dist/npm-artifacts
npm pack --pack-destination dist/npm-artifacts
npm pack "dist/scoped-packages/@opencc/opencc-${target}" \
--pack-destination dist/npm-artifacts
- name: Install tarballs and smoke test
shell: bash
run: |
main_tarball="$(ls "$PWD"/dist/npm-artifacts/opencc-[0-9]*.tgz)"
scoped_tarball="$(ls "$PWD"/dist/npm-artifacts/opencc-opencc-*.tgz)"
install_dir="$(mktemp -d)"
cd "$install_dir"
npm init -y
npm install --omit=optional --foreground-scripts \
"$main_tarball" "$scoped_tarball"
node -e '
const assert = require("assert");
const OpenCC = require("opencc");
const converter = new OpenCC("s2t.json");
const converted = converter.convertSync("汉字");
assert.strictEqual(converted, "漢字");
console.log("Packed opencc converts 汉字 ->", converted);
'

# Install the packed npm tarball without any scoped binary package, so the
# Bazel source-build fallback in scripts/install.js is exercised exactly as
# it runs on platforms without a prebuilt binary: the addon is compiled
# from source and the dictionaries are regenerated into prebuilds/assets.
pack-source-install:
name: Source build from npm tarball (${{ matrix.os }})
name: Bazel source build from npm tarball (${{ matrix.os }})
permissions:
contents: read
runs-on: ${{ matrix.os }}
Expand All @@ -119,9 +198,18 @@ jobs:
uses: actions/setup-node@v5
with:
node-version: 24
- run: npm ci
- uses: bazel-contrib/setup-bazel@0.19.0
with:
bazelisk-cache: true
repository-cache: true
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.12"
- run: npm ci --omit=optional
- name: Build prebuild assets
run: npm run prebuild
shell: bash
run: ./scripts/build-node-prebuild-bazel.sh
- name: Pack npm tarball
shell: bash
run: |
Expand All @@ -130,12 +218,13 @@ jobs:
- name: Install tarball from source and smoke test
shell: bash
run: |
tarball="$(ls "$PWD"/dist/npm-artifacts/opencc-*.tgz)"
tarball="$(ls "$PWD"/dist/npm-artifacts/opencc-[0-9]*.tgz)"
target="$(node -p "process.platform + '-' + process.arch")"
install_dir="$(mktemp -d)"
cd "$install_dir"
npm init -y
npm install --omit=optional --foreground-scripts "$tarball"
test -f node_modules/opencc/build/Release/opencc.node
test -f "node_modules/opencc/prebuilds/${target}/opencc.node"
node -e '
const assert = require("assert");
const OpenCC = require("opencc");
Expand Down
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,10 @@ This document compiles the Open Chinese Convert (OpenCC) project information to
- The command-line tool `opencc_dict` supports `text ↔ ocd2` (and optionally `ocd`) conversion. When adding or adjusting dictionaries, first edit `.txt`, then run the tool to generate the target format.

## Development and Testing
- The top-level build system supports CMake, Bazel, Node.js `binding.gyp`, Python `pyproject.toml`, with cross-platform CI integration.
- The top-level build system supports CMake, Bazel, Python `pyproject.toml`, with cross-platform CI integration. The Node.js native addon is built with Bazel (`//node:opencc`).
- `src/*Test.cpp`, `data/config/*Test.cpp`, `plugins/jieba/tests/`, `test/`, and `test/golden/` contain tests covering dictionary matching, conversion chains, configuration validation, plugin segmentation, CLI behavior, and golden conversion outputs.
- Tools `opencc_dict`, `opencc_phrase_extract` (`src/tools/`) help developers convert dictionary formats and extract phrases.
- Node.js tests live in `node/test.js`. Official npm prebuilt binaries are built with Bazel via `scripts/build-node-prebuild-bazel.sh` (see `node/PUBLISHING.md`); `npm run prebuild` (prebuildify/node-gyp) is no longer the release path, and the `binding.gyp` source fallback is planned for removal.
- Node.js tests live in `node/test.js`. npm prebuilt binaries are built with Bazel via `scripts/build-node-prebuild-bazel.sh` (see `node/PUBLISHING.md`). The `binding.gyp`/node-gyp source-build fallback has been replaced by a Bazel one: the native addon comes from `@opencc/opencc-<platform>-<arch>` scoped packages, and on platforms without one `scripts/install.js` runs a full Bazel source build at install time (compiles `//node:opencc`, regenerates `//data/dictionary:binary_dictionaries`, and refreshes `prebuilds/assets`; prebuilt assets still ship in the tarball for the scoped-package path).

### C++ ABI Versioning
- When a change introduces an ABI-incompatible modification to the public C++ interface, bump `OPENCC_ABI_VERSION` in `CMakeLists.txt` so downstream libraries and applications relink instead of silently loading an incompatible `libopencc` shared library.
Expand Down
1 change: 0 additions & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,6 @@ set(_OPENCC_TRIMMED_SOURCE_PACKAGE_IGNORE_FILES
"/Makefile"
"/OpenCC.egg-info/"
"/PUBLICATIONS.md"
"/binding.gyp"
"/build.cmd"
"/data/icon/"
"/deps/google-benchmark/"
Expand Down
4 changes: 1 addition & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,9 +192,7 @@ ctest --test-dir build --output-on-failure

### Node.js 測試

Node.js binding 與 npm CLI 修改請執行(addon 以 Bazel 構建,源碼倉庫內
`npm install` 不會再觸發 node-gyp 編譯;如需舊的 node-gyp 構建,可用
`npm install --build-from-source`):
Node.js binding 與 npm CLI 修改請執行(addon 以 Bazel 構建):

```bash
npm install --omit=optional
Expand Down
5 changes: 5 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# Change History of OpenCC

## Unreleased

* **Node.js**:
* npm 套件的源碼編譯 fallback 從 node-gyp 改為 Bazel:原生 addon 優先透過 `@opencc/opencc-<platform>-<arch>` 預編譯 scoped 套件安裝(涵蓋 macOS x64/arm64、Linux x64/arm64、Windows x64,均以 Bazel 構建,字典隨包預編譯);沒有預編譯二進位套件的平台,`npm install` 會以 Bazel 完整地從源碼構建——編譯 `//node:opencc` 並重新生成字典(`bazel` / `bazelisk` 取自 PATH,或經 `npx` 自動下載 bazelisk;字典生成腳本的 Python 由 Bazel hermetic 工具鏈提供)。`binding.gyp` 與 `.gypi` 構建定義刪除,`node-gyp`、`node-gyp-build`、`node-addon-api`、`prebuildify` 依賴與 `prebuild` npm script 一併移除;tarball 不再攜帶 `deps/`(C++ 三方依賴改由 Bazel Central Registry 提供),改為攜帶 Bazel workspace、`src/` C++ 源碼與 `data/` 字典源檔。

## Version 1.4.1

2026年7月12日
Expand Down
11 changes: 8 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,9 +78,14 @@ https://opencc.js.org/converter?config=s2t

`npm install opencc`

The npm package supports Node.js `>=20.17`. It uses bundled Node-API
prebuilds when available and falls back to a local `node-gyp` build when the
current platform does not have a matching prebuild.
The npm package supports Node.js `>=20.17`. The native addon is installed
through prebuilt `@opencc/opencc-<platform>-<arch>` packages covering macOS
(x64/arm64), Linux (x64/arm64), and Windows (x64). On platforms without a
prebuilt binary package, `npm install` builds from source with Bazel —
compiling the addon and regenerating the dictionaries — which requires a
C++ toolchain and network access; Bazel is found on PATH (`bazel` or
`bazelisk`) or fetched automatically via `npx`, and Bazel downloads its own
hermetic Python toolchain for the dictionary generation.

Bun and Deno can also use the npm package through their npm compatibility
support.
Expand Down
8 changes: 0 additions & 8 deletions binding.gyp

This file was deleted.

4 changes: 2 additions & 2 deletions data/config/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ filegroup(
"//src:simple_converter_lib",
"@bazel_tools//tools/cpp/runfiles",
"@googletest//:gtest_main",
"//deps/rapidjson-1.1.0:rapidjson",
"@rapidjson//:rapidjson",
],
)
for config in CONFIGS
Expand Down Expand Up @@ -77,7 +77,7 @@ test_suite(
deps = [
"@bazel_tools//tools/cpp/runfiles",
"@googletest//:gtest_main",
"//deps/rapidjson-1.1.0:rapidjson",
"@rapidjson//:rapidjson",
],
)
for config in CONFIGS
Expand Down
3 changes: 1 addition & 2 deletions node/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,7 @@ cc_header_pick(
# Node C headers itself (the rules_nodejs toolchain on Unix; the windows_node_api
# import library on Windows).
#
# node-gyp and Bazel both compile the same binding source. node-gyp lists the
# OpenCC and marisa sources directly, while Bazel links against //src:opencc_lib.
# The binding source is compiled here and linked against //src:opencc_lib.
node_addon(
name = "opencc",
srcs = ["opencc.cc"],
Expand Down
52 changes: 30 additions & 22 deletions node/PUBLISHING.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,17 @@ family.

The current npm layout is split by platform:

- `opencc` contains the JavaScript API, CLI, TypeScript declarations, source
fallback, and runtime dictionary/config assets.
- `opencc` contains the JavaScript API, CLI, TypeScript declarations, and
runtime dictionary/config assets.
- `@opencc/opencc-<platform>-<arch>` contains exactly one native
`opencc.node` binary for one platform.

The split-package layout is experimental. It keeps the main npm package smaller
while preserving source-build fallback for platforms without a scoped binary
package.
The native addon normally comes from the scoped binary packages. On platforms
without one, `npm install` falls back to a full Bazel source build
(`scripts/install.js`): it compiles the addon, regenerates the dictionaries,
and refreshes `prebuilds/assets`. The tarball ships the Bazel workspace, the
C++ sources, and the dictionary sources for this; prebuilt assets are still
included for the scoped-package fast path.

## Package names

Expand All @@ -34,8 +37,8 @@ binary refresh, publish all scoped packages at the same version as `opencc`.

- npm account access to publish `opencc`.
- npm organization access to publish public packages under `@opencc`.
- Node.js, npm, Python, and a native C++ build toolchain for the target
platform.
- Node.js, npm, Python, Bazel (bazelisk), and a native C++ build toolchain
for the target platform.
- GitHub Actions or another set of machines covering each supported target
platform.

Expand Down Expand Up @@ -64,11 +67,10 @@ prebuilds/assets/*.ocd2
prebuilds/<current-platform>-<current-arch>/opencc.node
```

The release workflow runs the Bazel build script separately on the runner for
each scoped binary target. The main `opencc` package still keeps `binding.gyp`
and source files so installs on platforms without a scoped binary package can
fall back to `node-gyp-build`. This fallback is planned for removal in a
future release, after which all Node builds will use Bazel exclusively.
The release workflow (`release-npm-binaries`) runs the Bazel build script
separately on the runner for each scoped binary target. All Node builds use
Bazel exclusively — including the install-time source-build fallback, which
replaced the former `binding.gyp`/node-gyp build.

## Prepare scoped binary packages

Expand Down Expand Up @@ -105,8 +107,10 @@ Dry-run the main package:
npm pack --dry-run
```

The main package tarball should contain `node/`, `src/`, `data/`, source build
files, and `prebuilds/assets/`, but not `prebuilds/<platform>/opencc.node`.
The main package tarball should contain the JavaScript API and CLI under
`node/`, the TypeScript declarations, `prebuilds/assets/`, and the Bazel
workspace plus the `src/` C++ sources and `data/` dictionary sources for the
install-time source build, but not `prebuilds/<platform>/opencc.node`.

Dry-run each scoped package:

Expand Down Expand Up @@ -167,15 +171,19 @@ node -e "const OpenCC = require('opencc'); const cc = new OpenCC('s2twp'); conso

The install should pull exactly one matching `@opencc/opencc-<platform>-<arch>`
optional package for the current machine. If no scoped binary package is
available, the install script falls back to `node-gyp-build`.
available, the install script builds from source with Bazel: it compiles the
addon, stages it under `prebuilds/<platform>-<arch>/`, and regenerates the
dictionaries into `prebuilds/assets`.

## Future CI shape
## CI shape

The intended CI release flow is:
The CI release flow is:

1. Build each scoped binary package on its matching platform or runner.
2. Upload scoped package artifacts.
3. Publish all scoped binary packages with the chosen version and dist-tag.
4. Publish the main `opencc` package after the scoped packages exist.
1. `release-npm-binaries` (workflow_dispatch) builds each scoped binary
package on its matching platform or runner and publishes them.
2. Pushing a `ver.*` tag creates the draft GitHub release.
3. Publishing the release triggers `release-npm`, which packs and publishes
the main `opencc` package after verifying the scoped packages exist.

Until that CI flow exists, the local flow above is the source of truth.
The local flow above remains available for building and publishing a single
platform by hand.
Loading
Loading