diff --git a/.dockerignore b/.dockerignore deleted file mode 100644 index 1383c917..00000000 --- a/.dockerignore +++ /dev/null @@ -1,3 +0,0 @@ -**/node_modules -**/build -**/.idea \ No newline at end of file diff --git a/cli/README.md b/cli/README.md new file mode 100644 index 00000000..191c36ea --- /dev/null +++ b/cli/README.md @@ -0,0 +1,68 @@ +# @bscript/cli + +The BlueScript CLI (`bscript`) is the primary tool for managing projects, setting up board environments, and running code on devices. + +For end-user documentation, see the [CLI Reference](https://csg-tokyo.github.io/bluescript/docs/reference/cli) on the project website. + +## Development + +From the repository root: + +```bash +npm install +``` + +Build and test the CLI package: + +```bash +cd cli +npm run build +npm test # unit tests only +npm run test:integration # host integration tests (macOS + cc) +npm run test:all # unit + integration +``` + +### Test layout + +| Script | Jest project | Location | Notes | +| :--- | :--- | :--- | :--- | +| `npm test` | `unit` | `tests/**/*.test.ts` (excludes `integration/`) | Mocks fs, shell, logger, devices | +| `npm run test:integration` | `integration` | `tests/integration/**/*.test.ts` | Real host `shell` process; macOS only | +| `npm run test:all` | both | — | Run before merging CLI changes | + +**Integration test requirements:** macOS, `cc`, and the `microcontroller/` tree at the repository root. On first run, tests build `microcontroller/ports/host/build/shell` and `c-runtime.so` if missing. Tests are skipped automatically on non-macOS platforms. + +**Integration coverage (14 tests):** + +- `tests/integration/project/run.host.test.ts` — `project run` on host: normal output, built-in library, functions/variables, local import, local package import, inline C, `.c` / `.h` includes, compile error +- `tests/integration/project/repl.host.test.ts` — `repl -b host`: entry line, built-in calls, variable/function persistence, compile-error recovery + +CLI step logs are suppressed during integration runs (`tests/integration-setup.ts`). Program output from BlueScript code is still asserted via captured stdout. + +Integration tests do **not** replace manual checks for ESP32 hardware, serial/BLE, Notebook UI, Git-based `project install`, or interactive TTY behavior (Ctrl-D, prompts). See **[docs/manual-test.md](./docs/manual-test.md)** for the manual QA checklist and coverage map. + +Run the CLI from source without a global install: + +```bash +npm start -- [options] +# Example: +npm start -- board list +``` + +Install locally for manual testing: + +```bash +npm run build +npm link +bscript -v +``` + +## Manual testing + +Before merging CLI changes or cutting a release: + +1. Run `npm run test:all` (or at least `npm test`; on macOS also `npm run test:integration`). +2. Follow the manual QA checklist: **[docs/manual-test.md](./docs/manual-test.md)** + +- **Daily PRs:** run automated tests plus **Quick smoke (host)** (~15 minutes). +- **Releases / ESP32 changes:** run the full checklist and ESP32 scenarios. diff --git a/cli/docs/manual-test.md b/cli/docs/manual-test.md new file mode 100644 index 00000000..27f5e286 --- /dev/null +++ b/cli/docs/manual-test.md @@ -0,0 +1,702 @@ +# CLI Manual Test + +Manual QA checklist for the BlueScript CLI (`bscript`). + +For command syntax and option descriptions, see the [CLI Reference](../../website/docs/reference/cli.md). This document covers **how to verify** behavior, **expected results**, and **test environments**. + +## When to run + +- Before merging PRs that change `cli/src/**` +- Before publishing a new `@bscript/cli` release +- After changes to `@bscript/lang`, `@bscript/notebook`, or the runtime bundle consumed by the CLI + +Run automated tests first (see [Automated tests](#automated-tests)); use this checklist for behavior that integration tests do not cover (ESP32 hardware, interactive prompts, Notebook UI, and so on). + +## Prerequisites + +### Test target + +Install the CLI version under test: + +```bash +# From npm (release candidate) +npm install -g @bscript/cli@ + +# From local checkout +cd cli && npm run build && npm link +``` + +Verify: + +```bash +bscript -v +``` + +Use a clean working directory for project commands (no existing `bsconfig.json` in the current path unless the test requires it). + +### Environment matrix + +| Profile | OS | Node.js | Additional requirements | +| :--- | :--- | :--- | :--- | +| **host** | macOS | v18+ (v20+ recommended) | `cc`, `make` | +| **esp32** | macOS | v18+ (v20+ recommended) | ESP32 board, USB cable, Bluetooth enabled | + +> **Note:** The host runtime currently requires **macOS**. ESP32 setup is also macOS-only in the current CLI implementation. + +### Automated integration tests (host) + +Host integration tests live in `cli/tests/integration/`. They spawn the real host `shell` binary and exercise `project run` and `repl` end-to-end on disk (no ESP32, no global `bscript` install). + +| Requirement | Detail | +| :--- | :--- | +| OS | macOS only (tests are skipped on other platforms) | +| Toolchain | `cc` (builds `microcontroller/ports/host/build/` on first run if missing) | +| Repo layout | Run from `cli/` with the `microcontroller/` tree at the repository root | + +```bash +cd cli +npm run test:integration # integration only (14 tests) +npm run test:all # unit + integration +``` + +CLI log output is suppressed during integration runs (`tests/integration-setup.ts` mocks `logger` and `runStep`). Program output from BlueScript code is still verified via captured stdout. + +--- + +## Quick smoke (host only, ~15 min) + +Run this before merging most CLI PRs. No hardware required. + +1. **MT-SMOKE-01** — `bscript -v` prints the expected version +2. **MT-SMOKE-02** — `bscript board list` shows `esp32` and `host` +3. **MT-SMOKE-03** — `bscript board setup host` completes successfully (skip if already set up) +4. **MT-SMOKE-04** — `bscript project create smoke-host -b host` creates the project +5. **MT-SMOKE-05** — `cd smoke-host && bscript project check` succeeds +6. **MT-SMOKE-06** — `bscript project run` prints `Hello world!` and exits with Ctrl-D +7. **MT-SMOKE-07** — `bscript project run --with-repl` starts REPL after entry execution; one line runs; Ctrl-D exits + +Cleanup (optional): remove `smoke-host/` and run `bscript board remove host -f` if you need a clean state. + +--- + +## Full checklist + +Each item has an ID for bug reports and test records. Record pass/fail in the [test record template](#test-record-template) below. + +**Priority:** P0 = must pass for release · P1 = important · P2 = edge cases / nice to have + +**Requires:** `host` · `esp32` · `both` + +--- + +### Global + +#### MT-GLOBAL-01: Version + +- **Priority:** P0 · **Requires:** both +- **Steps:** Run `bscript -v` and `bscript --version` +- **Expected:** Both print the same version string matching the installed package + +#### MT-GLOBAL-02: Top-level help + +- **Priority:** P1 · **Requires:** both +- **Steps:** Run `bscript -h` and `bscript --help` +- **Expected:** Usage shows `board`, `project`, and `repl` subcommands + +#### MT-GLOBAL-03: Unknown command + +- **Priority:** P2 · **Requires:** both +- **Steps:** Run `bscript unknown` +- **Expected:** Non-zero exit with a clear error message + +#### MT-GLOBAL-04: Subcommand help + +- **Priority:** P1 · **Requires:** both +- **Steps:** Run `bscript board -h` and `bscript project -h` +- **Expected:** Lists subcommands for each group + +--- + +### `bscript board list` + +#### MT-BOARD-LIST-01: Initial state + +- **Priority:** P1 · **Requires:** both +- **Precondition:** Fresh install or after `bscript board fullclean -f` +- **Steps:** Run `bscript board list` +- **Expected:** `esp32` and `host` listed as `not set up`; hint to run `bscript board setup` shown + +#### MT-BOARD-LIST-02: After setup + +- **Priority:** P1 · **Requires:** both +- **Precondition:** At least one board set up +- **Steps:** Run `bscript board list` +- **Expected:** Set-up boards show `set up` (green) + +--- + +### `bscript board setup` + +#### MT-BOARD-SETUP-01: Host setup + +- **Priority:** P0 · **Requires:** host +- **Steps:** + 1. Run `bscript board setup host` + 2. Confirm the setup plan prompt +- **Expected:** Success message; next-step hint mentions `bscript project create -b host` + +#### MT-BOARD-SETUP-02: Host already set up + +- **Priority:** P1 · **Requires:** host +- **Precondition:** Host already set up +- **Steps:** Run `bscript board setup host` again +- **Expected:** Warning that setup is already complete; no re-download + +#### MT-BOARD-SETUP-03: Setup cancelled + +- **Priority:** P2 · **Requires:** host +- **Steps:** Run `bscript board setup host` and answer **No** at the confirmation prompt +- **Expected:** `Setup cancelled by user.`; no partial state corruption + +#### MT-BOARD-SETUP-04: ESP32 setup + +- **Priority:** P0 · **Requires:** esp32 +- **Steps:** Run `bscript board setup esp32` and confirm +- **Expected:** Success; next-step hint mentions `bscript board flash-runtime esp32` + +#### MT-BOARD-SETUP-05: Unknown board + +- **Priority:** P2 · **Requires:** both +- **Steps:** Run `bscript board setup unknown` +- **Expected:** `Unsupported board name` error; non-zero exit + +--- + +### `bscript board flash-runtime` + +#### MT-BOARD-FLASH-01: Not supported on host + +- **Priority:** P1 · **Requires:** host +- **Steps:** Run `bscript board flash-runtime host` +- **Expected:** Error: `flash-runtime is not supported for the host board` + +#### MT-BOARD-FLASH-02: ESP32 before setup + +- **Priority:** P1 · **Requires:** esp32 +- **Precondition:** ESP32 not set up +- **Steps:** Run `bscript board flash-runtime esp32` +- **Expected:** Warning to run `bscript board setup esp32` first + +#### MT-BOARD-FLASH-03: Interactive port selection + +- **Priority:** P0 · **Requires:** esp32 +- **Precondition:** ESP32 set up; device connected via USB +- **Steps:** Run `bscript board flash-runtime esp32` (no `--port`) +- **Expected:** Serial port list appears; flash succeeds; success message with `bscript project run` hint + +#### MT-BOARD-FLASH-04: Explicit port + +- **Priority:** P1 · **Requires:** esp32 +- **Steps:** Run `bscript board flash-runtime esp32 --port ` +- **Expected:** Port selection skipped; flash succeeds + +#### MT-BOARD-FLASH-05: No serial ports + +- **Priority:** P2 · **Requires:** esp32 +- **Precondition:** No device connected +- **Steps:** Run `bscript board flash-runtime esp32` +- **Expected:** `No serial ports found` error + +--- + +### `bscript board remove` + +#### MT-BOARD-REMOVE-01: Nothing to remove + +- **Priority:** P2 · **Requires:** both +- **Precondition:** Board not set up +- **Steps:** Run `bscript board remove host` +- **Expected:** Warning: not set up; nothing removed + +#### MT-BOARD-REMOVE-02: Confirm and cancel + +- **Priority:** P1 · **Requires:** host +- **Steps:** Run `bscript board remove host`; answer **No** +- **Expected:** `Removal process cancelled by user.` + +#### MT-BOARD-REMOVE-03: Confirm removal + +- **Priority:** P1 · **Requires:** host +- **Steps:** Run `bscript board remove host`; answer **Yes** +- **Expected:** Success; `bscript board list` shows `host` as `not set up` + +#### MT-BOARD-REMOVE-04: Force flag + +- **Priority:** P1 · **Requires:** host +- **Steps:** Run `bscript board remove host -f` +- **Expected:** No confirmation prompt; board removed + +--- + +### `bscript board fullclean` + +#### MT-BOARD-FULLCLEAN-01: Cancel + +- **Priority:** P2 · **Requires:** both +- **Steps:** Run `bscript board fullclean`; answer **No** +- **Expected:** `Fullclean process cancelled by user.` + +#### MT-BOARD-FULLCLEAN-02: Remove all settings + +- **Priority:** P1 · **Requires:** both +- **Steps:** Run `bscript board fullclean`; answer **Yes** +- **Expected:** Success; all boards show `not set up` in `bscript board list` + +#### MT-BOARD-FULLCLEAN-03: Force flag + +- **Priority:** P1 · **Requires:** both +- **Steps:** Run `bscript board fullclean -f` +- **Expected:** No confirmation prompt; all settings removed + +--- + +### `bscript board update` + +#### MT-BOARD-UPDATE-01: Up to date + +- **Priority:** P1 · **Requires:** both +- **Precondition:** Latest runtime and environments installed +- **Steps:** Run `bscript board update` +- **Expected:** Steps report `not needed` / skip where appropriate; no errors + +#### MT-BOARD-UPDATE-02: After update, run still works + +- **Priority:** P0 · **Requires:** host +- **Steps:** Run `bscript board update`, then `bscript project run` in a host project +- **Expected:** Project runs normally + +--- + +### `bscript project create` + +#### MT-PROJ-CREATE-01: Host project + +- **Priority:** P0 · **Requires:** host +- **Steps:** Run `bscript project create test-host -b host` +- **Expected:** Directory created with: + - `bsconfig.json` (`boardName: "host"`) + - `src/index.bs` (Hello world sample) + - `.gitignore` + +#### MT-PROJ-CREATE-02: Interactive board selection + +- **Priority:** P1 · **Requires:** both +- **Steps:** Run `bscript project create test-interactive` (no `--board`) +- **Expected:** Prompt to choose `esp32` or `host`; project created for selected board + +#### MT-PROJ-CREATE-03: ESP32 project + +- **Priority:** P0 · **Requires:** esp32 +- **Steps:** Run `bscript project create test-esp32 -b esp32` +- **Expected:** `bsconfig.json` has `boardName: "esp32"` + +#### MT-PROJ-CREATE-04: Directory already exists + +- **Priority:** P2 · **Requires:** both +- **Steps:** Run `bscript project create test-host -b host` twice +- **Expected:** Second run fails with `already exists` + +#### MT-PROJ-CREATE-05: Board not set up + +- **Priority:** P1 · **Requires:** both +- **Precondition:** Target board not set up +- **Steps:** Run `bscript project create test -b host` +- **Expected:** Error: environment not set up + +#### MT-PROJ-CREATE-06: Invalid board + +- **Priority:** P2 · **Requires:** both +- **Steps:** Run `bscript project create test -b invalid` +- **Expected:** `Unsupported board name` error + +--- + +### `bscript project check` + +#### MT-PROJ-CHECK-01: Valid project + +- **Priority:** P0 · **Requires:** both +- **Steps:** In a valid project, run `bscript project check` +- **Expected:** `Successfully checked BlueScript program.`; no device connection required + +#### MT-PROJ-CHECK-02: Syntax error + +- **Priority:** P1 · **Requires:** host +- **Steps:** Introduce a syntax error in `src/index.bs`; run `bscript project check` +- **Expected:** Compile error displayed; non-zero exit + +#### MT-PROJ-CHECK-03: Inline C + +- **Priority:** P1 · **Requires:** both +- **Steps:** Add Inline C (`code` tagged template) to the project; run `bscript project check` +- **Expected:** Check succeeds on host; on ESP32, ESP-IDF-specific C also builds if used + +#### MT-PROJ-CHECK-04: Outside project directory + +- **Priority:** P2 · **Requires:** both +- **Steps:** Run `bscript project check` where no `bsconfig.json` exists +- **Expected:** Clear error; non-zero exit + +--- + +### `bscript project run` + +#### MT-PROJ-RUN-01: Host normal run + +- **Priority:** P0 · **Requires:** host +- **Steps:** + 1. `cd` into a host project + 2. Run `bscript project run` +- **Expected:** Steps Connecting → Initializing → Compiling → Loading → execution; `console.log` output visible; Ctrl-D exits cleanly + +#### MT-PROJ-RUN-02: Host compile error + +- **Priority:** P1 · **Requires:** host +- **Steps:** Run with invalid source +- **Expected:** `Failed to run BlueScript program.`; non-zero exit + +#### MT-PROJ-RUN-03: ESP32 run + +- **Priority:** P0 · **Requires:** esp32 +- **Precondition:** Runtime flashed; device powered and in range +- **Steps:** Run `bscript project run` in an ESP32 project +- **Expected:** Bluetooth scan/connect; compile; transfer; execution succeeds + +#### MT-PROJ-RUN-04: ESP32 disconnect + +- **Priority:** P2 · **Requires:** esp32 +- **Steps:** Disconnect Bluetooth or power off device during run +- **Expected:** `Disconnected.` message; non-zero exit + +#### MT-PROJ-RUN-05: With REPL + +- **Priority:** P0 · **Requires:** both +- **Steps:** Repeat in a **host** project and an **ESP32** project. + 1. Run `bscript project run --with-repl` + 2. Enter one valid line at the `>` prompt (e.g. `console.log("repl");`) +- **Expected:** Entry file runs first; then `>` REPL prompt; REPL line compiles and runs; compile errors shown without exiting REPL; Ctrl-D exits. On ESP32, Bluetooth connection succeeds before REPL starts. + +#### MT-PROJ-RUN-06: With Notebook + +- **Priority:** P1 · **Requires:** both +- **Steps:** Repeat in a **host** project and an **ESP32** project. + 1. Run `bscript project run --with-notebook` + 2. Run one cell in the browser UI +- **Expected:** Entry runs; browser opens `http://localhost:3000`; WebSocket at `ws://localhost:8080`; cell execution works; Ctrl-D in terminal exits. On ESP32, device connection succeeds before Notebook starts. + +#### MT-PROJ-RUN-07: Conflicting options + +- **Priority:** P2 · **Requires:** both +- **Steps:** Run `bscript project run --with-repl --with-notebook` +- **Expected:** Commander reports option conflict; command does not run + +#### MT-PROJ-RUN-08: Built-in library + +- **Priority:** P0 · **Requires:** both +- **Steps:** Repeat in a **host** project and an **ESP32** project. Set `src/index.bs` to use built-in APIs (no `import`): + ```typescript + console.log("built-in"); + print("via print"); + console.log(time.now()); + ``` + Run `bscript project run`. + On **ESP32**, optionally add `time.delay(500);` before the last line and confirm it also works (ESP32-only API). +- **Expected:** All lines produce output; no compile or runtime error. See [Built-in Library](../../website/docs/reference/libraries/builtin.md). + +#### MT-PROJ-RUN-09: User-defined functions and variables + +- **Priority:** P0 · **Requires:** both +- **Steps:** Repeat in a **host** project and an **ESP32** project. Set `src/index.bs` to: + ```typescript + const message = "hello"; + function greet(): void { + console.log(message); + } + greet(); + ``` + Run `bscript project run`. +- **Expected:** `hello` printed on both boards; function and variable bindings work at runtime. + +#### MT-PROJ-RUN-10: Local module import + +- **Priority:** P0 · **Requires:** both +- **Steps:** Repeat in a **host** project and an **ESP32** project. + 1. Create `src/math-utils.bs`: + ```typescript + export function add(a: integer, b: integer): integer { + return a + b; + } + ``` + 2. Set `src/index.bs` to: + ```typescript + import { add } from "./math-utils"; + console.log(add(10, 20)); + ``` + 3. Run `bscript project run`. +- **Expected:** `30` printed on both boards; relative import resolves under `srcDir`. + +#### MT-PROJ-RUN-11: Installed package + +- **Priority:** P0 · **Requires:** esp32 +- **Precondition:** GPIO package installed (`bscript project install https://github.com/bluescript-lang/pkg-gpio-esp32.git`) +- **Steps:** Set `src/index.bs` to: + ```typescript + import { GPIO, PinMode } from "gpio"; + const led = new GPIO(2, PinMode.InputOutput); + console.log("GPIO ready"); + ``` + Run `bscript project run`. +- **Expected:** Compiles and runs without import/resolve errors; `GPIO ready` printed. See [Standard Libraries](../../website/docs/reference/libraries/standard.md). + +#### MT-PROJ-RUN-12: Built-in library in project REPL + +- **Priority:** P1 · **Requires:** both +- **Steps:** Repeat in a **host** project and an **ESP32** project. + 1. Set `src/index.bs` to `console.log("entry done");` + 2. Run `bscript project run --with-repl` + 3. At the `>` prompt, enter: `console.log(time.now());` +- **Expected:** Entry runs first; REPL line executes using built-in `console.log` and `time.now` without error on both boards. + +#### MT-PROJ-RUN-13: Entry variables and functions in project REPL + +- **Priority:** P1 · **Requires:** both +- **Steps:** Repeat in a **host** project and an **ESP32** project. + 1. Set `src/index.bs` to: + ```typescript + const msg = "from entry"; + function show(): void { + console.log(msg); + } + console.log("entry done"); + ``` + 2. Run `bscript project run --with-repl` + 3. At the `>` prompt, enter: `show();` + 4. Enter: `console.log(msg);` +- **Expected:** Both REPL lines succeed on both boards; `from entry` printed twice (once per line); entry-defined function and variable remain available in REPL. + +#### MT-PROJ-RUN-14: Installed package in project REPL + +- **Priority:** P1 · **Requires:** esp32 +- **Precondition:** GPIO package installed; entry file sets up a `led` instance (see [REPL & Notebook tutorial](../../website/docs/tutorial/guides/repl.md)) +- **Steps:** + 1. Run `bscript project run --with-repl` + 2. At the `>` prompt, enter a line that uses `led` (e.g. `console.log("LED ready in REPL");`) +- **Expected:** Entry-imported package symbols (e.g. `led`) are usable in REPL without re-importing. + +--- + +### `bscript project install` + +#### MT-PROJ-INSTALL-01: Install all (no dependencies) + +- **Priority:** P1 · **Requires:** both +- **Steps:** In a project with empty `dependencies`, run `bscript project install` +- **Expected:** Completes without error + +#### MT-PROJ-INSTALL-02: Add package by URL + +- **Priority:** P0 · **Requires:** esp32 +- **Steps:** Run `bscript project install ` for a valid BlueScript package +- **Expected:** Package under `packages/`; `bsconfig.json` updated + +#### MT-PROJ-INSTALL-03: Install with tag + +- **Priority:** P1 · **Requires:** esp32 +- **Steps:** Run `bscript project install --tag ` +- **Expected:** Specified tag/branch checked out + +#### MT-PROJ-INSTALL-04: Restore from bsconfig + +- **Priority:** P1 · **Requires:** esp32 +- **Precondition:** Project with dependencies in `bsconfig.json` +- **Steps:** Delete `packages/`; run `bscript project install` +- **Expected:** All dependencies restored + +#### MT-PROJ-INSTALL-05: Invalid URL + +- **Priority:** P2 · **Requires:** both +- **Steps:** Run `bscript project install https://invalid.example/repo.git` +- **Expected:** Download failure; non-zero exit + +#### MT-PROJ-INSTALL-06: Run with installed package + +- **Priority:** P0 · **Requires:** esp32 +- **Steps:** After install, `import` the package in source; run `bscript project run` +- **Expected:** Compiles and runs with package symbols available + +--- + +### `bscript project uninstall` + +#### MT-PROJ-UNINSTALL-01: Remove package + +- **Priority:** P1 · **Requires:** esp32 +- **Precondition:** Package installed +- **Steps:** Run `bscript project uninstall ` +- **Expected:** `packages//` removed; entry removed from `bsconfig.json` + +#### MT-PROJ-UNINSTALL-02: Unknown package + +- **Priority:** P2 · **Requires:** both +- **Steps:** Run `bscript project uninstall nonexistent` +- **Expected:** Error: not listed in dependencies + +--- + +### `bscript repl` + +#### MT-REPL-01: Missing board option + +- **Priority:** P2 · **Requires:** both +- **Steps:** Run `bscript repl` (no `-b`) +- **Expected:** Required option error + +#### MT-REPL-02: Host global REPL + +- **Priority:** P1 · **Requires:** host +- **Steps:** Run `bscript repl -b host` +- **Expected:** Connecting → REPL prompt; first line treated as entry; subsequent lines as fragments; Ctrl-D exits cleanly + +#### MT-REPL-03: ESP32 global REPL + +- **Priority:** P1 · **Requires:** esp32 +- **Steps:** Run `bscript repl -b esp32` with device available +- **Expected:** Bluetooth connection; REPL works + +#### MT-REPL-04: Compile error in REPL + +- **Priority:** P2 · **Requires:** both +- **Steps:** Run `bscript repl -b host` and `bscript repl -b esp32`. Enter invalid syntax at the REPL prompt on each board. +- **Expected:** Compile error shown; REPL continues on both boards. + +#### MT-REPL-05: No hardware libraries + +- **Priority:** P1 · **Requires:** esp32 +- **Steps:** Try importing a project-only package (e.g. GPIO) in global REPL +- **Expected:** Not available (project REPL / Notebook required for installed packages) + +#### MT-REPL-06: Built-in library + +- **Priority:** P0 · **Requires:** both +- **Steps:** Run `bscript repl -b host` and `bscript repl -b esp32`. At the `>` prompt on each board, enter lines that use built-in APIs (no `import`): + 1. `console.log("built-in");` + 2. `print("via print");` + 3. `console.log(time.now());` + On **ESP32**, optionally enter `time.delay(500);` and confirm it works. +- **Expected:** Each line compiles and runs; output appears for all entries on both boards. See [Built-in Library](../../website/docs/reference/libraries/builtin.md). + +#### MT-REPL-07: Variables persist across REPL lines + +- **Priority:** P0 · **Requires:** both +- **Steps:** Run `bscript repl -b host` and `bscript repl -b esp32`. On each board: + 1. First line: `const x = 42; console.log("init");` + 2. Second line: `console.log(x);` +- **Expected:** First line prints `init`; second line prints `42` on both boards; variable defined on an earlier line remains in scope. + +#### MT-REPL-08: User-defined functions persist across REPL lines + +- **Priority:** P1 · **Requires:** both +- **Steps:** Run `bscript repl -b host` and `bscript repl -b esp32`. On each board: + 1. First line: `function double(n: integer): integer { return n * 2; } console.log("fn defined");` + 2. Second line: `console.log(double(21));` +- **Expected:** First line prints `fn defined`; second line prints `42` on both boards; function defined on an earlier line can be called later. + +--- + +## End-to-end scenarios + +### Scenario A: Host (no hardware) + +1. `bscript board fullclean -f` *(optional clean start)* +2. `bscript board setup host` +3. `bscript board list` — host shows `set up` +4. `bscript project create hello-host -b host` +5. `cd hello-host` +6. Edit `src/index.bs` — add a `console.log` with a distinct message +7. `bscript project check` +8. `bscript project run` — verify output (MT-PROJ-RUN-08–10: built-in, functions/variables, local import) +9. `bscript project run --with-repl` — run REPL lines (MT-PROJ-RUN-12–13) +10. `bscript project run --with-notebook` — run a cell in the browser +11. `bscript repl -b host` — verify built-in and REPL state (MT-REPL-06–08) + +### Scenario B: ESP32 (hardware required) + +1. `bscript board setup esp32` +2. `bscript board flash-runtime esp32` *(select port or use `--port`)* +3. `bscript project create hello-esp32 -b esp32` +4. `cd hello-esp32` +5. `bscript project check` +6. `bscript project run` — verify Bluetooth connection and execution (MT-PROJ-RUN-08–10) +7. `bscript project run --with-repl` — verify REPL built-in and entry state (MT-PROJ-RUN-12–13) +8. `bscript repl -b esp32` — verify built-in and REPL state (MT-REPL-06–08) +9. `bscript project install ` *(e.g. GPIO library)* +10. Update source to use the package; `bscript project run` (MT-PROJ-RUN-11) +11. `bscript project run --with-notebook` — use package symbols in a cell (MT-PROJ-RUN-14) +12. `bscript project uninstall ` + +### Scenario C: Board lifecycle + +1. `bscript board setup host` +2. `bscript board remove host` — confirm **Yes** +3. `bscript board setup host` — re-setup succeeds +4. `bscript board fullclean -f` +5. `bscript board list` — all boards `not set up` + +--- + +## Test record template + +Copy and fill in one row per test session: + +| Date | Tester | CLI version | OS | Scope | Result | Notes | +| :--- | :--- | :--- | :--- | :--- | :--- | :--- | +| YYYY-MM-DD | | | macOS … | Quick smoke (host) | PASS / FAIL | | + +For failures, include the item ID (e.g. `MT-PROJ-RUN-03`) in Notes or link to an issue. + +--- + +## Coverage map: automated vs manual + +Jest **unit** tests in `cli/tests/` mock filesystem, network, and device I/O. **Integration** tests in `cli/tests/integration/` use real host runtime processes on macOS. Use this table to avoid re-testing automated behavior manually while ensuring gaps are covered. + +| Area | Unit tests | Integration tests (host, macOS) | Manual testing still needed | +| :--- | :--- | :--- | :--- | +| `board setup` | Handler logic, macOS paths, skip-if-done | — | Real download, ESP-IDF install, host runtime build | +| `board flash-runtime` | ESP32 handler, host rejection, port prompt mocked | — | Actual USB flash on hardware | +| `board remove` / `fullclean` | File removal, prompts mocked | — | Confirm disk state after real removal | +| `board update` | Update steps, rollback logic | — | End-to-end after real version bump | +| `board list` | — | — | Visual output, setup status labels | +| `project create` | File generation, validation | — | Interactive board picker | +| `project install` | Git clone mocked | — | Real Git URLs, tag checkout, board mismatch | +| `project uninstall` | — | — | Full uninstall flow | +| `project check` | — | — | Real compiler, Inline C | +| `project run` | Handler wiring | Normal run; built-in; functions/variables; local import; local package import; inline C; `.c` / `.h` includes; compile error (`run.host.test.ts`) | Ctrl-D / TTY; `--with-repl`; `--with-notebook`; ESP32; BLE; real `project install` packages | +| `repl` | — | Entry line; built-in; variable/function persistence; compile-error recovery (`repl.host.test.ts`) | Interactive session; ESP32; device connection; global REPL without mocked readline | +| WebSocket / device protocol | Unit tests | — | Browser Notebook integration | +| Global help / version | — | — | Quick smoke items | + +### Integration test ↔ manual item map (host) + +| Integration test (`run.host.test.ts` / `repl.host.test.ts`) | Related manual IDs | +| :--- | :--- | +| Runs a program and prints output | MT-PROJ-RUN-01 (host) | +| Built-in library | MT-PROJ-RUN-08 (host), MT-REPL-06 (host) | +| User-defined functions and variables | MT-PROJ-RUN-09 (host) | +| Local module import | MT-PROJ-RUN-10 (host) | +| Package import (local `packages/` fixture) | MT-PROJ-RUN-11 (partial — not Git install) | +| Inline C / `.c` / `.h` includes | MT-PROJ-CHECK-03 (host) | +| Compile error on run | MT-PROJ-RUN-02 (host) | +| REPL entry line | MT-REPL-02 (host) | +| REPL variables / functions across lines | MT-REPL-07, MT-REPL-08 (host) | +| REPL continues after compile error | MT-REPL-04 (host) | diff --git a/cli/jest.config.js b/cli/jest.config.js index c325c61e..c55fed4e 100644 --- a/cli/jest.config.js +++ b/cli/jest.config.js @@ -1,6 +1,20 @@ export default { - preset: 'ts-jest', - testEnvironment: 'node', - roots: ["/tests", "/src"], - setupFilesAfterEnv: ['/tests/global-mocks.ts'], + projects: [ + { + displayName: 'unit', + preset: 'ts-jest', + testEnvironment: 'node', + roots: ['/tests', '/src'], + testPathIgnorePatterns: ['/integration/'], + setupFilesAfterEnv: ['/tests/global-mocks.ts'], + }, + { + displayName: 'integration', + preset: 'ts-jest', + testEnvironment: 'node', + testMatch: ['/tests/integration/**/*.test.ts'], + setupFilesAfterEnv: ['/tests/integration-setup.ts'], + maxWorkers: 1, + }, + ], }; \ No newline at end of file diff --git a/cli/package.json b/cli/package.json index 8e75a7d4..82259d36 100644 --- a/cli/package.json +++ b/cli/package.json @@ -11,7 +11,15 @@ "publishConfig": { "access": "public" }, - "keywords": ["BlueScript", "TypeScript", "Microcontroller", "MCU", "IoT", "Robotics", "ESP32"], + "keywords": [ + "BlueScript", + "TypeScript", + "Microcontroller", + "MCU", + "IoT", + "Robotics", + "ESP32" + ], "author": "Fumika Mochizuki ", "description": "A CLI for BlueScript", "repository": { @@ -23,10 +31,16 @@ "scripts": { "build": "tsc --build", "start": "ts-node src/index.ts", - "test": "jest", + "test": "jest --selectProjects unit", + "test:integration": "jest --selectProjects integration", + "test:all": "jest", - "prepack": "shx cp ../README.md README.md", - "postpack": "shx rm README.md", + "prepack:backup": "shx mv README.md README.md.bak || shx echo 'No existing README to backup'", + "prepack:copy": "shx cp ../README.md README.md", + "prepack": "npm run prepack:backup && npm run prepack:copy", + "postpack:cleanup": "shx rm README.md", + "postpack:restore": "shx mv README.md.bak README.md || shx echo 'No backup to restore'", + "postpack": "npm run postpack:cleanup && npm run postpack:restore", "prepublishOnly": "npm run build" }, "bin": { diff --git a/cli/src/boards/compiler-adapters.ts b/cli/src/boards/compiler-adapters.ts deleted file mode 100644 index 724863bf..00000000 --- a/cli/src/boards/compiler-adapters.ts +++ /dev/null @@ -1,114 +0,0 @@ -import { GlobalConfigHandler, Esp32BoardConfig } from "../config/global-config"; -import { ProjectConfigHandler, PROJECT_DEFAULT_PATHS } from "../config/project-config"; -import { BoardName } from "../config/board-utils"; -import { - CompilerSession, Project, ExecutableBinary, MemoryLayout, - PackageForEsp32, Esp32Toolchain, Esp32ToolchainConfig -} from "@bscript/lang"; -import * as path from 'path'; - - -export interface CompilerAdapter { - readonly boardName: BoardName; - getDummyMemoryLayout(): MemoryLayout; - buildProject(memoryLayout: MemoryLayout): Promise; - compileFragment(src: string): Promise; -} - -export class ESP32CompilerAdapter implements CompilerAdapter { - readonly boardName: BoardName = 'esp32'; - private globalConfigHandler: GlobalConfigHandler; - private projectConfigHandler: ProjectConfigHandler; - private boardConfig: Esp32BoardConfig; - private compiler?: CompilerSession; - - readonly dummyMemoryLayout: MemoryLayout = { - iram: { address: 0x40096c34, size: 1000000 }, - dram: { address: 0x3ffd5b1c, size: 1000000 }, - iflash: { address: 0x40150000, size: 1000000 }, - dflash: { address: 0x3f43a000, size: 1000000 }, - }; - - constructor(globalConfigHandler: GlobalConfigHandler, projectConfigHandler: ProjectConfigHandler) { - this.globalConfigHandler = globalConfigHandler; - const boardConfig = this.globalConfigHandler.getBoardConfig(this.boardName); - if (boardConfig === undefined) { - throw new Error(`The environment for ${this.boardName} is not set up.`); - } - this.boardConfig = boardConfig; - this.projectConfigHandler = projectConfigHandler; - } - - getDummyMemoryLayout(): MemoryLayout { - return this.dummyMemoryLayout; - } - - async buildProject(memoryLayout: MemoryLayout): Promise { - const project = Project.load( - this.projectConfigHandler.getConfig().projectName, - this.packageReader.bind(this), - ); - const toolchain = new Esp32Toolchain(this.getCompilerConfig(), memoryLayout); - this.compiler = new CompilerSession(toolchain); - return this.compiler.buildProject(project); - } - - async compileFragment(src: string) { - if (!this.compiler) { - throw new Error("Cannot compile fragment before building the project."); - } - return this.compiler.compileFragment(src); - } - - private getCompilerConfig(): Esp32ToolchainConfig { - const runtimeDir = this.projectConfigHandler?.getConfig().runtimeDir - ?? this.globalConfigHandler.getConfig().runtimeDir; - if (!runtimeDir) { - throw new Error('An unexpected error occurred: cannot find runtime directory path.'); - } - return { - runtimeDir, - compilerToolchainDir: this.boardConfig.xtensaGccDir, - espDir: this.boardConfig.rootDir - } - } - - private packageReader(name: string): PackageForEsp32 { - const mainRoot = this.projectConfigHandler.root; - const subPackageRoot = path.join(mainRoot, PROJECT_DEFAULT_PATHS.PACKAGES_DIR, name); - const isMain = name === this.projectConfigHandler.getConfig().projectName; - const root = isMain ? mainRoot : subPackageRoot; - try { - const projectConfigHandler = isMain - ? this.projectConfigHandler.asBoard(this.boardName) - : ProjectConfigHandler.load(root).asBoard(this.boardName); - return new PackageForEsp32( - name, - { - rootDir: root, - entry: projectConfigHandler.entryFile ?? PROJECT_DEFAULT_PATHS.ENTRY_FILE, - sourceDir: projectConfigHandler.srcDir ?? PROJECT_DEFAULT_PATHS.SRC_DIR, - distDir: PROJECT_DEFAULT_PATHS.DIST_DIR, - buildDir: PROJECT_DEFAULT_PATHS.BUILD_DIR, - packageDir: PROJECT_DEFAULT_PATHS.PACKAGES_DIR, - }, - Object.keys(projectConfigHandler.dependencies), - projectConfigHandler.espIdfComponents, - ) - } catch (error) { - throw new Error(`Failed to read ${name}.`, { cause: error }); - } - } -} - -export function getCompilerAdapter( - boardName: BoardName, - globalConfigHandler: GlobalConfigHandler, - projectConfigHandler: ProjectConfigHandler -): CompilerAdapter { - if (boardName === 'esp32') { - return new ESP32CompilerAdapter(globalConfigHandler, projectConfigHandler); - } else { - throw new Error(`Unsupported board name: ${boardName}`); - } -} diff --git a/cli/src/commands/board/flash-runtime.ts b/cli/src/commands/board/flash-runtime.ts index 14dcd789..3eb13eb0 100644 --- a/cli/src/commands/board/flash-runtime.ts +++ b/cli/src/commands/board/flash-runtime.ts @@ -3,7 +3,7 @@ import inquirer from 'inquirer'; import * as path from 'path'; import { SerialPort } from 'serialport' import { BoardName } from "../../config/board-utils"; -import { logger, LogStep, showErrorMessages } from "../../core/logger"; +import { logger, runStep } from "../../core/logger"; import { exec } from '../../core/shell'; import chalk from "chalk"; import { CommandHandler } from "../command"; @@ -14,6 +14,10 @@ const RUNTIME_ESP_PORT_DIR = (runtimeDir: string) => path.join(runtimeDir, 'port abstract class FlashRuntimeHandler extends CommandHandler { abstract isSetup(): boolean; abstract flashRuntime(port: string, monitor: boolean): Promise; + + async flash(port: string) { + return runStep('Flashing...', () => this.flashRuntime(port, false)); + } } class ESP32FlashRuntimeHandler extends FlashRuntimeHandler { @@ -23,14 +27,13 @@ class ESP32FlashRuntimeHandler extends FlashRuntimeHandler { return this.globalConfigHandler.isBoardSetup(this.boardName); } - @LogStep('Flashing...') async flashRuntime(port: string) { const runtimeDir = this.globalConfigHandler.getConfig().runtimeDir; if (!runtimeDir) { throw new Error('An unexpected error occurred: cannot find runtime directory path.'); } - const boardConfig = this.globalConfigHandler.getBoardConfig(this.boardName); + const boardConfig = this.globalConfigHandler.getBoardConfig('esp32'); if (!boardConfig) { throw new Error('An unexpected error occurred: cannot find board config.'); } @@ -43,11 +46,13 @@ class ESP32FlashRuntimeHandler extends FlashRuntimeHandler { } function getFlashRuntimeHandler(board: string) { + if (board === 'host') { + throw new Error('flash-runtime is not supported for the host board'); + } if (board === 'esp32') { return new ESP32FlashRuntimeHandler(); - } else { - throw new Error(`Unsupported board name: ${board}`); } + throw new Error(`Unsupported board name: ${board}`); } export async function handleFlashRuntimeCommand(board: string, options: { port?: string }) { @@ -88,7 +93,7 @@ export async function handleFlashRuntimeCommand(board: string, options: { port?: logger.info(`Using port: ${selectedPort}`); // Flash runtime. - await flashRuntimeHandler.flashRuntime(selectedPort); + await flashRuntimeHandler.flash(selectedPort); logger.br(); logger.success(`Success to flash the BlueScript runtime to ${board}`); @@ -96,7 +101,7 @@ export async function handleFlashRuntimeCommand(board: string, options: { port?: } catch (error) { logger.error(`Failed to flash the runtime to ${board}`); - showErrorMessages(error); + logger.showError(error); process.exit(1); } } diff --git a/cli/src/commands/board/full-clean.ts b/cli/src/commands/board/full-clean.ts index cb2ab129..83660fa4 100644 --- a/cli/src/commands/board/full-clean.ts +++ b/cli/src/commands/board/full-clean.ts @@ -1,6 +1,6 @@ import { Command } from "commander"; import inquirer from 'inquirer'; -import { logger, showErrorMessages } from "../../core/logger"; +import { logger } from "../../core/logger"; import * as fs from '../../core/fs'; import { CommandHandler } from "../command"; import { GLOBAL_SETTINGS } from "../../config/constants"; @@ -47,7 +47,7 @@ export async function handleFullcleanCommand(options: { force?: boolean }) { } catch (error) { logger.error(`Failed to delete entire settings.`); - showErrorMessages(error); + logger.showError(error); process.exit(1); } } diff --git a/cli/src/commands/board/list.ts b/cli/src/commands/board/list.ts index f7c19529..dc384d68 100644 --- a/cli/src/commands/board/list.ts +++ b/cli/src/commands/board/list.ts @@ -1,7 +1,7 @@ import { Command } from "commander"; import chalk from 'chalk'; import { BOARD_NAMES } from "../../config/board-utils"; -import { logger, showErrorMessages } from "../../core/logger"; +import { logger } from "../../core/logger"; import { CommandHandler } from "../command"; @@ -26,7 +26,7 @@ export async function handleListCommand() { } catch (error) { logger.error(`Failed to list up available board names`); - showErrorMessages(error); + logger.showError(error); process.exit(1); } } diff --git a/cli/src/commands/board/remove.ts b/cli/src/commands/board/remove.ts index caa5f6ca..e9aafb75 100644 --- a/cli/src/commands/board/remove.ts +++ b/cli/src/commands/board/remove.ts @@ -1,20 +1,40 @@ import { Command } from "commander"; import inquirer from 'inquirer'; import { BoardName } from "../../config/board-utils"; -import { logger, LogStep, showErrorMessages } from "../../core/logger"; +import { logger, runStep } from "../../core/logger"; import * as fs from '../../core/fs'; import { CommandHandler } from "../command"; abstract class RemoveHandler extends CommandHandler { async remove() { - await this.removeBoard(); + await runStep('Removing...', () => this.removeBoard()); this.globalConfigHandler.save(); } abstract isSetup(): boolean; abstract removeBoard(): Promise; } +class HostRemoveHandler extends RemoveHandler { + readonly boardName: BoardName = 'host'; + + isSetup(): boolean { + return this.globalConfigHandler.isBoardSetup(this.boardName); + } + + async removeBoard() { + const boardConfig = this.globalConfigHandler.getBoardConfig('host'); + if (boardConfig === undefined) { + throw new Error(`Cannot find config for ${this.boardName}.`); + } + if (fs.exists(boardConfig.buildDir)) { + fs.removeDir(boardConfig.buildDir); + } + + this.globalConfigHandler.removeBoardConfig(this.boardName); + } +} + class ESP32RemoveHandler extends RemoveHandler { readonly boardName: BoardName = 'esp32'; @@ -22,9 +42,8 @@ class ESP32RemoveHandler extends RemoveHandler { return this.globalConfigHandler.isBoardSetup(this.boardName); } - @LogStep(`Removing...`) async removeBoard() { - const boardConfig = this.globalConfigHandler.getBoardConfig(this.boardName); + const boardConfig = this.globalConfigHandler.getBoardConfig('esp32'); if (boardConfig === undefined) { throw new Error(`Cannot find config for ${this.boardName}.`); } @@ -39,9 +58,11 @@ class ESP32RemoveHandler extends RemoveHandler { function getRemoveHandler(board: string) { if (board === 'esp32') { return new ESP32RemoveHandler(); - } else { - throw new Error(`Unsupported board name: ${board}`); } + if (board === 'host') { + return new HostRemoveHandler(); + } + throw new Error(`Unsupported board name: ${board}`); } export async function handleRemoveCommand(board: string, options: { force?: boolean }) { @@ -81,7 +102,7 @@ export async function handleRemoveCommand(board: string, options: { force?: bool } catch (error) { logger.error(`Failed to remove ${board}`); - showErrorMessages(error); + logger.showError(error); process.exit(1); } } diff --git a/cli/src/commands/board/setup.ts b/cli/src/commands/board/setup.ts index 8ad160f0..6b3ff2ef 100644 --- a/cli/src/commands/board/setup.ts +++ b/cli/src/commands/board/setup.ts @@ -2,13 +2,14 @@ import { Command } from "commander"; import * as path from 'path'; import * as os from 'os'; import inquirer from 'inquirer'; -import { logger, LogStep, showErrorMessages, SkipStep } from "../../core/logger"; +import { logger, runStep, skip } from "../../core/logger"; import { BoardName } from "../../config/board-utils"; import { exec } from '../../core/shell'; import * as fs from '../../core/fs'; import chalk from "chalk"; import { CommandHandler } from "../command"; import { GLOBAL_SETTINGS } from "../../config/constants"; +import { buildHostRuntime } from "../../platforms/runtime/host-board-runtime"; abstract class SetupHandler extends CommandHandler { @@ -22,7 +23,7 @@ abstract class SetupHandler extends CommandHandler { async setup(): Promise { this.ensureBlueScriptDir(); - await this.downloadBlueScriptRuntime(); + await this.downloadBlueScriptRuntimeStep(); await this.setupBoard(); this.globalConfigHandler.save(); } @@ -33,19 +34,19 @@ abstract class SetupHandler extends CommandHandler { } } - private needToDownloadBlueScriptRuntime() { - return !this.globalConfigHandler.isRuntimeSetup(); + private async downloadBlueScriptRuntimeStep() { + return runStep('Downloading BlueScript runtime...', async () => { + if (this.globalConfigHandler.isRuntimeSetup()) { + return skip('already downloaded.'); + } + await this.downloadBlueScriptRuntime(); + }); } - @LogStep(`Downloading BlueScript runtime...`) private async downloadBlueScriptRuntime() { - if (!this.needToDownloadBlueScriptRuntime()) { - throw new SkipStep('already downloaded.', undefined); - } if (fs.exists(GLOBAL_SETTINGS.RUNTIME_DIR)) { fs.removeDir(GLOBAL_SETTINGS.RUNTIME_DIR); } - await fs.downloadAndUnzip(GLOBAL_SETTINGS.RUNTIME_ZIP_URL, GLOBAL_SETTINGS.BLUESCRIPT_DIR); this.globalConfigHandler.setRuntimeDir(GLOBAL_SETTINGS.RUNTIME_DIR); } @@ -93,10 +94,10 @@ export class ESP32SetupHandler extends SetupHandler { } async setupBoard(): Promise { - await this.installEspidfRequiredPackages(); - await this.installPython3(); - await this.cloneEspIdf(); - await this.runEspIdfInstallScript(); + await this.installRequiredPackagesStep(); + await this.installPython3Step(); + await this.cloneEspIdfStep(); + await this.runEspIdfInstallScriptStep(); this.globalConfigHandler.updateBoardConfig(this.boardName, { idfVersion: GLOBAL_SETTINGS.ESP_IDF_VERSION, @@ -106,17 +107,44 @@ export class ESP32SetupHandler extends SetupHandler { }); } - @LogStep('Installing required packages...') - private async installEspidfRequiredPackages() { - let packages: string[] = []; - if (!(await this.isPackageInstalled('cmake'))) { packages.push('cmake'); } - if (!(await this.isPackageInstalled('ninja'))) { packages.push('ninja'); } - if (!(await this.isPackageInstalled('dfu-util'))) { packages.push('dfu-util'); } - if (!(await this.isPackageInstalled('ccache'))) { packages.push('ccache'); } - if (packages.length === 0) { - throw new SkipStep('already installed.', undefined); - } + private async installRequiredPackagesStep() { + return runStep('Installing required packages...', async () => { + let packages: string[] = []; + if (!(await this.isPackageInstalled('cmake'))) { packages.push('cmake'); } + if (!(await this.isPackageInstalled('ninja'))) { packages.push('ninja'); } + if (!(await this.isPackageInstalled('dfu-util'))) { packages.push('dfu-util'); } + if (!(await this.isPackageInstalled('ccache'))) { packages.push('ccache'); } + if (packages.length === 0) { + return skip('already installed.'); + } + await this.installEspidfRequiredPackages(packages); + }); + } + private async installPython3Step() { + return runStep('Installing Python3...', async () => { + if ((await this.isPythonVersionGreaterThan3()) || (await this.isPackageInstalled('python3'))) { + return skip('already installed.'); + } + await this.installPython3(); + }); + } + + private cloneEspIdfStep() { + return runStep( + `Cloning ESP-IDF ${GLOBAL_SETTINGS.ESP_IDF_VERSION} from ${GLOBAL_SETTINGS.ESP_IDF_GIT_REPO}... It may take a while.`, + () => this.cloneEspIdf(), + ); + } + + private runEspIdfInstallScriptStep() { + return runStep( + 'Running ESP-IDF install script...', + () => this.runEspIdfInstallScript() + ); + } + + private async installEspidfRequiredPackages(packages: string[]) { let installer: string; if (await this.isPackageInstalled('brew')) { installer = 'brew'; @@ -129,12 +157,7 @@ export class ESP32SetupHandler extends SetupHandler { await exec(`${installer} install ${packages.join(' ')}`); } - @LogStep('Installing Python3...') private async installPython3() { - if ((await this.isPythonVersionGreaterThan3()) || (await this.isPackageInstalled('python3'))) { - throw new SkipStep('already installed.', undefined); - } - if (await this.isPackageInstalled('brew')) { await exec('brew install python3'); } else if (await this.isPackageInstalled('port')) { @@ -162,9 +185,6 @@ export class ESP32SetupHandler extends SetupHandler { } } - @LogStep( - `Cloning ESP-IDF ${GLOBAL_SETTINGS.ESP_IDF_VERSION} from ${GLOBAL_SETTINGS.ESP_IDF_GIT_REPO}... It may take a while.` - ) private async cloneEspIdf() { if (fs.exists(GLOBAL_SETTINGS.ESP_ROOT_DIR)) { fs.removeDir(GLOBAL_SETTINGS.ESP_ROOT_DIR); @@ -174,11 +194,10 @@ export class ESP32SetupHandler extends SetupHandler { } fs.makeDir(GLOBAL_SETTINGS.ESP_ROOT_DIR); - await exec(`git clone --depth 1 -b ${GLOBAL_SETTINGS.ESP_IDF_VERSION} --recursive ${GLOBAL_SETTINGS.ESP_IDF_GIT_REPO}`, + await exec(`git clone --depth 1 -b ${GLOBAL_SETTINGS.ESP_IDF_VERSION} --recursive ${GLOBAL_SETTINGS.ESP_IDF_GIT_REPO}`, { cwd: GLOBAL_SETTINGS.ESP_ROOT_DIR }); } - @LogStep('Running ESP-IDF install script...') private async runEspIdfInstallScript() { await exec(GLOBAL_SETTINGS.ESP_IDF_INSTALL_FILE); } @@ -190,7 +209,68 @@ export class ESP32SetupHandler extends SetupHandler { } catch (error) { throw new Error('Failed to get xtensa gcc path.', {cause: error}); } - + + } +} + +export class HostSetupHandler extends SetupHandler { + readonly boardName: BoardName = 'host'; + + constructor() { + super(); + if (os.platform() !== 'darwin') { + throw new Error('Unsupported OS.'); + } + } + + needSetup(): boolean { + return !this.globalConfigHandler.isBoardSetup(this.boardName); + } + + getBoardSetupPlan(): string[] { + return [ + 'Verify that cc and make are installed (Xcode Command Line Tools).', + 'Build host runtime process.', + ]; + } + + async setupBoard(): Promise { + await this.verifyBuildToolsStep(); + const buildDir = await this.buildHostRuntimeStep(); + this.globalConfigHandler.updateBoardConfig(this.boardName, { buildDir: buildDir! }); + } + + private async verifyBuildToolsStep() { + return runStep('Verifying that cc and make are installed...', async () => { + const missing: string[] = []; + if (!(await this.isCommandInstalled('cc'))) { missing.push('cc'); } + if (!(await this.isCommandInstalled('make'))) { missing.push('make'); } + if (missing.length === 0) { + return; + } + throw new Error( + `Missing required tools: ${missing.join(', ')}. Install Xcode Command Line Tools and try again.`, + ); + }); + } + + private buildHostRuntimeStep() { + return runStep('Building host runtime...', async () => { + const runtimeDir = this.globalConfigHandler.getConfig().runtimeDir; + if (!runtimeDir) { + throw new Error('An unexpected error occurred: cannot find runtime directory path.'); + } + return await buildHostRuntime(runtimeDir); + }); + } + + private async isCommandInstalled(name: string) { + try { + await exec(`which ${name}`, { silent: true }); + return true; + } catch { + return false; + } } } @@ -198,9 +278,11 @@ export class ESP32SetupHandler extends SetupHandler { function getSetupHandler(board: string): SetupHandler { if (board === 'esp32') { return new ESP32SetupHandler(); - } else { - throw new Error(`Unsupported board name: ${board}`); } + if (board === 'host') { + return new HostSetupHandler(); + } + throw new Error(`Unsupported board name: ${board}`); } export async function handleSetupCommand(board: string) { @@ -235,11 +317,15 @@ export async function handleSetupCommand(board: string) { logger.br(); logger.success(`Success to set up ${board}`); - logger.info(`Next step: run ${chalk.yellow(`bscript board flash-runtime ${board}`)}`); + if (board === 'host') { + logger.info(`Next step: run ${chalk.yellow('bscript project create -b host')}`); + } else { + logger.info(`Next step: run ${chalk.yellow(`bscript board flash-runtime ${board}`)}`); + } } catch (error) { logger.error(`Failed to set up ${board}`); - showErrorMessages(error); + logger.showError(error); process.exit(1); } } @@ -249,9 +335,8 @@ export function registerSetupCommand(program: Command) { program .command('setup') .description('set up the environment for the specified board') - .argument('', 'name of the board to setup (e.g., esp32)') + .argument('', 'name of the board to setup (e.g., esp32)') .action(handleSetupCommand); } - diff --git a/cli/src/commands/board/update.ts b/cli/src/commands/board/update.ts index bd7230f3..a3d3d52b 100644 --- a/cli/src/commands/board/update.ts +++ b/cli/src/commands/board/update.ts @@ -1,10 +1,11 @@ import { Command } from "commander"; -import { logger, LogStep, showErrorMessages, SkipStep } from "../../core/logger"; +import { logger, runStep, skip } from "../../core/logger"; import { CommandHandler } from "../command"; import { GLOBAL_SETTINGS } from "../../config/constants"; import * as fs from '../../core/fs'; import { exec } from "../../core/shell"; import * as path from 'path'; +import { buildHostRuntime } from "../../platforms/runtime/host-board-runtime"; class UpdateHandler extends CommandHandler { @@ -19,9 +20,12 @@ class UpdateHandler extends CommandHandler { async update() { try { - await this.updateRuntime(); + await this.updateRuntimeStep(); if (this.globalConfigHandler.isBoardSetup('esp32')) { - await this.updateEsp32(); + await this.updateEsp32Step(); + } + if (this.globalConfigHandler.isBoardSetup('host')) { + await this.updateHostStep(); } this.globalConfigHandler.setVersion(GLOBAL_SETTINGS.VM_VERSION); } catch (error) { @@ -44,28 +48,51 @@ class UpdateHandler extends CommandHandler { this.globalConfigHandler.save(); } - @LogStep('Updating Runtime...') - async updateRuntime() { - const globalConfig = this.globalConfigHandler.getConfig(); - if (globalConfig.runtimeDir === undefined || globalConfig.version === GLOBAL_SETTINGS.VM_VERSION) { - throw new SkipStep('not needed', undefined); - } - this.existingRuntimeDir = globalConfig.runtimeDir; - - fs.moveDir(this.existingRuntimeDir, this.tmpRuntimeDir); + private updateRuntimeStep() { + return runStep('Updating Runtime...', async () => { + const globalConfig = this.globalConfigHandler.getConfig(); + if (globalConfig.runtimeDir === undefined || globalConfig.version === GLOBAL_SETTINGS.VM_VERSION) { + return skip('not needed'); + } + this.existingRuntimeDir = globalConfig.runtimeDir; + await this.updateRuntime(this.existingRuntimeDir); + }); + } + + private updateEsp32Step() { + return runStep('Updating the environment for esp32...', async () => { + const esp32Config = this.globalConfigHandler.getBoardConfig('esp32')!; + if (esp32Config.idfVersion === GLOBAL_SETTINGS.ESP_IDF_VERSION) { + return skip('not needed'); + } + this.existingEspDir = esp32Config.rootDir; + await this.updateEsp32(this.existingEspDir); + }); + } + + private updateHostStep() { + return runStep('Updating the environment for host...', async () => { + const globalConfig = this.globalConfigHandler.getConfig(); + if (globalConfig.runtimeDir === undefined || globalConfig.version === GLOBAL_SETTINGS.VM_VERSION) { + return skip('not needed'); + } + await this.updateHost(globalConfig.runtimeDir); + }); + } + + private async updateRuntime(existingRuntimeDir: string) { + fs.moveDir(existingRuntimeDir, this.tmpRuntimeDir); await fs.downloadAndUnzip(GLOBAL_SETTINGS.RUNTIME_ZIP_URL, GLOBAL_SETTINGS.BLUESCRIPT_DIR); this.globalConfigHandler.setRuntimeDir(GLOBAL_SETTINGS.RUNTIME_DIR); } - @LogStep('Updating the environment for esp32...') - async updateEsp32() { - const esp32Config = this.globalConfigHandler.getBoardConfig('esp32')!; - if (esp32Config.idfVersion === GLOBAL_SETTINGS.ESP_IDF_VERSION) { - throw new SkipStep('not needed', undefined); - } - this.existingEspDir = esp32Config.rootDir; + private async updateHost(runtimeDir: string) { + const hostConfig = this.globalConfigHandler.getBoardConfig('host')!; + await buildHostRuntime(runtimeDir, hostConfig.buildDir); + } - fs.moveDir(this.existingEspDir, this.tmpEspDir); + private async updateEsp32(existingEspDir: string) { + fs.moveDir(existingEspDir, this.tmpEspDir); fs.makeDir(GLOBAL_SETTINGS.ESP_ROOT_DIR); await this.cloneEspIdf(); await this.runEspIdfInstallScript(); @@ -78,7 +105,7 @@ class UpdateHandler extends CommandHandler { } private async cloneEspIdf() { - await exec(`git clone --depth 1 -b ${GLOBAL_SETTINGS.ESP_IDF_VERSION} --recursive ${GLOBAL_SETTINGS.ESP_IDF_GIT_REPO}`, + await exec(`git clone --depth 1 -b ${GLOBAL_SETTINGS.ESP_IDF_VERSION} --recursive ${GLOBAL_SETTINGS.ESP_IDF_GIT_REPO}`, { cwd: GLOBAL_SETTINGS.ESP_ROOT_DIR }); } @@ -93,7 +120,6 @@ class UpdateHandler extends CommandHandler { } catch (error) { throw new Error('Failed to get xtensa gcc path.', {cause: error}); } - } } @@ -107,7 +133,7 @@ export async function handleUpdateCommand() { } catch (error) { logger.error(`Failed to update board environments.`); - showErrorMessages(error); + logger.showError(error); process.exit(1); } } @@ -117,4 +143,4 @@ export function registerUpdateCommand(program: Command) { .command('update') .description('update the board environments.') .action(handleUpdateCommand); -} \ No newline at end of file +} diff --git a/cli/src/commands/project/check.ts b/cli/src/commands/project/check.ts index 445e3f99..191bc20b 100644 --- a/cli/src/commands/project/check.ts +++ b/cli/src/commands/project/check.ts @@ -1,9 +1,9 @@ import { Command } from "commander"; -import { logger, runAsyncWithLogStep, showErrorMessages } from "../../core/logger"; +import { logger, runStep } from "../../core/logger"; import { ProjectConfigHandler } from "../../config/project-config"; import { cwd } from "../../core/shell"; import { CommandHandler } from "../command"; -import { CompilerAdapter, getCompilerAdapter } from "../../boards/compiler-adapters"; +import { CompilerAdapter, getCompilerAdapter } from "../../platforms"; class CheckHandler extends CommandHandler { private compilerAdapter: CompilerAdapter; @@ -16,8 +16,7 @@ class CheckHandler extends CommandHandler { } async check() { - const memoryLayout = this.compilerAdapter.getDummyMemoryLayout(); - await runAsyncWithLogStep('Compiling...', () => this.compilerAdapter.buildProject(memoryLayout)); + await runStep('Compiling...', () => this.compilerAdapter.buildForCheck()); } } @@ -31,7 +30,7 @@ export async function handleCheckCommand() { logger.success('Successfully checked BlueScript program.'); } catch (error) { logger.error(`Failed to check BlueScript program.`); - showErrorMessages(error); + logger.showError(error); process.exit(1); } } @@ -41,4 +40,4 @@ export function registerCheckCommand(program: Command) { .command('check') .description('check your project') .action(handleCheckCommand); -} \ No newline at end of file +} diff --git a/cli/src/commands/project/create.ts b/cli/src/commands/project/create.ts index b98977a7..07ab6428 100644 --- a/cli/src/commands/project/create.ts +++ b/cli/src/commands/project/create.ts @@ -2,8 +2,8 @@ import { Command } from "commander"; import inquirer from 'inquirer'; import chalk from "chalk"; import * as path from 'path'; -import { logger, showErrorMessages } from "../../core/logger"; -import { ProjectConfigHandler, PROJECT_DEFAULT_PATHS } from "../../config/project-config"; +import { logger } from "../../core/logger"; +import { ProjectConfigHandler } from "../../config/project-config"; import { cwd } from "../../core/shell"; import { BOARD_NAMES, BoardName, isValidBoard } from "../../config/board-utils"; import * as fs from '../../core/fs'; @@ -100,7 +100,7 @@ export async function handleCreateProjectCommand(name: string, options: { board? logger.info(`Next step: go to the project directory and run ${chalk.yellow('bscript project run')}`); } catch (error) { logger.error(`Failed to create a new project.`); - showErrorMessages(error); + logger.showError(error); process.exit(1); } } diff --git a/cli/src/commands/project/install.ts b/cli/src/commands/project/install.ts index a5af324a..3b977d06 100644 --- a/cli/src/commands/project/install.ts +++ b/cli/src/commands/project/install.ts @@ -1,5 +1,5 @@ import { Command } from "commander"; -import { logger, showErrorMessages } from "../../core/logger"; +import { logger } from "../../core/logger"; import { ProjectConfigHandler, PackageSource ,PROJECT_DEFAULT_PATHS } from "../../config/project-config"; import { cwd, exec } from "../../core/shell"; import * as fs from '../../core/fs'; @@ -46,12 +46,13 @@ class InstallationHandler extends CommandHandler { if (installedPackages.has(currentPkg.name)) continue; const pkgConfigHandler = await this.downloadPackage(currentPkg.url, currentPkg.version); - // pkgConfigHandler.checkVmVersion(this.projectConfigHandler.getConfig().vmVersion); pkgConfigHandler.checkBoardName(this.projectConfigHandler.getBoardName()); installedPackages.add(currentPkg.name); - pkgConfigHandler.getDepenencies().forEach((pkgDep) => { - installedPackages.add(pkgDep.name); - }); + for (const pkgDep of pkgConfigHandler.getDepenencies()) { + if (!installedPackages.has(pkgDep.name)) { + queue.push(pkgDep); + } + } } } @@ -102,7 +103,7 @@ export async function handleInstallCommand(url: string|undefined, options: {tag? const errorMessage = url ? `Failed to install ${url}.` : `Failed to install packages.`; logger.error(errorMessage); - showErrorMessages(error); + logger.showError(error); process.exit(1); } } @@ -114,4 +115,4 @@ export function registerInstallCommand(program: Command) { .argument('[git-url]', 'git repository URL to add as a dependency') .option('-t, --tag ', 'git tag or branch to checkout (e.g., v1.0.0)') .action(handleInstallCommand); -} \ No newline at end of file +} diff --git a/cli/src/commands/project/run.ts b/cli/src/commands/project/run.ts index 328f02fb..75a7949d 100644 --- a/cli/src/commands/project/run.ts +++ b/cli/src/commands/project/run.ts @@ -4,56 +4,64 @@ import * as readline from 'readline'; import http from 'http'; import sirv from 'sirv'; import path from 'path'; -import { logger, runAsyncWithLogStep, ProgramLogger, showErrorMessages, replLogger } from "../../core/logger"; +import { logger, ProgramOutput, createBoxedOutput, createConsoleOutput, createWebSocketOutput, runStep } from "../../core/logger"; import { DEFAULT_DEVICE_NAME, ProjectConfigHandler } from "../../config/project-config"; import { cwd, exec } from "../../core/shell"; import { CommandHandler } from "../command"; -import { CompilerAdapter, getCompilerAdapter } from "../../boards/compiler-adapters"; -import { BleDeviceManager } from "../../services/device-manager"; -import { CompileError, ExecutableBinary } from "@bscript/lang"; +import { BoardRuntime, CompilerAdapter, createPlatformSession } from "../../platforms"; +import { CompileError, CompileOutput } from "@bscript/lang"; import { WebSocketConnection } from "../../services/websocket"; +import { SerialTaskQueue } from "../../core/serial-task-queue"; class RunHandler extends CommandHandler { - protected compilerAdapter: CompilerAdapter; - protected deviceManager: BleDeviceManager; - protected programLogger: ProgramLogger; + protected compiler: CompilerAdapter; + protected runtime: BoardRuntime; + protected programOutput: ProgramOutput; private globalKeypressHandler?: (str: string, key: any) => void; - private ctrlDKeypressHandler?: (str: string, key: any) => void; + private ctrlDKeypressHandler?: (str: string, key: any) => void; constructor(protected projectConfigHandler: ProjectConfigHandler) { super(); const boardName = this.projectConfigHandler.getBoardName(); - this.compilerAdapter = getCompilerAdapter(boardName, this.globalConfigHandler, this.projectConfigHandler); - const deviceName = this.projectConfigHandler.getConfig().deviceName ?? DEFAULT_DEVICE_NAME; - this.programLogger = new ProgramLogger(); - this.deviceManager = new BleDeviceManager(deviceName, this.programLogger, () => { - this.programLogger.end(); - logger.error('BLE disconnected.'); - process.exit(1); - }); + this.programOutput = createBoxedOutput(); + + const platform = createPlatformSession( + boardName, + this.globalConfigHandler, + this.projectConfigHandler, + deviceName, + this.programOutput, + () => { + this.programOutput.onRunEnd?.(); + logger.error("Disconnected."); + process.exit(1); + }, + ); + this.compiler = platform.compiler; + this.runtime = platform.runtime; } async run(): Promise { - await runAsyncWithLogStep('Connecting via BLE...', () => this.deviceManager.connect()); - const memoryLayout = await runAsyncWithLogStep('Initializing Device...', () => this.deviceManager.initDevice()); - const bin = await runAsyncWithLogStep('Compiling...', () => this.compilerAdapter.buildProject(memoryLayout)); - await runAsyncWithLogStep('Loading...', () => this.deviceManager.load(bin)); - return this.executeBinary(bin); + await runStep('Connecting...', () => this.runtime.connect()); + const compileContext = await runStep('Initializing', () => this.runtime.prepare()); + const compileOutput = await runStep('Compiling...', () => this.compiler.buildProject(compileContext)); + await runStep('Loading...', () => this.runtime.load(compileOutput!)); + return this.executeProgram(compileOutput!); } async close() { - await runAsyncWithLogStep('Disconnecting...', () => this.deviceManager.disconnect()); - process.exit(0); + await runStep('Disconnecting...', async () => this.runtime.disconnect()); } - private setupStdin() { - readline.emitKeypressEvents(process.stdin); - if (process.stdin.isTTY) { - process.stdin.setRawMode(true); + protected setupStdin() { + if (!process.stdin.isTTY) { + return; } + readline.emitKeypressEvents(process.stdin); + process.stdin.setRawMode(true); this.globalKeypressHandler = (str, key) => { if (key && key.ctrl && key.name === 'c') { process.exit(0); @@ -64,6 +72,9 @@ class RunHandler extends CommandHandler { } private resetStdin() { + if (!process.stdin.isTTY) { + return; + } if (this.globalKeypressHandler) { process.stdin.off('keypress', this.globalKeypressHandler); this.globalKeypressHandler = undefined; @@ -72,36 +83,43 @@ class RunHandler extends CommandHandler { process.stdin.off('keypress', this.ctrlDKeypressHandler); this.ctrlDKeypressHandler = undefined; } + process.stdin.setRawMode(false); } - private async executeBinary(bin: ExecutableBinary) { - this.setupStdin(); + private async executeProgram(output: CompileOutput) { logger.info("Start executing program. Type 'Ctrl-D' to exit."); - this.programLogger.start(); + this.programOutput.onRunStart?.(); try { + if (!process.stdin.isTTY) { + await this.runtime.execute(output); + return false; + } + + this.setupStdin(); const interrupted = await new Promise((resolve, reject) => { this.ctrlDKeypressHandler = (str, key) => { if (key && key.ctrl && key.name === 'd') { - resolve(true); + resolve(true); if (str) process.stdout.write(str); } }; process.stdin.on('keypress', this.ctrlDKeypressHandler); - this.deviceManager.execute(bin) + this.runtime.execute(output) .then(() => resolve(false)) .catch(reject); }); return interrupted; } finally { - this.programLogger.end(); - this.resetStdin(); + this.programOutput.onRunEnd?.(); + this.resetStdin(); } } } class RunWithReplHandler extends RunHandler { private rl: readline.Interface; + private readonly taskQueue = new SerialTaskQueue(); constructor(projectConfigHandler: ProjectConfigHandler) { super(projectConfigHandler); @@ -118,7 +136,7 @@ class RunWithReplHandler extends RunHandler { return interrupted; } - this.deviceManager.updateLogger(replLogger); + this.runtime.setOutput(createConsoleOutput()); await this.runRepl(); return false; } @@ -127,20 +145,25 @@ class RunWithReplHandler extends RunHandler { logger.info("Start REPL. Type 'Ctrl-D' to exit."); this.rl.prompt(); return new Promise((resolve, reject) => { - this.rl.on('line', async (line) => { - try { - const bin = await this.compilerAdapter.compileFragment(line); - await this.deviceManager.load(bin); - await this.deviceManager.execute(bin); - this.rl.prompt(); - } catch (error) { - if (error instanceof CompileError) { - replLogger.error("** compile error: " + error.toString()); + this.rl.on('line', (line) => { + this.rl.pause(); + this.taskQueue.enqueue(async () => { + try { + const output = await this.compiler.compileFragment(line); + await this.runtime.load(output); + await this.runtime.execute(output); + } catch (error) { + if (error instanceof CompileError) { + logger.error("** compile error: " + error.toString()); + } else { + reject(error); + return; + } + } finally { + this.rl.resume(); this.rl.prompt(); - } else { - reject(error); } - } + }); }); this.rl.on('close', () => { resolve(); @@ -152,6 +175,7 @@ class RunWithReplHandler extends RunHandler { class RunWithNotebookHandler extends RunHandler { private ws: WebSocketConnection | null = null; private server: http.Server | null = null; + private readonly executeQueue = new SerialTaskQueue(); constructor(projectConfigHandler: ProjectConfigHandler) { super(projectConfigHandler); @@ -166,6 +190,7 @@ class RunWithNotebookHandler extends RunHandler { await this.startUiServer(); logger.info("Type 'Ctrl-D' to exit."); + this.setupStdin(); return new Promise((resolve) => { process.stdin.on('keypress', (str, key) => { if (key && key.ctrl && key.name === 'd') { @@ -176,9 +201,9 @@ class RunWithNotebookHandler extends RunHandler { } async close(): Promise { - await super.close(); this.server?.close(); this.ws?.close(); + await super.close(); } private startUiServer() { @@ -199,15 +224,15 @@ class RunWithNotebookHandler extends RunHandler { resolve(); }); }); - + } private openBrowser(port: number|string) { const url = `http://localhost:${port}`; - const startCommand = - process.platform === 'win32' ? 'start' : + const startCommand = + process.platform === 'win32' ? 'start' : process.platform === 'darwin' ? 'open' : 'xdg-open'; - + exec(`${startCommand} ${url}`, {silent: true}); } @@ -216,53 +241,50 @@ class RunWithNotebookHandler extends RunHandler { this.ws = new WebSocketConnection(port); const service = this.ws.getService('repl'); this.ws.open(); - this.deviceManager.updateLogger({ - log: (message: string) => { service.log(message); }, - error: (message: string) => { service.error(message); } - }); - service.on('execute', async (code: string) => { - try { - let {bin, time} = await this.compile(code); - service.finishCompilation(time); - time = await this.load(bin); - service.finishLoading(time); - time = await this.execute(bin); - service.finishExecution(time); - } catch (error) { - if (error instanceof CompileError) { - service.finishCompilation(-1, error.toString()); - } else { - console.log(error) - throw error; + this.runtime.setOutput(createWebSocketOutput(service)); + service.on('execute', (code: string) => { + this.executeQueue.enqueue(async () => { + try { + let {output, time} = await this.compile(code); + service.finishCompilation(time); + time = await this.load(output); + service.finishLoading(time); + time = await this.execute(output); + service.finishExecution(time); + } catch (error) { + if (error instanceof CompileError) { + service.finishCompilation(-1, error.toString()); + } else { + logger.showError(error); + throw error; + } } - } + }); }); logger.info(`WebSocket server is running at ws://localhost:${port}`); } private async compile(code: string) { const start = performance.now(); - const bin = await this.compilerAdapter.compileFragment(code); - return {bin, time: performance.now() - start}; + const output = await this.compiler.compileFragment(code); + return {output, time: performance.now() - start}; } - private async load(bin: ExecutableBinary) { + private async load(output: CompileOutput) { const start = performance.now(); - await this.deviceManager.load(bin); + await this.runtime.load(output); return performance.now() - start; } - private async execute(bin: ExecutableBinary) { - const start = performance.now(); - await this.deviceManager.execute(bin); - return performance.now() - start; + private async execute(output: CompileOutput) { + return await this.runtime.execute(output); } } export async function handleRunCommand(options: {withRepl: boolean, withNotebook: boolean}) { + let handler: RunHandler | undefined; try { const projectConfigHandler = ProjectConfigHandler.load(cwd()); - let handler: RunHandler; if (options.withRepl) { handler = new RunWithReplHandler(projectConfigHandler); } else if (options.withNotebook) { @@ -273,9 +295,18 @@ export async function handleRunCommand(options: {withRepl: boolean, withNotebook await handler.run(); await handler.close(); + process.exit(0); + } catch (error) { + if (handler) { + try { + await handler.close(); + } catch { + // Ignore cleanup errors after a run failure. + } + } logger.error(`Failed to run BlueScript program.`); - showErrorMessages(error); + logger.showError(error); process.exit(1); } } @@ -293,4 +324,4 @@ export function registerRunCommand(program: Command) { .conflicts('withRepl') ) .action(handleRunCommand); -} \ No newline at end of file +} diff --git a/cli/src/commands/project/uninstall.ts b/cli/src/commands/project/uninstall.ts index 79231cba..5aa920aa 100644 --- a/cli/src/commands/project/uninstall.ts +++ b/cli/src/commands/project/uninstall.ts @@ -1,5 +1,5 @@ import { Command } from "commander"; -import { logger, showErrorMessages } from "../../core/logger"; +import { logger } from "../../core/logger"; import { ProjectConfigHandler, PROJECT_DEFAULT_PATHS } from "../../config/project-config"; import { cwd } from "../../core/shell"; import * as fs from '../../core/fs'; @@ -37,7 +37,7 @@ export async function handleUninstallCommand(packageName: string) { uninstallHandler.uninstall(packageName); } catch (error) { logger.error(`Failed to uninstall ${packageName}.`); - showErrorMessages(error); + logger.showError(error); process.exit(1); } } diff --git a/cli/src/commands/repl.ts b/cli/src/commands/repl.ts index d1e4781f..426d0da6 100644 --- a/cli/src/commands/repl.ts +++ b/cli/src/commands/repl.ts @@ -1,87 +1,100 @@ import { Command } from "commander"; -import { logger, runAsyncWithLogStep, replLogger, showErrorMessages } from "../core/logger"; +import { logger, runStep } from "../core/logger"; +import { createConsoleOutput } from "../core/logger/program-output"; import { DEFAULT_DEVICE_NAME, PROJECT_DEFAULT_PATHS, ProjectConfigHandler } from "../config/project-config"; -import { CompileError, ExecutableBinary, MemoryLayout } from "@bscript/lang"; import * as path from 'path'; import * as readline from 'readline'; import chalk from "chalk"; import * as fs from '../core/fs'; import { CommandHandler } from "./command"; import { GLOBAL_SETTINGS } from "../config/constants"; -import { CompilerAdapter, getCompilerAdapter } from "../boards/compiler-adapters"; -import { BleDeviceManager } from "../services/device-manager"; +import { CompileContext, createPlatformSession } from "../platforms"; import { BoardName } from "../config/board-utils"; +import { CompileError, CompileOutput } from "@bscript/lang"; +import { SerialTaskQueue } from "../core/serial-task-queue"; + +type ReplReadlineFactory = () => readline.Interface; + +function defaultReplReadlineFactory(): readline.Interface { + return readline.createInterface({ + input: process.stdin, + output: process.stdout, + prompt: chalk.blue.bold('> '), + }); +} class ReplHandler extends CommandHandler { static readonly TEMP_PROJECT_NAME = 'temp'; static readonly tempProjectDir = path.join(GLOBAL_SETTINGS.BLUESCRIPT_DIR, this.TEMP_PROJECT_NAME); private projectConfigHandler: ProjectConfigHandler; - private compilerAdapter: CompilerAdapter; - private deviceManager: BleDeviceManager; + private platform: ReturnType; private rl: readline.Interface; + private compileContext?: CompileContext; private isFirstCompile: boolean; + private readonly taskQueue = new SerialTaskQueue(); - constructor(private boardName: string) { + constructor( + private boardName: string, + private createReadline: ReplReadlineFactory = defaultReplReadlineFactory, + ) { super(); - this.projectConfigHandler = - ProjectConfigHandler.createTemplate(ReplHandler.TEMP_PROJECT_NAME, this.boardName as BoardName, ReplHandler.tempProjectDir); - this.compilerAdapter = getCompilerAdapter(this.boardName as BoardName, this.globalConfigHandler, this.projectConfigHandler); + const board = this.boardName as BoardName; + this.projectConfigHandler = + ProjectConfigHandler.createTemplate(ReplHandler.TEMP_PROJECT_NAME, board, ReplHandler.tempProjectDir); - const deviceName = DEFAULT_DEVICE_NAME; - this.deviceManager = new BleDeviceManager(deviceName, replLogger, () => { - logger.error('BLE disconnected.'); - this.deleteTempProject(); - process.exit(1); - }); + this.platform = createPlatformSession( + board, + this.globalConfigHandler, + this.projectConfigHandler, + DEFAULT_DEVICE_NAME, + createConsoleOutput(), + () => { + logger.error('Disconnected.'); + this.deleteTempProject(); + process.exit(1); + }, + ); - this.rl = readline.createInterface({ - input: process.stdin, - output: process.stdout, - prompt: chalk.blue.bold('> ') - }); + this.rl = this.createReadline(); this.isFirstCompile = true; } async start() { - await runAsyncWithLogStep('Connecting via BLE...', () => this.deviceManager.connect()); - const memoryLayout = await runAsyncWithLogStep('Initializing Device...', () => this.deviceManager.initDevice()); + await runStep('Connecting...', () => this.platform.runtime.connect()); + this.compileContext = await runStep('Initializing...', () => this.platform.runtime.prepare())!; this.createTempProject(); - await this.runRepl(memoryLayout); + await this.runRepl(); this.deleteTempProject(); + this.rl.close(); - await runAsyncWithLogStep('Disconnecting...', () => this.deviceManager.disconnect()); + await runStep('Disconnecting...', () => this.platform.runtime.disconnect()); process.exit(0); } - private runRepl(memoryLayout: MemoryLayout) { + private runRepl() { logger.info("Start REPL. Type 'Ctrl-D' to exit."); this.rl.prompt(); return new Promise((resolve, reject) => { - this.rl.on('line', async (line) => { - try { - let bin: ExecutableBinary; - if (this.isFirstCompile) { - // Compile first line as index.bs. - this.writeEntryFile(line); - bin = await this.compilerAdapter.buildProject(memoryLayout); - this.isFirstCompile = false; - } else { - bin = await this.compilerAdapter.compileFragment(line); - } - await this.deviceManager.load(bin); - await this.deviceManager.execute(bin); - this.rl.prompt(); - } catch (error) { - if (error instanceof CompileError) { - replLogger.error("** compile error: " + error.toString()); + this.rl.on('line', (line) => { + this.rl.pause(); + this.taskQueue.enqueue(async () => { + try { + await this.processReplLine(line); + } catch (error) { + if (error instanceof CompileError) { + logger.error("** compile error: " + error.toString()); + } else { + reject(error); + return; + } + } finally { + this.rl.resume(); this.rl.prompt(); - } else { - reject(error); } - } + }); }); this.rl.on('close', () => { resolve(); @@ -89,11 +102,28 @@ class ReplHandler extends CommandHandler { }); } + private async processReplLine(line: string) { + let output: CompileOutput; + if (this.isFirstCompile) { + this.writeEntryFile(line); + output = await this.platform.compiler.buildProject(this.compileContext); + this.isFirstCompile = false; + } else { + output = await this.platform.compiler.compileFragment(line); + } + await this.platform.runtime.load(output); + await this.platform.runtime.execute(output); + } + private createTempProject() { if (fs.exists(ReplHandler.tempProjectDir)) { fs.removeDir(ReplHandler.tempProjectDir) } fs.makeDir(ReplHandler.tempProjectDir); + const runtimeDir = this.globalConfigHandler.getConfig().runtimeDir; + if (runtimeDir) { + this.projectConfigHandler.update({ runtimeDir }); + } this.projectConfigHandler.save(ReplHandler.tempProjectDir); } @@ -111,13 +141,16 @@ class ReplHandler extends CommandHandler { } } -export async function handleReplCommand(options: { board: string }) { +export async function handleReplCommand( + options: { board: string }, + deps?: { createReadline?: ReplReadlineFactory }, +) { try { - const handler = new ReplHandler(options.board); + const handler = new ReplHandler(options.board, deps?.createReadline); await handler.start(); } catch (error) { logger.error(`Error while running REPL.`); - showErrorMessages(error); + logger.showError(error); process.exit(1); } } @@ -128,4 +161,4 @@ export function registerReplCommand(program: Command) { .description('start REPL') .requiredOption('-b, --board ', 'board name') .action(handleReplCommand); -} \ No newline at end of file +} diff --git a/cli/src/config/board-utils.ts b/cli/src/config/board-utils.ts index 2ace623f..2563f65e 100644 --- a/cli/src/config/board-utils.ts +++ b/cli/src/config/board-utils.ts @@ -1,3 +1,3 @@ -export const BOARD_NAMES = ['esp32'] as const; +export const BOARD_NAMES = ['esp32', 'host'] as const; export type BoardName = (typeof BOARD_NAMES)[number]; export const isValidBoard = (board: string): board is BoardName => (BOARD_NAMES as readonly string[]).includes(board); diff --git a/cli/src/config/global-config.ts b/cli/src/config/global-config.ts index 27810b47..223ee6e3 100644 --- a/cli/src/config/global-config.ts +++ b/cli/src/config/global-config.ts @@ -11,8 +11,13 @@ const esp32BoardSchema = z.object({ xtensaGccDir: z.string(), }); +const hostBoardSchema = z.object({ + buildDir: z.string(), +}); + const boardConfigSchema = z.object({ esp32: esp32BoardSchema.optional(), + host: hostBoardSchema.optional(), }); const globalConfigSchema = z.object({ @@ -22,6 +27,7 @@ const globalConfigSchema = z.object({ }); export type Esp32BoardConfig = z.infer; +export type HostBoardConfig = z.infer; export type BoardConfig = z.infer; export type GlobalConfig = z.infer; diff --git a/cli/src/config/project-config.ts b/cli/src/config/project-config.ts index 29642fdd..da1944f3 100644 --- a/cli/src/config/project-config.ts +++ b/cli/src/config/project-config.ts @@ -34,8 +34,13 @@ const esp32ProjectSchema = baseConfigSchema.extend({ espIdfComponents: z.array(z.string()).default([]), }); +const hostProjectSchema = baseConfigSchema.extend({ + boardName: z.literal('host'), +}); + const projectConfigSchema = z.discriminatedUnion('boardName', [ esp32ProjectSchema, + hostProjectSchema, ]); export type ProjectConfig = z.infer; diff --git a/cli/src/core/logger.ts b/cli/src/core/logger.ts deleted file mode 100644 index b74a506c..00000000 --- a/cli/src/core/logger.ts +++ /dev/null @@ -1,220 +0,0 @@ -import chalk from 'chalk'; -import readline from 'readline'; - -const ERROR_PREFIX = chalk.red.bold('ERROR:'); -const WARN_PREFIX = chalk.yellow.bold('WARN:'); -const INFO_PREFIX = chalk.blue.bold('INFO:'); -const SUCCESS_PREFIX = chalk.green.bold('SUCCESS:'); - -export class LogUpdater { - private stream: NodeJS.WriteStream; - private lastOutput = ''; - private isUpdating = false; - - constructor(stream: NodeJS.WriteStream) { - this.stream = stream; - } - - public update(...text: string[]): void { - this.clear(); - const newOutput = text.join(' '); - this.stream.write(newOutput); - this.lastOutput = newOutput; - this.isUpdating = true; - } - - public persistent(...text: string[]): void { - this.update(...text); - this.done(); - } - - public done(): void { - if (!this.isUpdating) { - return; - } - this.stream.write('\n'); - this.isUpdating = false; - this.lastOutput = ''; - } - - public clear(): void { - if (!this.isUpdating) { - return; - } - const lines = this.getLineCount(this.lastOutput); - for (let i = 0; i < lines; i++) { - if (i > 0) { - readline.moveCursor(this.stream, 0, -1); - } - readline.cursorTo(this.stream, 0); - readline.clearLine(this.stream, 1); - } - this.isUpdating = false; - this.lastOutput = ''; - } - - private getLineCount(str: string): number { - const columns = this.stream.columns || 80; - let lineCount = 0; - for (const line of str.split('\n')) { - const strippedLine = line.replace(/[\u001b\u009b][[()#;?]*(?:[0-9]{1,4}(?:;[0-9]{0,4})*)?[0-9A-ORZcf-nqry=><]/g, ''); - lineCount += Math.max(1, Math.ceil(strippedLine.length / columns)); - } - return lineCount; - } -} - -export const logUpdater = new LogUpdater(process.stdout); - -export const logger = { - error(...messages: string[]): void { - logUpdater.done(); - console.log(ERROR_PREFIX, ...messages); - }, - - warn(...messages: string[]): void { - logUpdater.done(); - console.log(WARN_PREFIX, ...messages); - }, - - info(...messages: string[]): void { - logUpdater.done(); - console.log(INFO_PREFIX, ...messages); - }, - - success(...messages: string[]): void { - logUpdater.done(); - console.log(SUCCESS_PREFIX, ...messages); - }, - - log(...messages: string[]): void { - logUpdater.done(); - console.log(...messages); - }, - - br(): void { - console.log(); - } -}; - -export const replLogger = { - log(message: string): void { - console.log(message.trimEnd()); - }, - - error(message: string): void { - console.log(chalk.red.bold(message.trimEnd())); - } -} - - -export class ProgramLogger { - private isLogging = false; - private boxWidth: number; - - constructor() { - const columns = process.stdout.columns || 60; - this.boxWidth = columns & ~1; - } - - start() { - this.isLogging = true; - const lineLength = (this.boxWidth - 8) / 2 - process.stdout.write(`\n${'='.repeat(lineLength)} OUTPUT ${'='.repeat(lineLength)}\n`); - } - end() { - if (!this.isLogging) return; - process.stdout.write(`${'='.repeat(this.boxWidth)}\n\n`); - this.isLogging = false; - } - - log(message: string) { - if (!this.isLogging) return; - process.stdout.write(message); - } - - error(message: string) { - if (!this.isLogging) return; - process.stdout.write(chalk.red.bold(message)); - } -} - -export class SkipStep { - result: any; - message: string; - - constructor(message: string, result: any) { - this.message = message; - this.result = result; - } -} - -export function LogStep(message: string) { - return function ( - target: any, - propertyKey: string, - descriptor: PropertyDescriptor - ) { - const originalMethod = descriptor.value; - - descriptor.value = async function (...args: any[]) { - logUpdater.update(INFO_PREFIX, message); - try { - const result = await originalMethod.apply(this, args); - logUpdater.persistent(INFO_PREFIX, message, chalk.green('OK')); - return result; - } catch (error) { - if (error instanceof SkipStep) { - logUpdater.persistent(INFO_PREFIX, message, chalk.yellow(`Skipped - ${error.message}`)); - return error.result; - } else { - logUpdater.persistent(INFO_PREFIX, message, chalk.red('Failed')); - throw error; - } - } - }; - - return descriptor; - } -} - -export async function runAsyncWithLogStep(message: string, action: () => Promise): Promise { - logUpdater.update(INFO_PREFIX, message); - try { - const result = await action(); - logUpdater.persistent(INFO_PREFIX, message, chalk.green('OK')); - return result; - } catch (error) { - logUpdater.persistent(INFO_PREFIX, message, chalk.red('Failed')); - throw error; - } -} - -export function runWithLogStep(message: string, action: () => T): T { - logUpdater.update(INFO_PREFIX, message); - try { - const result = action(); - logUpdater.persistent(INFO_PREFIX, message, chalk.green('OK')); - return result; - } catch (error) { - logUpdater.persistent(INFO_PREFIX, message, chalk.red('Failed')); - throw error; - } -} - -export function showErrorMessages(error: unknown, indent: number = 2) { - const messages: string[] = []; - let currentError = error; - while (currentError) { - if (currentError instanceof Error) { - messages.push(currentError.message); - currentError = currentError.cause; - } else { - messages.push(`Unknown Error: ${error}`); - break; - } - } - messages.forEach(m => { - console.log(' '.repeat(indent) + m); - }); -} \ No newline at end of file diff --git a/cli/src/core/logger/cli-logger.ts b/cli/src/core/logger/cli-logger.ts new file mode 100644 index 00000000..36c8401b --- /dev/null +++ b/cli/src/core/logger/cli-logger.ts @@ -0,0 +1,71 @@ +import chalk from 'chalk'; +import { logUpdater } from './log-updater'; + + +export const ERROR_PREFIX = chalk.red.bold('ERROR:'); +export const WARN_PREFIX = chalk.yellow.bold('WARN:'); +export const INFO_PREFIX = chalk.blue.bold('INFO:'); +export const SUCCESS_PREFIX = chalk.green.bold('SUCCESS:'); + +export interface CliLogger { + error(...messages: string[]): void; + warn(...messages: string[]): void; + info(...messages: string[]): void; + success(...messages: string[]): void; + log(...messages: string[]): void; + br(): void; + showError(error: unknown, indent?: number): void; +} + +export const logger: CliLogger = { + error(...messages: string[]): void { + logUpdater.done(); + console.log(ERROR_PREFIX, ...messages); + }, + + warn(...messages: string[]): void { + logUpdater.done(); + console.log(WARN_PREFIX, ...messages); + }, + + info(...messages: string[]): void { + logUpdater.done(); + console.log(INFO_PREFIX, ...messages); + }, + + success(...messages: string[]): void { + logUpdater.done(); + console.log(SUCCESS_PREFIX, ...messages); + }, + + log(...messages: string[]): void { + logUpdater.done(); + console.log(...messages); + }, + + br(): void { + console.log(); + }, + + showError(error: unknown, indent: number = 2): void { + logUpdater.done(); + for (const message of collectErrorMessages(error)) { + console.log(' '.repeat(indent) + message); + } + }, +}; + +function collectErrorMessages(error: unknown): string[] { + const messages: string[] = []; + let currentError = error; + while (currentError) { + if (currentError instanceof Error) { + messages.push(currentError.message); + currentError = currentError.cause; + } else { + messages.push(`Unknown Error: ${String(error)}`); + break; + } + } + return messages; +} diff --git a/cli/src/core/logger/index.ts b/cli/src/core/logger/index.ts new file mode 100644 index 00000000..15e2e3ea --- /dev/null +++ b/cli/src/core/logger/index.ts @@ -0,0 +1,9 @@ +export { logger } from './cli-logger'; +export { logUpdater } from './log-updater'; +export { runStep, skip } from './step-runner'; +export { + ProgramOutput, + createBoxedOutput, + createConsoleOutput, + createWebSocketOutput, +} from './program-output'; diff --git a/cli/src/core/logger/log-updater.ts b/cli/src/core/logger/log-updater.ts new file mode 100644 index 00000000..90c2fc1f --- /dev/null +++ b/cli/src/core/logger/log-updater.ts @@ -0,0 +1,61 @@ +import readline from 'readline'; + +export class LogUpdater { + private stream: NodeJS.WriteStream; + private lastOutput = ''; + private isUpdating = false; + + constructor(stream: NodeJS.WriteStream) { + this.stream = stream; + } + + public update(...text: string[]): void { + this.clear(); + const newOutput = text.join(' '); + this.stream.write(newOutput); + this.lastOutput = newOutput; + this.isUpdating = true; + } + + public persistent(...text: string[]): void { + this.update(...text); + this.done(); + } + + public done(): void { + if (!this.isUpdating) { + return; + } + this.stream.write('\n'); + this.isUpdating = false; + this.lastOutput = ''; + } + + public clear(): void { + if (!this.isUpdating) { + return; + } + const lines = this.getLineCount(this.lastOutput); + for (let i = 0; i < lines; i++) { + if (i > 0) { + readline.moveCursor(this.stream, 0, -1); + } + readline.cursorTo(this.stream, 0); + readline.clearLine(this.stream, 1); + } + this.isUpdating = false; + this.lastOutput = ''; + } + + private getLineCount(str: string): number { + const columns = this.stream.columns || 80; + let lineCount = 0; + for (const line of str.split('\n')) { + const strippedLine = line.replace(/[\u001b\u009b][[()#;?]*(?:[0-9]{1,4}(?:;[0-9]{0,4})*)?[0-9A-ORZcf-nqry=><]/g, ''); + lineCount += Math.max(1, Math.ceil(strippedLine.length / columns)); + } + return lineCount; + } +} + +export const logUpdater = new LogUpdater(process.stdout); diff --git a/cli/src/core/logger/program-output.ts b/cli/src/core/logger/program-output.ts new file mode 100644 index 00000000..99b52626 --- /dev/null +++ b/cli/src/core/logger/program-output.ts @@ -0,0 +1,60 @@ +import chalk from 'chalk'; + +export interface ProgramOutput { + onRunStart?(): void; + onRunEnd?(): void; + write(message: string): void; + writeError(message: string): void; +} + +export function createConsoleOutput(): ProgramOutput { + return { + write(message: string) { + console.log(message.trimEnd()); + }, + writeError(message: string) { + console.log(chalk.red.bold(message.trimEnd())); + }, + }; +} + +export function createBoxedOutput(): ProgramOutput { + let isRunning = false; + const columns = process.stdout.columns || 60; + const boxWidth = columns & ~1; + + return { + onRunStart() { + isRunning = true; + const lineLength = (boxWidth - 8) / 2; + process.stdout.write(`\n${'='.repeat(lineLength)} OUTPUT ${'='.repeat(lineLength)}\n`); + }, + onRunEnd() { + if (!isRunning) return; + process.stdout.write(`${'='.repeat(boxWidth)}\n\n`); + isRunning = false; + }, + write(message: string) { + if (!isRunning) return; + process.stdout.write(message); + }, + writeError(message: string) { + if (!isRunning) return; + process.stdout.write(chalk.red.bold(message)); + }, + }; +} + +export function createWebSocketOutput(service: { + log(message: string): void | Promise; + error(message: string): void | Promise; +}): ProgramOutput { + return { + write(message: string) { + void service.log(message); + }, + writeError(message: string) { + void service.error(message); + }, + }; +} diff --git a/cli/src/core/logger/step-runner.ts b/cli/src/core/logger/step-runner.ts new file mode 100644 index 00000000..ba2d65e7 --- /dev/null +++ b/cli/src/core/logger/step-runner.ts @@ -0,0 +1,27 @@ +import chalk from 'chalk'; +import { INFO_PREFIX } from './cli-logger'; +import { logUpdater } from './log-updater'; + +export class StepSkip { + constructor(public readonly reason: string) {} +} + +export function skip(reason: string): StepSkip { + return new StepSkip(reason); +} + +export async function runStep(message: string, action: () => Promise): Promise { + logUpdater.update(INFO_PREFIX, message); + try { + const result = await action(); + if (result instanceof StepSkip) { + logUpdater.persistent(INFO_PREFIX, message, chalk.yellow(`Skipped - ${result.reason}`)); + return undefined; + } + logUpdater.persistent(INFO_PREFIX, message, chalk.green('OK')); + return result; + } catch (error) { + logUpdater.persistent(INFO_PREFIX, message, chalk.red('Failed')); + throw error; + } +} diff --git a/cli/src/core/serial-task-queue.ts b/cli/src/core/serial-task-queue.ts new file mode 100644 index 00000000..02393bc5 --- /dev/null +++ b/cli/src/core/serial-task-queue.ts @@ -0,0 +1,21 @@ +export class SerialTaskQueue { + private processing = false; + private queue: Array<() => Promise> = []; + + enqueue(task: () => Promise): void { + this.queue.push(task); + void this.drain(); + } + + private async drain(): Promise { + if (this.processing) { + return; + } + this.processing = true; + while (this.queue.length > 0) { + const task = this.queue.shift()!; + await task(); + } + this.processing = false; + } +} diff --git a/cli/src/core/shell.ts b/cli/src/core/shell.ts index 70d98c7b..a3490381 100644 --- a/cli/src/core/shell.ts +++ b/cli/src/core/shell.ts @@ -54,7 +54,7 @@ export function exec(command: string, options?: {cwd?: string, silent?: boolean} } function getErrorMessage(command: string, code: number|null, stdout: string, stderr: string) { - let message = `Command faild: ${command}\n`; + let message = `Command failed: ${command}\n`; if (code) message += `> Exit code: ${code}\n`; message += `> Stdout: ${stdout === '' ? 'N/A' : stdout}\n`; diff --git a/cli/src/platforms/compiler/compiler-adapter.ts b/cli/src/platforms/compiler/compiler-adapter.ts new file mode 100644 index 00000000..70a3d0db --- /dev/null +++ b/cli/src/platforms/compiler/compiler-adapter.ts @@ -0,0 +1,13 @@ +import { CompileOutput, MemoryLayout } from "@bscript/lang"; +import { BoardName } from "../../config/board-utils"; + +export type CompileContext = { + memoryLayout?: MemoryLayout; +}; + +export interface CompilerAdapter { + readonly boardName: BoardName; + buildForCheck(): Promise; + buildProject(context?: CompileContext): Promise; + compileFragment(src: string): Promise; +} diff --git a/cli/src/platforms/compiler/esp32-compiler-adapter.ts b/cli/src/platforms/compiler/esp32-compiler-adapter.ts new file mode 100644 index 00000000..3d4d3613 --- /dev/null +++ b/cli/src/platforms/compiler/esp32-compiler-adapter.ts @@ -0,0 +1,104 @@ +import { GlobalConfigHandler, Esp32BoardConfig } from "../../config/global-config"; +import { ProjectConfigHandler, PROJECT_DEFAULT_PATHS } from "../../config/project-config"; +import { BoardName } from "../../config/board-utils"; +import { + CompilerSession, MemoryImage, MemoryLayout, + Esp32Toolchain, Esp32ToolchainConfig, ProjectForEsp32, PackageForEsp32 +} from "@bscript/lang"; +import { CompilerAdapter, CompileContext } from "./compiler-adapter"; +import * as path from 'path'; + + +const DUMMY_MEMORY_LAYOUT: MemoryLayout = { + iram: { address: 0x40096c34, size: 1000000 }, + dram: { address: 0x3ffd5b1c, size: 1000000 }, + iflash: { address: 0x40150000, size: 1000000 }, + dflash: { address: 0x3f43a000, size: 1000000 }, +}; + +export class Esp32CompilerAdapter implements CompilerAdapter { + readonly boardName: BoardName = 'esp32'; + private boardConfig: Esp32BoardConfig; + private compiler?: CompilerSession; + + constructor( + private globalConfigHandler: GlobalConfigHandler, + private projectConfigHandler: ProjectConfigHandler, + ) { + const boardConfig = this.globalConfigHandler.getBoardConfig('esp32'); + if (boardConfig === undefined) { + throw new Error(`The environment for ${this.boardName} is not set up.`); + } + this.boardConfig = boardConfig; + } + + async buildForCheck(): Promise { + return this.buildProject({ memoryLayout: DUMMY_MEMORY_LAYOUT }); + } + + async buildProject(context?: CompileContext): Promise { + const memoryLayout = context?.memoryLayout; + if (!memoryLayout) { + throw new Error('Memory layout is required to build an ESP32 project.'); + } + const project = ProjectForEsp32.load( + this.projectConfigHandler.getConfig().projectName, + createEsp32PackageReader(this.boardName, this.projectConfigHandler), + ); + const toolchain = new Esp32Toolchain(this.getCompilerConfig(), memoryLayout); + this.compiler = new CompilerSession(toolchain); + return this.compiler.buildProject(project); + } + + async compileFragment(src: string): Promise { + if (!this.compiler) { + throw new Error("Cannot compile fragment before building the project."); + } + return this.compiler.compileFragment(src); + } + + private getCompilerConfig(): Esp32ToolchainConfig { + const runtimeDir = this.projectConfigHandler.getConfig().runtimeDir + ?? this.globalConfigHandler.getConfig().runtimeDir; + if (!runtimeDir) { + throw new Error('An unexpected error occurred: cannot find runtime directory path.'); + } + return { + runtimeDir, + compilerToolchainDir: this.boardConfig.xtensaGccDir, + espDir: this.boardConfig.rootDir, + }; + } +} + +export function createEsp32PackageReader( + _boardName: BoardName, + projectConfigHandler: ProjectConfigHandler, +): (name: string) => PackageForEsp32 { + return (name: string) => { + const mainRoot = projectConfigHandler.root; + const subPackageRoot = path.join(mainRoot, PROJECT_DEFAULT_PATHS.PACKAGES_DIR, name); + const isMain = name === projectConfigHandler.getConfig().projectName; + const root = isMain ? mainRoot : subPackageRoot; + try { + const configHandler = isMain + ? projectConfigHandler.asBoard('esp32') + : ProjectConfigHandler.load(root).asBoard('esp32'); + return new PackageForEsp32( + name, + { + rootDir: root, + entry: configHandler.entryFile ?? PROJECT_DEFAULT_PATHS.ENTRY_FILE, + sourceDir: configHandler.srcDir ?? PROJECT_DEFAULT_PATHS.SRC_DIR, + distDir: PROJECT_DEFAULT_PATHS.DIST_DIR, + buildDir: PROJECT_DEFAULT_PATHS.BUILD_DIR, + packageDir: PROJECT_DEFAULT_PATHS.PACKAGES_DIR, + }, + Object.keys(configHandler.dependencies), + configHandler.espIdfComponents, + ); + } catch (error) { + throw new Error(`Failed to read ${name}.`, { cause: error }); + } + }; +} \ No newline at end of file diff --git a/cli/src/platforms/compiler/host-compiler-adapter.ts b/cli/src/platforms/compiler/host-compiler-adapter.ts new file mode 100644 index 00000000..75a886b4 --- /dev/null +++ b/cli/src/platforms/compiler/host-compiler-adapter.ts @@ -0,0 +1,85 @@ +import { GlobalConfigHandler } from "../../config/global-config"; +import { ProjectConfigHandler, PROJECT_DEFAULT_PATHS } from "../../config/project-config"; +import { BoardName } from "../../config/board-utils"; +import { + CompilerSession, SharedObject, + HostToolchain, ProjectForHost, Package +} from "@bscript/lang"; +import { CompilerAdapter, CompileContext } from "./compiler-adapter"; +import * as path from 'path'; + + +export class HostCompilerAdapter implements CompilerAdapter { + readonly boardName: BoardName = 'host'; + private compiler?: CompilerSession; + + constructor( + private globalConfigHandler: GlobalConfigHandler, + private projectConfigHandler: ProjectConfigHandler, + ) { + if (!this.globalConfigHandler.isBoardSetup(this.boardName)) { + throw new Error(`The environment for ${this.boardName} is not set up.`); + } + } + + async buildForCheck(): Promise { + return this.buildProject(); + } + + async buildProject(_context?: CompileContext): Promise { + const project = ProjectForHost.load( + this.projectConfigHandler.getConfig().projectName, + createHostPackageReader(this.boardName, this.projectConfigHandler), + ); + const toolchain = new HostToolchain(this.getRuntimeDir()); + this.compiler = new CompilerSession(toolchain); + return this.compiler.buildProject(project); + } + + async compileFragment(src: string): Promise { + if (!this.compiler) { + throw new Error("Cannot compile fragment before building the project."); + } + return this.compiler.compileFragment(src); + } + + private getRuntimeDir(): string { + const runtimeDir = this.projectConfigHandler.getConfig().runtimeDir + ?? this.globalConfigHandler.getConfig().runtimeDir; + if (!runtimeDir) { + throw new Error('An unexpected error occurred: cannot find runtime directory path.'); + } + return runtimeDir; + } +} + +export function createHostPackageReader( + _boardName: BoardName, + projectConfigHandler: ProjectConfigHandler, +): (name: string) => Package { + return (name: string) => { + const mainRoot = projectConfigHandler.root; + const subPackageRoot = path.join(mainRoot, PROJECT_DEFAULT_PATHS.PACKAGES_DIR, name); + const isMain = name === projectConfigHandler.getConfig().projectName; + const root = isMain ? mainRoot : subPackageRoot; + try { + const configHandler = isMain + ? projectConfigHandler.asBoard('host') + : ProjectConfigHandler.load(root).asBoard('host'); + return new Package( + name, + { + rootDir: root, + entry: configHandler.entryFile ?? PROJECT_DEFAULT_PATHS.ENTRY_FILE, + sourceDir: configHandler.srcDir ?? PROJECT_DEFAULT_PATHS.SRC_DIR, + distDir: PROJECT_DEFAULT_PATHS.DIST_DIR, + buildDir: PROJECT_DEFAULT_PATHS.BUILD_DIR, + packageDir: PROJECT_DEFAULT_PATHS.PACKAGES_DIR, + }, + Object.keys(configHandler.dependencies), + ); + } catch (error) { + throw new Error(`Failed to read ${name}.`, { cause: error }); + } + }; +} diff --git a/cli/src/platforms/index.ts b/cli/src/platforms/index.ts new file mode 100644 index 00000000..942dd473 --- /dev/null +++ b/cli/src/platforms/index.ts @@ -0,0 +1,62 @@ +import { GlobalConfigHandler } from "../config/global-config"; +import { ProjectConfigHandler } from "../config/project-config"; +import { BoardName } from "../config/board-utils"; +import { ProgramOutput } from "../core/logger/program-output"; +import { CompilerAdapter } from "./compiler/compiler-adapter"; +import { Esp32CompilerAdapter } from "./compiler/esp32-compiler-adapter"; +import { HostCompilerAdapter } from "./compiler/host-compiler-adapter"; +import { BoardRuntime } from "./runtime/board-runtime"; +import { Esp32BoardRuntime } from "./runtime/esp32-board-runtime"; +import { HostBoardRuntime } from "./runtime/host-board-runtime"; + +export { CompilerAdapter, CompileContext } from "./compiler/compiler-adapter"; +export { BoardRuntime } from "./runtime/board-runtime"; + + +export function getCompilerAdapter( + boardName: BoardName, + globalConfigHandler: GlobalConfigHandler, + projectConfigHandler: ProjectConfigHandler, +): CompilerAdapter { + if (boardName === 'esp32') { + return new Esp32CompilerAdapter(globalConfigHandler, projectConfigHandler); + } + if (boardName === 'host') { + return new HostCompilerAdapter(globalConfigHandler, projectConfigHandler); + } + throw new Error(`Unsupported board name: ${boardName}`); +} + +export function getBoardRuntime( + boardName: BoardName, + globalConfigHandler: GlobalConfigHandler, + deviceName: string, + programOutput: ProgramOutput, + onUnexpectedDisconnect?: () => void, +): BoardRuntime { + if (boardName === 'esp32') { + return new Esp32BoardRuntime(deviceName, programOutput, onUnexpectedDisconnect); + } + if (boardName === 'host') { + const boardConfig = globalConfigHandler.getBoardConfig('host'); + if (!boardConfig) { + throw new Error('The environment for host is not set up.'); + } + return new HostBoardRuntime(boardConfig, programOutput, onUnexpectedDisconnect); + } + throw new Error(`Unsupported board name: ${boardName}`); +} + +export function createPlatformSession( + boardName: BoardName, + globalConfigHandler: GlobalConfigHandler, + projectConfigHandler: ProjectConfigHandler, + deviceName: string, + programOutput: ProgramOutput, + onUnexpectedDisconnect?: () => void, +): { compiler: CompilerAdapter; runtime: BoardRuntime } { + return { + compiler: getCompilerAdapter(boardName, globalConfigHandler, projectConfigHandler), + runtime: getBoardRuntime(boardName, globalConfigHandler, deviceName, programOutput, onUnexpectedDisconnect), + }; +} diff --git a/cli/src/platforms/runtime/board-runtime.ts b/cli/src/platforms/runtime/board-runtime.ts new file mode 100644 index 00000000..2d163324 --- /dev/null +++ b/cli/src/platforms/runtime/board-runtime.ts @@ -0,0 +1,12 @@ +import { CompileOutput } from "@bscript/lang"; +import { ProgramOutput } from "../../core/logger/program-output"; +import { CompileContext } from "../compiler/compiler-adapter"; + +export interface BoardRuntime { + connect(): Promise; + disconnect(): Promise; + prepare(): Promise; + load(output: Output): Promise; + execute(output: Output): Promise; + setOutput(output: ProgramOutput): void; +} diff --git a/cli/src/services/device-manager.ts b/cli/src/platforms/runtime/esp32-board-runtime.ts similarity index 52% rename from cli/src/services/device-manager.ts rename to cli/src/platforms/runtime/esp32-board-runtime.ts index d76f1e29..8878ee9e 100644 --- a/cli/src/services/device-manager.ts +++ b/cli/src/platforms/runtime/esp32-board-runtime.ts @@ -1,25 +1,27 @@ -import { BleConnection, DeviceService } from "./ble"; -import { ExecutableBinary, MemoryLayout } from "@bscript/lang"; +import { BleConnection, DeviceService } from "../../services/ble"; +import { MemoryImage } from "@bscript/lang"; +import { ProgramOutput } from "../../core/logger/program-output"; +import { BoardRuntime } from "./board-runtime"; +import { CompileContext } from "../compiler/compiler-adapter"; -export interface DeviceLogger { - log(message: string): void; - error(message: string): void; -} -export class BleDeviceManager { +export class Esp32BoardRuntime implements BoardRuntime { private ble: BleConnection | null = null; private deviceService: DeviceService | null = null; + private programOutput: ProgramOutput; constructor( private deviceName: string, - private deviceLogger: DeviceLogger, - private onUnexpectedDisconnect?: () => void - ) {} + programOutput: ProgramOutput, + private onUnexpectedDisconnect?: () => void, + ) { + this.programOutput = programOutput; + } async connect(): Promise { this.ble = new BleConnection(this.deviceName); await this.ble.connect(); - + this.ble.on('disconnected', () => { if (this.ble?.status !== 'disconnecting') { if (this.onUnexpectedDisconnect) { @@ -33,8 +35,8 @@ export class BleDeviceManager { }); this.deviceService = this.ble.getService('device'); - this.deviceService.on('log', (message) => this.deviceLogger.log(message)); - this.deviceService.on('error', (message) => this.deviceLogger.error(message)); + this.deviceService.on('log', (message) => this.programOutput.write(message)); + this.deviceService.on('error', (message) => this.programOutput.writeError(message)); } async disconnect(): Promise { @@ -43,28 +45,29 @@ export class BleDeviceManager { } } - async initDevice(): Promise { + async prepare(): Promise { if (!this.ble || !this.deviceService) { throw new Error('Failed to initialize device. BLE is not connected.'); } - return this.deviceService.init(); + const memoryLayout = await this.deviceService.init(); + return { memoryLayout }; } - async load(bin: ExecutableBinary): Promise { + async load(output: MemoryImage): Promise { if (!this.ble || !this.deviceService) { throw new Error('Failed to load binary. BLE is not connected.'); } - return this.deviceService.load(bin); + return this.deviceService.load(output); } - async execute(bin: ExecutableBinary): Promise { + async execute(output: MemoryImage): Promise { if (!this.ble || !this.deviceService) { throw new Error('Failed to execute binary. BLE is not connected.'); } - return this.deviceService.execute(bin); + return this.deviceService.execute(output); } - updateLogger(logger: DeviceLogger) { - this.deviceLogger = logger; + setOutput(output: ProgramOutput): void { + this.programOutput = output; } } diff --git a/cli/src/platforms/runtime/host-board-runtime.ts b/cli/src/platforms/runtime/host-board-runtime.ts new file mode 100644 index 00000000..5382b7ba --- /dev/null +++ b/cli/src/platforms/runtime/host-board-runtime.ts @@ -0,0 +1,93 @@ +import * as path from 'path'; +import { exec } from '../../core/shell'; +import { SharedObject } from "@bscript/lang"; +import { ProgramOutput } from "../../core/logger/program-output"; +import { BoardRuntime } from "./board-runtime"; +import { CompileContext } from "../compiler/compiler-adapter"; +import { HostBoardConfig } from "../../config/global-config"; +import * as fs from '../../core/fs'; +import { HostService, ProcessConnection } from '../../services/process'; + + +export async function buildHostRuntime(runtimeDir: string, buildDir?: string): Promise { + const resolvedBuildDir = buildDir ?? path.join(runtimeDir, 'ports/host/build'); + const builtinModuleC = path.join(runtimeDir, 'ports/host/std-module.c'); + const shellC = path.join(runtimeDir, 'ports/host/shell.c'); + const runtimeC = path.join(runtimeDir, 'core/src/c-runtime.c'); + const commC = path.join(runtimeDir, 'ports/host/comm.c'); + const runtimeSo = path.join(resolvedBuildDir, 'c-runtime.so'); + const shell = path.join(resolvedBuildDir, 'shell'); + + fs.makeDir(resolvedBuildDir); + + await exec( + `cc -DLINUX64 -O2 -shared -fPIC -o "${runtimeSo}" "${runtimeC}" "${builtinModuleC}" "${commC}"`, + { silent: true }, + ); + await exec( + `cc -DLINUX64 -O2 -o "${shell}" "${shellC}" "${runtimeSo}" -lm -ldl`, + { silent: true }, + ); + + return resolvedBuildDir; +} + + +export class HostBoardRuntime implements BoardRuntime { + private programOutput: ProgramOutput; + private shellProcess: ProcessConnection; + private hostService: HostService; + + constructor( + private boardConfig: HostBoardConfig, + programOutput: ProgramOutput, + private onUnexpectedDisconnect?: () => void, + ) { + this.programOutput = programOutput; + this.shellProcess = new ProcessConnection(this.getShellPath()); + this.shellProcess.on('disconnected', (code) => { + if (code !== 0) { + this.onUnexpectedDisconnect?.(); + } + }); + this.hostService = this.shellProcess.getService('host'); + } + + async connect(): Promise { + await this.shellProcess.connect(); + this.hostService.on('log', (message) => { + this.programOutput.write(message); + }); + this.hostService.on('error', (message) => { + this.programOutput.writeError(message); + }); + } + + async disconnect(): Promise { + await this.shellProcess.disconnect(); + } + + async prepare(): Promise { + return {}; + } + + async load(output: SharedObject): Promise { + return this.hostService.load(output.soFile); + } + + async execute(output: SharedObject): Promise { + let exectime = 0; + for (const entry of output.entryNames) { + exectime += await this.hostService.execute(entry.name); + } + return exectime; + } + + setOutput(output: ProgramOutput): void { + this.programOutput = output; + } + + private getShellPath(): string { + return path.join(this.boardConfig.buildDir, 'shell'); + } +} diff --git a/cli/src/services/ble.ts b/cli/src/services/ble.ts index adc54489..3f264130 100644 --- a/cli/src/services/ble.ts +++ b/cli/src/services/ble.ts @@ -1,6 +1,7 @@ import noble, { Characteristic, Peripheral } from '@abandonware/noble'; import { Buffer } from "node:buffer"; -import { ExecutableBinary, MemoryLayout } from "@bscript/lang"; +import { MemoryImage, MemoryLayout } from "@bscript/lang"; +import { logger } from "../core/logger"; import { Connection, ConnectionMessage, Service } from "./common"; import { Protocol, ProtocolPacketBuilder, ProtocolParser } from './device-protocol'; @@ -21,9 +22,12 @@ export type DeviceServiceEvents = { export class DeviceService extends Service { constructor(connection: BleConnection) { super('device', connection); + this.connection.on('receiveData', data => { + this.handleReceivedData(data); + }) } - public async load(bin: ExecutableBinary): Promise { + public async load(bin: MemoryImage): Promise { const builder = new ProtocolPacketBuilder(MTU); if (bin.iram) builder.load(bin.iram.address, bin.iram.data); if (bin.dram) builder.load(bin.dram.address, bin.dram.data); @@ -34,7 +38,7 @@ export class DeviceService extends Service { return performance.now() - startLoading; } - public async execute(bin: ExecutableBinary): Promise { + public async execute(bin: MemoryImage): Promise { const builder = new ProtocolPacketBuilder(MTU); const isMain = 1; for (const entryPoint of bin.entryPoints) { @@ -62,6 +66,26 @@ export class DeviceService extends Service { }); }); } + + private handleReceivedData(data: Buffer) { + const parseResult = new ProtocolParser().parse(data); + switch(parseResult.protocol) { + case Protocol.Log: + this.handleMessage('log', [parseResult.log]); + break; + case Protocol.Error: + this.handleMessage('error', [parseResult.error]); + break; + case Protocol.Profile: + this.handleMessage('profile', [parseResult.fid, parseResult.paramtypes]); + break; + case Protocol.Exectime: + this.handleMessage('exectime', [parseResult.id, parseResult.time]); + break; + case Protocol.Memory: + this.handleMessage('memory', [parseResult.layout]); + } + } } @@ -133,7 +157,7 @@ export class BleConnection extends Connection { this.characteristic = characteristics[0]; this.characteristic.on('data', (data, isNotification) => { if (isNotification) { - this.handleReceivedData(data); + this.emit('receiveData', data); } }) await this.characteristic.subscribeAsync(); @@ -154,28 +178,6 @@ export class BleConnection extends Connection { }); } - private handleReceivedData(data: Buffer) { - const service = this.services.get('device'); - if (!service) { return } - const parseResult = new ProtocolParser().parse(data); - switch(parseResult.protocol) { - case Protocol.Log: - service.handleMessage('log', [parseResult.log]); - break; - case Protocol.Error: - service.handleMessage('error', [parseResult.error]); - break; - case Protocol.Profile: - service.handleMessage('profile', [parseResult.fid, parseResult.paramtypes]); - break; - case Protocol.Exectime: - service.handleMessage('exectime', [parseResult.id, parseResult.time]); - break; - case Protocol.Memory: - service.handleMessage('memory', [parseResult.layout]); - } - } - public async disconnect(): Promise { if (this.characteristic) { await this.characteristic.unsubscribeAsync(); @@ -194,7 +196,7 @@ export class BleConnection extends Connection { await this.characteristic.writeAsync(buff, false); } } else { - console.error("BLE is not connected."); + logger.error("BLE is not connected."); } } diff --git a/cli/src/services/common.ts b/cli/src/services/common.ts index 5b3f46eb..e87a5719 100644 --- a/cli/src/services/common.ts +++ b/cli/src/services/common.ts @@ -82,13 +82,15 @@ export abstract class Service extends EventEmitter< } } -export type ConnectionEvents = { +export type ConnectionEvents = { connected: () => void; disconnected: (event: any) => void; error: (error: Error) => void; + receiveData: (data: T) => void; + receiveError: (error: T) => void; } -export abstract class Connection extends EventEmitter { +export abstract class Connection extends EventEmitter> { abstract send(message: ConnectionMessage): Promise; abstract getService>(serviceName: string): K; } \ No newline at end of file diff --git a/cli/src/services/host-protocol.ts b/cli/src/services/host-protocol.ts new file mode 100644 index 00000000..d76da443 --- /dev/null +++ b/cli/src/services/host-protocol.ts @@ -0,0 +1,101 @@ +export enum HostProtocol { + None = 0, + Load = 1, + Call = 2, + + Log = 3, + Error = 4, + Exectime = 5, + Loadtime = 6, + Max +} + +export function hostProtocolBuilder(protocol: HostProtocol, payload: string) { + const protocolStr = String(protocol).padStart(2, '0'); + const payloadLen = String(payload.length).padStart(4, '0'); + return `${protocolStr} ${payloadLen} ${payload}\n`; +} + + +type HostProtocolPayloads = { + [HostProtocol.None]: {}; + [HostProtocol.Load]: {}; + [HostProtocol.Call]: {}; + [HostProtocol.Log]: { log: string }; + [HostProtocol.Error]: { error: string }; + [HostProtocol.Exectime]: { time: number }; + [HostProtocol.Loadtime]: { time: number }; + [HostProtocol.Max]: {}; +} + +export type HostParseResult = { + [K in T]: { protocol: K } & HostProtocolPayloads[K] +}[T]; + +type HostParserFunction = (payload: string) => HostProtocolPayloads[K]; + +export class HostProtocolParser { + private readonly parsers: {[K in HostProtocol]?: HostParserFunction}; + + constructor() { + this.parsers = { + [HostProtocol.Log]: HostProtocolParser.parseLog, + [HostProtocol.Error]: HostProtocolParser.parseError, + [HostProtocol.Exectime]: HostProtocolParser.parseExectime, + [HostProtocol.Loadtime]: HostProtocolParser.parseLoadtime, + } + } + + public parse(line: string): {parsed: HostParseResult[], remain: string} { + // The format is [xx yyyy zz...] + // xx is protocol, yyyy is payload length, zz... is payload + const headerLength = 8; + const parsed: HostParseResult[] = []; + let remain: string = line; + while (remain.length >= headerLength) { + try { + const protocol = Number(remain.substring(0, 2)); + const payloadLength = Number(remain.substring(3, 7)); + if (remain.length < headerLength + payloadLength) { + return { parsed, remain }; + } + const payload = remain.substring(headerLength, headerLength + payloadLength); + remain = remain.substring(headerLength + payloadLength); + parsed.push(this.parsePayload(protocol, payload)); + } catch (error) { + throw new Error("Failed to parse message.", { cause: error }); + } + + } + return { parsed, remain }; + } + + private parsePayload(protocol: number, payload: string): HostParseResult { + if (!this.isParseableProtocol(protocol)) { + throw new Error(`Failed to parse buffer. The protocol ${protocol} is not parsable.`); + } + const parser = this.parsers[protocol]!; + const parsedPayload = parser(payload); + return {protocol, ...parsedPayload} as HostParseResult; + } + + private isParseableProtocol(value: number): value is keyof typeof this.parsers { + return value in this.parsers; + } + + static parseLog(payload: string): { log: string } { + return { log: payload }; + } + + static parseError(payload: string): { error: string } { + return { error: payload }; + } + + static parseExectime(payload: string): { time: number } { + return { time: Number(payload) }; + } + + static parseLoadtime(payload: string): { time: number } { + return { time: Number(payload) }; + } +} diff --git a/cli/src/services/process.ts b/cli/src/services/process.ts new file mode 100644 index 00000000..c6de8458 --- /dev/null +++ b/cli/src/services/process.ts @@ -0,0 +1,215 @@ +import { logger } from "../core/logger"; +import { Connection, ConnectionMessage, Service } from "./common"; +import { hostProtocolBuilder, HostProtocolParser, HostProtocol, HostParseResult } from "./host-protocol"; +import { ChildProcessWithoutNullStreams, spawn } from 'node:child_process'; + + +export type HostServiceEvents = { + log: (message: string) => void; + error: (message: string) => void; + exectime: (time: number) => void; + loadtime: (time: number) => void; +} + +export class HostService extends Service { + messageQueue: HostMessageQueue; + + constructor(connection: ProcessConnection) { + super('host', connection); + this.messageQueue = new HostMessageQueue(this); + this.connection.on('receiveData', (message) => { + this.messageQueue.addChunk(message); + }); + this.connection.on('receiveError', (message) => { + this.handleMessage('error', [message]); + }); + } + + public async load(soFile: string): Promise { + const line = hostProtocolBuilder(HostProtocol.Load, soFile); + await this.send('load', [line]); + return new Promise((resolve) => { + this.on('loadtime', (time) => { + resolve(time); + this.off('loadtime'); + }); + }); + } + + public async execute(entryPointName: string): Promise { + const line = hostProtocolBuilder(HostProtocol.Call, entryPointName); + await this.send('execute', [line]); + return new Promise((resolve) => { + this.on('exectime', (time) => { + resolve(time); + this.off('exectime'); + }); + }); + } +} + + +class HostMessageQueue { + service: HostService; + private queue: HostParseResult[]; + private incompleteChunk: string; + private parser: HostProtocolParser; + + constructor(service: HostService) { + this.service = service; + this.queue = []; + this.incompleteChunk = ""; + this.parser = new HostProtocolParser(); + } + + addChunk(chunk: string) { + this.incompleteChunk += chunk; + const { parsed, remain } = this.parser.parse(this.incompleteChunk); + this.incompleteChunk = remain; + this.queue = this.queue.concat(parsed); + queueMicrotask(() => { + this.publish(); + }); + } + + publish() { + if (this.queue.length > 0) { + const message = this.queue.shift()!; + switch(message.protocol) { + case HostProtocol.Log: + this.service.handleMessage('log', [message.log]); + break; + case HostProtocol.Error: + this.service.handleMessage('error', [message.error]); + break; + case HostProtocol.Exectime: + this.service.handleMessage('exectime', [message.time]); + break; + case HostProtocol.Loadtime: + this.service.handleMessage('loadtime', [message.time]); + break; + default: + throw new Error("Unexpected error."); + } + } + if (this.queue.length > 0) { + queueMicrotask(() => { + this.publish(); + }); + } + } +} + + +export class ProcessConnection extends Connection { + private shellFile: string; + private services = new Map>(); + private shellProcess: ChildProcessWithoutNullStreams | null = null; + private disconnecting = false; + + + constructor(shellFile: string) { + super(); + this.shellFile = shellFile; + } + + public async connect(): Promise { + this.shellProcess = spawn(this.shellFile); + this.shellProcess.on('exit', (code) => { + if (this.disconnecting) { + return; + } + if (code === 0) { + this.emit('disconnected', 0); + } else { + this.emit('disconnected', 1); + } + }); + + this.shellProcess.stdout.setEncoding('utf8'); + this.shellProcess.stderr.setEncoding('utf8'); + this.shellProcess.stdout.on('data', (message) => { + this.emit('receiveData', message); + }); + this.shellProcess.stderr.on('data', (message) => { + this.emit('receiveError', message); + }); + this.emit('connected'); + } + + public async disconnect(): Promise { + if (!this.checkProcessRunning(this.shellProcess)) { + return; + } + + const proc = this.shellProcess; + this.shellProcess = null; + this.disconnecting = true; + + await new Promise((resolve) => { + const timeout = setTimeout(() => { + proc.kill('SIGKILL'); + proc.stdout.destroy(); + proc.stderr.destroy(); + proc.stdin.destroy(); + resolve(); + }, 3_000); + timeout.unref(); + + proc.once('exit', () => { + clearTimeout(timeout); + proc.stdout.destroy(); + proc.stderr.destroy(); + proc.stdin.destroy(); + resolve(); + }); + + proc.stdout.removeAllListeners('data'); + proc.stderr.removeAllListeners('data'); + proc.stdin.end(); + proc.kill(); + }); + + this.emit('disconnected', 0); + this.disconnecting = false; + } + + public async send(message: ConnectionMessage): Promise { + if (this.checkProcessRunning(this.shellProcess)) { + for (const line of message.payload) { + this.shellProcess.stdin.cork(); + this.shellProcess.stdin.write(line); + process.nextTick(() => this.shellProcess?.stdin.uncork()); + } + } + } + + private checkProcessRunning(process: ChildProcessWithoutNullStreams | null): process is ChildProcessWithoutNullStreams { + if (process) { + return true; + } else { + logger.error("The process is not running."); + return false; + } + } + + public getService(serviceName: 'host'): HostService; + public getService>(serviceName: string): T; + public getService(serviceName: string): Service { + if (this.services.has(serviceName)) { + return this.services.get(serviceName)!; + } + + let service: Service; + switch (serviceName) { + case 'host': + service = new HostService(this); + break; + default: + throw new Error(`Unknown service: ${serviceName}`); + } + + this.services.set(serviceName, service); + return service; + } +} diff --git a/cli/src/services/websocket.ts b/cli/src/services/websocket.ts index 4c6f0dc4..109288a7 100644 --- a/cli/src/services/websocket.ts +++ b/cli/src/services/websocket.ts @@ -1,4 +1,5 @@ import { Connection, ConnectionMessage, EventMap, Service } from "./common"; +import { logger } from "../core/logger"; import { WebSocketServer, WebSocket } from 'ws'; export type ReplServiceEvents = { @@ -56,10 +57,10 @@ export class WebSocketConnection extends Connection { if (service) { service.handleMessage(parsedMessage.event, parsedMessage.payload); } else { - console.warn(`Service "${parsedMessage.service}" not found.`); + logger.warn(`Service "${parsedMessage.service}" not found.`); } } catch (error) { - console.error("Failed to parse message:", message, error); + logger.error("Failed to parse message:", String(message), String(error)); } }); @@ -76,16 +77,15 @@ export class WebSocketConnection extends Connection { } public close(): void { - if(this.server) { - this.server.close(); - } + this.server?.close(); + this.client?.close(); } public async send(message: ConnectionMessage): Promise { if (this.client && this.client.readyState === WebSocket.OPEN) { this.client.send(JSON.stringify(message)); } else { - console.error("WebSocket is not connected."); + logger.error("WebSocket is not connected."); } } diff --git a/cli/tests/commands/board/flash-runtime.test.ts b/cli/tests/commands/board/flash-runtime.test.ts index ff94ebc1..92cf6001 100644 --- a/cli/tests/commands/board/flash-runtime.test.ts +++ b/cli/tests/commands/board/flash-runtime.test.ts @@ -1,10 +1,9 @@ import { handleFlashRuntimeCommand } from '../../../src/commands/board/flash-runtime'; import { SerialPort } from 'serialport'; -import { deleteGlobalEnv, setupDefaultGlobalEnv, setupGlobalEnvWithEsp32, spyGlobalSettings } from '../global-env-helper'; +import { deleteGlobalEnv, setupDefaultGlobalEnv, setupGlobalEnvWithEsp32, setupGlobalEnvWithHost, spyGlobalSettings } from '../global-env-helper'; import { mockedInquirer, mockedLogger, - mockedShowErrorMessages, mockProcessExit, mockedExec, } from '../mock-helpers'; @@ -84,7 +83,7 @@ describe('board flash-runtime command', () => { // --- Assert --- expect(mockedLogger.error).toHaveBeenCalledWith('Failed to flash the runtime to unknown-board'); - expect(mockedShowErrorMessages).toHaveBeenCalledWith(new Error('Unsupported board name: unknown-board')); + expect(mockedLogger.showError).toHaveBeenCalledWith(new Error('Unsupported board name: unknown-board')); expect(process.exit).toHaveBeenCalledWith(1); // --- Clean up --- @@ -123,4 +122,20 @@ describe('board flash-runtime command', () => { expect(mockedExec).not.toHaveBeenCalled(); }); }); + + describe('for host board', () => { + it('should exit with an error because flash-runtime is not supported', async () => { + setupGlobalEnvWithHost(); + const exitSpy = mockProcessExit(); + + await handleFlashRuntimeCommand('host', {}); + + expect(mockedLogger.error).toHaveBeenCalledWith('Failed to flash the runtime to host'); + expect(mockedLogger.showError).toHaveBeenCalledWith( + new Error('flash-runtime is not supported for the host board'), + ); + expect(process.exit).toHaveBeenCalledWith(1); + exitSpy.mockRestore(); + }); + }); }); \ No newline at end of file diff --git a/cli/tests/commands/board/remove.test.ts b/cli/tests/commands/board/remove.test.ts index 9c1517f4..69b48770 100644 --- a/cli/tests/commands/board/remove.test.ts +++ b/cli/tests/commands/board/remove.test.ts @@ -1,9 +1,8 @@ import { handleRemoveCommand } from '../../../src/commands/board/remove'; -import { setupDefaultGlobalEnv, deleteGlobalEnv, setupGlobalEnvWithEsp32, getGlobalConfig, spyGlobalSettings } from '../global-env-helper'; +import { setupDefaultGlobalEnv, deleteGlobalEnv, setupGlobalEnvWithEsp32, setupGlobalEnvWithHost, getGlobalConfig, spyGlobalSettings } from '../global-env-helper'; import { mockedInquirer, mockedLogger, - mockedShowErrorMessages, mockProcessExit, } from '../mock-helpers'; @@ -68,7 +67,7 @@ describe('board remove command', () => { // --- Assert --- expect(mockedLogger.error).toHaveBeenCalledWith('Failed to remove unknown-board'); - expect(mockedShowErrorMessages).toHaveBeenCalledWith(new Error('Unsupported board name: unknown-board')); + expect(mockedLogger.showError).toHaveBeenCalledWith(new Error('Unsupported board name: unknown-board')); expect(process.exit).toHaveBeenCalledWith(1); exitSpy.mockRestore(); }); @@ -101,4 +100,25 @@ describe('board remove command', () => { expect(mockedInquirer.prompt).not.toHaveBeenCalled(); }); }); + + describe('for host board', () => { + it('should perform removal if setup for host exists', async () => { + setupGlobalEnvWithHost(); + mockedInquirer.prompt.mockResolvedValue({ proceed: true }); + + await handleRemoveCommand('host', {}); + + expect(Object.keys(getGlobalConfig().boards)).not.toContain('host'); + expect(mockedLogger.error).not.toHaveBeenCalled(); + }); + + it('should warn and exit if setup is not completed', async () => { + setupDefaultGlobalEnv(); + + await handleRemoveCommand('host', {}); + + expect(mockedLogger.warn).toHaveBeenCalledWith('The environment for host is not set up. Nothing to remove.'); + expect(mockedInquirer.prompt).not.toHaveBeenCalled(); + }); + }); }); \ No newline at end of file diff --git a/cli/tests/commands/board/setup.test.ts b/cli/tests/commands/board/setup.test.ts index 7f1ac73f..a0f02a61 100644 --- a/cli/tests/commands/board/setup.test.ts +++ b/cli/tests/commands/board/setup.test.ts @@ -5,10 +5,17 @@ import { mockedExec, mockedInquirer, mockedLogger, - mockedShowErrorMessages, mockProcessExit, } from '../mock-helpers'; -import { deleteGlobalEnv, getGlobalConfig, setupDefaultGlobalEnv, setupEmpyGlobalEnv, setupGlobalEnvWithEsp32, spyGlobalSettings } from '../global-env-helper'; +import { deleteGlobalEnv, getGlobalConfig, setupDefaultGlobalEnv, setupEmpyGlobalEnv, setupGlobalEnvWithEsp32, setupGlobalEnvWithHost, spyGlobalSettings } from '../global-env-helper'; + +jest.mock('../../../src/platforms/runtime/host-board-runtime', () => ({ + buildHostRuntime: jest.fn().mockResolvedValue('/mock/host/build'), + getHostBuildDir: jest.fn(), +})); + +import { buildHostRuntime } from '../../../src/platforms/runtime/host-board-runtime'; +const mockedBuildHostRuntime = buildHostRuntime as jest.Mock; jest.mock('os', () => ({ ...jest.requireActual('os'), @@ -67,7 +74,7 @@ describe('board setup command', () => { // --- Assert --- expect(mockedLogger.error).toHaveBeenCalledWith('Failed to set up unknown-board'); - expect(mockedShowErrorMessages).toHaveBeenCalledWith(new Error('Unsupported board name: unknown-board')); + expect(mockedLogger.showError).toHaveBeenCalledWith(new Error('Unsupported board name: unknown-board')); expect(process.exit).toHaveBeenCalledWith(1); exitSpy.mockRestore(); }); @@ -88,7 +95,7 @@ describe('board setup command', () => { // --- Assert --- expect(mockedLogger.error).toHaveBeenCalledWith('Failed to set up esp32'); - expect(mockedShowErrorMessages).toHaveBeenCalledWith(expect.any(Error)); + expect(mockedLogger.showError).toHaveBeenCalledWith(expect.any(Error)); expect(process.exit).toHaveBeenCalledWith(1); exitSpy.mockRestore(); }); @@ -228,9 +235,67 @@ describe('board setup command', () => { // --- Assert --- expect(mockedLogger.error).toHaveBeenCalledWith('Failed to set up esp32'); - expect(mockedShowErrorMessages).toHaveBeenCalledWith(new Error('Unsupported OS.')); + expect(mockedLogger.showError).toHaveBeenCalledWith(new Error('Unsupported OS.')); expect(process.exit).toHaveBeenCalledWith(1); exitSpy.mockRestore(); }); }); + + describe('for host board on macOS', () => { + beforeEach(() => { + mockedOs.platform.mockReturnValue('darwin'); + }); + + it('should perform a full setup if not already set up', async () => { + setupEmpyGlobalEnv(); + mockedInquirer.prompt.mockResolvedValue({ proceed: true }); + mockedExec.mockImplementation(async (command: string) => { + if (command.startsWith('which')) { + return ''; + } + return ''; + }); + + await handleSetupCommand('host'); + + expect(mockedInquirer.prompt).toHaveBeenCalledTimes(1); + expect(mockedDownloadAndUnzip).toHaveBeenCalledTimes(1); + expect(mockedBuildHostRuntime).toHaveBeenCalledTimes(1); + expect(getGlobalConfig().boards.host).toEqual({ buildDir: '/mock/host/build' }); + expect(mockedLogger.info).toHaveBeenCalledWith(expect.stringContaining('bscript project create')); + expect(mockedLogger.info).not.toHaveBeenCalledWith(expect.stringContaining('flash-runtime')); + expect(mockedLogger.error).not.toHaveBeenCalled(); + }); + + it('should exit with an error when cc is missing', async () => { + setupEmpyGlobalEnv(); + const exitSpy = mockProcessExit(); + mockedInquirer.prompt.mockResolvedValue({ proceed: true }); + mockedExec.mockImplementation(async (command: string) => { + if (command.includes('which cc')) { + throw new Error('not found'); + } + if (command.startsWith('which')) { + return ''; + } + return ''; + }); + + await handleSetupCommand('host'); + + expect(mockedLogger.error).toHaveBeenCalledWith('Failed to set up host'); + expect(process.exit).toHaveBeenCalledWith(1); + exitSpy.mockRestore(); + }); + + it('should warn and exit if setup is already completed', async () => { + setupGlobalEnvWithHost(); + + await handleSetupCommand('host'); + + expect(mockedLogger.warn).toHaveBeenCalledWith('The setup for host has already been completed.'); + expect(mockedInquirer.prompt).not.toHaveBeenCalled(); + expect(mockedBuildHostRuntime).not.toHaveBeenCalled(); + }); + }); }); \ No newline at end of file diff --git a/cli/tests/commands/global-env-helper.ts b/cli/tests/commands/global-env-helper.ts index 6fe87a53..4e4ef05c 100644 --- a/cli/tests/commands/global-env-helper.ts +++ b/cli/tests/commands/global-env-helper.ts @@ -53,6 +53,33 @@ export function setupDefaultGlobalEnv(isOldVersion = false) { fs.makeDir(GLOBAL_SETTINGS.RUNTIME_DIR); } +export function setupGlobalEnvWithHost(isOldVersion = false, buildDir?: string) { + const resolvedBuildDir = buildDir ?? path.join(GLOBAL_SETTINGS.RUNTIME_DIR, 'ports/host/build'); + setupGlobalEnv({ + version: isOldVersion ? DUMMY_OLD_VM_VERSION : DUMMY_VM_VERSION, + runtimeDir: GLOBAL_SETTINGS.RUNTIME_DIR, + boards: { + host: { + buildDir: resolvedBuildDir, + }, + }, + }); + fs.makeDir(resolvedBuildDir); + fs.makeDir(GLOBAL_SETTINGS.RUNTIME_DIR); +} + +export function setupGlobalEnvWithHostIntegration(runtimeDir: string, buildDir: string) { + setupGlobalEnv({ + version: DUMMY_VM_VERSION, + runtimeDir, + boards: { + host: { + buildDir, + }, + }, + }); +} + export function setupGlobalEnvWithEsp32(isOldVersion = false, isEspIdfOldVersion = false) { setupGlobalEnv({ version: isOldVersion ? DUMMY_OLD_VM_VERSION : DUMMY_VM_VERSION, diff --git a/cli/tests/commands/mock-helpers.ts b/cli/tests/commands/mock-helpers.ts index 3a44ec1e..c7fa4163 100644 --- a/cli/tests/commands/mock-helpers.ts +++ b/cli/tests/commands/mock-helpers.ts @@ -1,6 +1,6 @@ import { exec, cwd } from '../../src/core/shell'; import inquirer from 'inquirer'; -import { logger, showErrorMessages } from '../../src/core/logger'; +import { logger } from '../../src/core/logger'; import { downloadAndUnzip } from '../../src/core/fs'; @@ -9,11 +9,9 @@ export const mockedCwd = cwd as jest.Mock; export const mockedDownloadAndUnzip = downloadAndUnzip as jest.Mock; export const mockedInquirer = inquirer as jest.Mocked; export const mockedLogger = logger as jest.Mocked; -export const mockedShowErrorMessages = showErrorMessages as jest.Mock; - export function mockProcessExit() { return jest .spyOn(process, 'exit') .mockImplementation((() => {}) as (code?: number | string | null | undefined) => never); -} \ No newline at end of file +} diff --git a/cli/tests/commands/project/create.test.ts b/cli/tests/commands/project/create.test.ts index 94975011..7e15eeb6 100644 --- a/cli/tests/commands/project/create.test.ts +++ b/cli/tests/commands/project/create.test.ts @@ -5,7 +5,6 @@ import { mockedCwd, mockedInquirer, mockedLogger, - mockedShowErrorMessages, mockProcessExit, } from '../mock-helpers'; import { deleteGlobalEnv, setupDefaultGlobalEnv, setupGlobalEnvWithEsp32, spyGlobalSettings } from '../global-env-helper'; @@ -94,7 +93,7 @@ describe('create project command', () => { // --- Assert --- expect(mockedLogger.error).toHaveBeenCalledWith('Failed to create a new project.'); - expect(mockedShowErrorMessages).toHaveBeenCalledWith(new Error('The environment for esp32 is not set up.')); + expect(mockedLogger.showError).toHaveBeenCalledWith(new Error('The environment for esp32 is not set up.')); expect(process.exit).toHaveBeenCalled(); // --- Clean up --- @@ -111,7 +110,7 @@ describe('create project command', () => { // --- Assert --- expect(mockedLogger.error).toHaveBeenCalledWith('Failed to create a new project.'); - expect(mockedShowErrorMessages).toHaveBeenCalledWith(new Error('Unsupported board name: unknown-board')); + expect(mockedLogger.showError).toHaveBeenCalledWith(new Error('Unsupported board name: unknown-board')); expect(process.exit).toHaveBeenCalled(); // --- Clean up --- diff --git a/cli/tests/commands/project/install.test.ts b/cli/tests/commands/project/install.test.ts index e95fc3ba..d9e604ed 100644 --- a/cli/tests/commands/project/install.test.ts +++ b/cli/tests/commands/project/install.test.ts @@ -5,7 +5,6 @@ import { mockedCwd, mockedExec, mockedLogger, - mockedShowErrorMessages, mockProcessExit, } from '../mock-helpers'; import { deleteGlobalEnv, setupDefaultGlobalEnv, setupGlobalEnvWithEsp32 } from '../global-env-helper'; @@ -118,7 +117,7 @@ describe('install command', () => { expect(getProjectConfig(projectRoot).dependencies['pkg-led-esp32-project']).toBe('https://github.com/bluescript-lang/pkg-led-esp32.git'); }); - it('should install all packages', async () => { + it('should install all packages from bsconfig', async () => { // --- Arrange --- setupGlobalEnvWithEsp32(); createDummyProject(projectRoot, { @@ -140,7 +139,7 @@ describe('install command', () => { }); // --- Act --- - await handleInstallCommand('https://github.com/bluescript-lang/pkg-led-esp32.git', {}); + await handleInstallCommand(undefined, {}); // --- Assert --- expect(fs.exists(path.join(projectRoot, PROJECT_DEFAULT_PATHS.PACKAGES_DIR, 'pkg-led-esp32-project'))); @@ -164,7 +163,7 @@ describe('install command', () => { // --- Assert --- expect(mockedLogger.error).toHaveBeenCalledWith('Failed to install https://github.com/bluescript-lang/pkg-gpio-esp32.git.'); - expect(mockedShowErrorMessages).toHaveBeenCalledWith(new Error('The environment for esp32 is not set up.')); + expect(mockedLogger.showError).toHaveBeenCalledWith(new Error('The environment for esp32 is not set up.')); expect(process.exit).toHaveBeenCalled(); // --- Clean up --- diff --git a/cli/tests/global-mocks.ts b/cli/tests/global-mocks.ts index 8d78697f..7cb1a027 100644 --- a/cli/tests/global-mocks.ts +++ b/cli/tests/global-mocks.ts @@ -1,28 +1,21 @@ jest.mock('../src/core/logger', () => { - const { SkipStep } = jest.requireActual('../src/core/logger'); - const mockDecorator = jest.fn().mockImplementation( - (message: string) => { - return function ( - target: any, - propertyKey: string, - descriptor: PropertyDescriptor - ) { - const originalMethod = descriptor.value; - descriptor.value = async function (...args: any[]) { - try { - return await originalMethod.apply(this, args); - } catch (error) { - if (error instanceof SkipStep) {return;} - throw error; - } - }; - return descriptor; - }; - } - ) + const actual = jest.requireActual('../src/core/logger/step-runner'); + const { StepSkip } = actual; return { ...jest.requireActual('../src/core/logger'), - LogStep: mockDecorator, + runStep: jest.fn(async (_message: string, action: () => Promise) => { + const result = await action(); + if (result instanceof StepSkip) { + return undefined; + } + return result; + }), + runPipeline: jest.fn(async (ctx: unknown, ...steps: { action: (ctx: unknown) => Promise }[]) => { + for (const { action } of steps) { + await action(ctx); + } + return ctx; + }), logger: { error: jest.fn(), warn: jest.fn(), @@ -30,8 +23,8 @@ jest.mock('../src/core/logger', () => { success: jest.fn(), log: jest.fn(), br: jest.fn(), + showError: jest.fn(), }, - showErrorMessages: jest.fn(), } }); @@ -46,4 +39,4 @@ jest.mock('../src/core/fs', () => { jest.mock('../src/core/shell'); // jest.mock('../src/core/fs'); -jest.mock('inquirer'); \ No newline at end of file +jest.mock('inquirer'); diff --git a/cli/tests/integration-setup.ts b/cli/tests/integration-setup.ts new file mode 100644 index 00000000..1596ac39 --- /dev/null +++ b/cli/tests/integration-setup.ts @@ -0,0 +1,29 @@ +jest.mock('../src/core/logger', () => { + const actual = jest.requireActual('../src/core/logger/step-runner'); + const { StepSkip } = actual; + return { + ...jest.requireActual('../src/core/logger'), + runStep: jest.fn(async (_message: string, action: () => Promise) => { + const result = await action(); + if (result instanceof StepSkip) { + return undefined; + } + return result; + }), + runPipeline: jest.fn(async (ctx: unknown, ...steps: { action: (ctx: unknown) => Promise }[]) => { + for (const { action } of steps) { + await action(ctx); + } + return ctx; + }), + logger: { + error: jest.fn(), + warn: jest.fn(), + info: jest.fn(), + success: jest.fn(), + log: jest.fn(), + br: jest.fn(), + showError: jest.fn(), + }, + }; +}); diff --git a/cli/tests/integration/host-run-helper.ts b/cli/tests/integration/host-run-helper.ts new file mode 100644 index 00000000..eb205809 --- /dev/null +++ b/cli/tests/integration/host-run-helper.ts @@ -0,0 +1,131 @@ +import * as path from 'path'; +import * as fs from '../../src/core/fs'; +import { ProjectConfigHandler } from '../../src/config/project-config'; +import { PROJECT_DEFAULT_PATHS } from '../../src/config/project-config'; + +export type HostPackageSpec = { + name: string; + sources: Record; +}; + +function writeSources(root: string, sources: Record) { + for (const [relativePath, code] of Object.entries(sources)) { + const filePath = path.join(root, relativePath); + fs.makeDir(path.dirname(filePath)); + fs.writeFile(filePath, code); + } +} + +function createHostPackage( + projectRoot: string, + packageName: string, + sources: Record, + runtimeDir: string, +) { + const packageRoot = path.join(projectRoot, PROJECT_DEFAULT_PATHS.PACKAGES_DIR, packageName); + writeSources(packageRoot, sources); + + const handler = ProjectConfigHandler.createTemplate(packageName, 'host', packageRoot); + handler.update({ + srcDir: './src', + entryFile: './src/index.bs', + runtimeDir, + }); + handler.save(packageRoot); +} + +export function createHostProject( + root: string, + sources: Record, + runtimeDir: string, + projectName = 'test-run', + packages: HostPackageSpec[] = [], +) { + for (const pkg of packages) { + createHostPackage(root, pkg.name, pkg.sources, runtimeDir); + } + + writeSources(root, sources); + + const handler = ProjectConfigHandler.createTemplate(projectName, 'host', root); + handler.update({ + srcDir: './src', + entryFile: './src/index.bs', + runtimeDir, + }); + for (const pkg of packages) { + handler.addDependency({ + name: pkg.name, + url: `https://example.com/${pkg.name}.git`, + }); + } + handler.save(root); +} + +export function removeDirIfExists(dir: string) { + if (fs.exists(dir)) { + fs.removeDir(dir); + } +} + +export function mockProcessExit() { + return jest + .spyOn(process, 'exit') + .mockImplementation((() => {}) as (code?: number | string | null | undefined) => never); +} + +export function captureStdout() { + const chunks: string[] = []; + const spy = jest.spyOn(process.stdout, 'write').mockImplementation((chunk) => { + chunks.push(typeof chunk === 'string' ? chunk : chunk.toString()); + return true; + }); + + return { + text: () => chunks.join(''), + restore: () => spy.mockRestore(), + }; +} + +export function captureOutput() { + const stdout = captureStdout(); + const consoleLogs: string[] = []; + const consoleSpy = jest.spyOn(console, 'log').mockImplementation((...args: unknown[]) => { + consoleLogs.push(args.map(String).join(' ')); + }); + + return { + text: () => stdout.text() + consoleLogs.join('\n'), + restore: () => { + stdout.restore(); + consoleSpy.mockRestore(); + }, + }; +} + +export async function waitFor( + predicate: () => boolean, + timeoutMs = 10000, + intervalMs = 50, +): Promise { + const start = Date.now(); + while (!predicate()) { + if (Date.now() - start > timeoutMs) { + throw new Error('waitFor timed out'); + } + await new Promise((resolve) => setTimeout(resolve, intervalMs)); + } +} + +export async function waitForStdoutContains( + output: { text: () => string }, + text: string, + timeoutMs = 10000, +): Promise { + await waitFor(() => output.text().includes(text), timeoutMs); +} + +export const HOST_INTEGRATION_RUNTIME_DIR = + path.resolve(__dirname, '../../../microcontroller'); +export const HOST_INTEGRATION_BUILD_DIR = + path.join(HOST_INTEGRATION_RUNTIME_DIR, 'ports/host/build'); diff --git a/cli/tests/integration/project/repl.host.test.ts b/cli/tests/integration/project/repl.host.test.ts new file mode 100644 index 00000000..03f8403f --- /dev/null +++ b/cli/tests/integration/project/repl.host.test.ts @@ -0,0 +1,199 @@ +jest.mock('../../../src/core/shell', () => ({ + ...jest.requireActual('../../../src/core/shell'), + cwd: jest.fn(), +})); + +import * as readline from 'readline'; +import * as path from 'path'; +import * as fs from '../../../src/core/fs'; +import { handleReplCommand } from '../../../src/commands/repl'; +import { logger } from '../../../src/core/logger'; +import { buildHostRuntime } from '../../../src/platforms/runtime/host-board-runtime'; +import { + deleteGlobalEnv, + setupGlobalEnvWithHostIntegration, + spyGlobalSettings, +} from '../../commands/global-env-helper'; +import { + captureOutput, + HOST_INTEGRATION_BUILD_DIR, + HOST_INTEGRATION_RUNTIME_DIR, + mockProcessExit, + removeDirIfExists, + waitFor, + waitForStdoutContains, +} from '../host-run-helper'; + +const TEMP_DIR = path.join(__dirname, '../../../temp-files/integration-repl'); +const SHELL_PATH = path.join(HOST_INTEGRATION_BUILD_DIR, 'shell'); +const RUNTIME_SO_PATH = path.join(HOST_INTEGRATION_BUILD_DIR, 'c-runtime.so'); + +const describeHost = process.platform === 'darwin' ? describe : describe.skip; + +let replLineHandler: ((line: string) => void) | undefined; +let replCloseHandler: (() => void) | undefined; + +function createMockReadline(): readline.Interface { + return { + prompt: jest.fn(), + pause: jest.fn(), + resume: jest.fn(), + on: jest.fn((event: string, handler: (...args: never[]) => void) => { + if (event === 'line') { + replLineHandler = handler as (line: string) => void; + } + if (event === 'close') { + replCloseHandler = handler as () => void; + } + }), + close: jest.fn(() => { + replCloseHandler?.(); + }), + } as unknown as readline.Interface; +} + +describeHost('repl command (host integration)', () => { + jest.setTimeout(30000); + + beforeAll(async () => { + spyGlobalSettings('repl-integration'); + fs.makeDir(TEMP_DIR); + + if (!fs.exists(SHELL_PATH) || !fs.exists(RUNTIME_SO_PATH)) { + await buildHostRuntime(HOST_INTEGRATION_RUNTIME_DIR, HOST_INTEGRATION_BUILD_DIR); + } + }); + + beforeEach(() => { + jest.clearAllMocks(); + deleteGlobalEnv(); + setupGlobalEnvWithHostIntegration( + HOST_INTEGRATION_RUNTIME_DIR, + HOST_INTEGRATION_BUILD_DIR, + ); + replLineHandler = undefined; + replCloseHandler = undefined; + }); + + afterAll(() => { + deleteGlobalEnv(); + removeDirIfExists(TEMP_DIR); + }); + + async function sendReplLine( + line: string, + output: ReturnType, + expectedOutput?: string, + ) { + if (!replLineHandler) { + throw new Error('REPL line handler is not ready'); + } + replLineHandler(line); + if (expectedOutput) { + await waitForStdoutContains(output, expectedOutput); + } else { + await new Promise((resolve) => setTimeout(resolve, 800)); + } + } + + async function closeRepl() { + if (!replCloseHandler) { + throw new Error('REPL close handler is not ready'); + } + replCloseHandler(); + } + + async function startReplSession() { + const exitSpy = mockProcessExit(); + const output = captureOutput(); + const replPromise = handleReplCommand( + { board: 'host' }, + { createReadline: createMockReadline }, + ); + await waitFor(() => replLineHandler !== undefined); + return { exitSpy, output, replPromise }; + } + + it('executes the first REPL line as the entry program', async () => { + const { exitSpy, output, replPromise } = await startReplSession(); + + await sendReplLine('console.log("repl entry");', output, 'repl entry'); + await closeRepl(); + await replPromise; + + expect(exitSpy).toHaveBeenCalledWith(0); + expect(output.text()).toContain('repl entry'); + + output.restore(); + exitSpy.mockRestore(); + }); + + it('executes built-in library calls in REPL lines', async () => { + const { exitSpy, output, replPromise } = await startReplSession(); + + await sendReplLine('console.log("init");', output, 'init'); + await sendReplLine('print("via print");', output, 'via print'); + await closeRepl(); + await replPromise; + + expect(exitSpy).toHaveBeenCalledWith(0); + expect(output.text()).toContain('init'); + expect(output.text()).toContain('via print'); + + output.restore(); + exitSpy.mockRestore(); + }); + + it('keeps variables available across REPL lines', async () => { + const { exitSpy, output, replPromise } = await startReplSession(); + + await sendReplLine('const x = 42; console.log("init");', output, 'init'); + await sendReplLine('console.log(x);', output, '42'); + await closeRepl(); + await replPromise; + + expect(exitSpy).toHaveBeenCalledWith(0); + + output.restore(); + exitSpy.mockRestore(); + }); + + it('keeps functions available across REPL lines', async () => { + const { exitSpy, output, replPromise } = await startReplSession(); + + await sendReplLine( + 'function double(n: integer): integer { return n * 2; } console.log("fn defined");', + output, + 'fn defined', + ); + await sendReplLine('console.log(double(21));', output, '42'); + await closeRepl(); + await replPromise; + + expect(exitSpy).toHaveBeenCalledWith(0); + + output.restore(); + exitSpy.mockRestore(); + }); + + it('continues REPL after a compile error', async () => { + const { exitSpy, output, replPromise } = await startReplSession(); + + await sendReplLine('this is not valid bluescript', output); + await waitFor(() => + (logger.error as jest.Mock).mock.calls.some(([message]) => + String(message).includes('** compile error:'), + ), + ); + + await sendReplLine('console.log("after error");', output, 'after error'); + await closeRepl(); + await replPromise; + + expect(exitSpy).toHaveBeenCalledWith(0); + expect(output.text()).toContain('after error'); + + output.restore(); + exitSpy.mockRestore(); + }); +}); diff --git a/cli/tests/integration/project/run.host.test.ts b/cli/tests/integration/project/run.host.test.ts new file mode 100644 index 00000000..43267d19 --- /dev/null +++ b/cli/tests/integration/project/run.host.test.ts @@ -0,0 +1,270 @@ +jest.mock('../../../src/core/shell', () => ({ + ...jest.requireActual('../../../src/core/shell'), + cwd: jest.fn(), +})); + +import * as path from 'path'; +import { cwd } from '../../../src/core/shell'; +import * as fs from '../../../src/core/fs'; +import { handleRunCommand } from '../../../src/commands/project/run'; +import { buildHostRuntime } from '../../../src/platforms/runtime/host-board-runtime'; +import { + deleteGlobalEnv, + setupGlobalEnvWithHostIntegration, + spyGlobalSettings, +} from '../../commands/global-env-helper'; +import { + captureStdout, + createHostProject, + mockProcessExit, + removeDirIfExists, +} from '../host-run-helper'; + +const mockedCwd = cwd as jest.Mock; + +const TEMP_DIR = path.join(__dirname, '../../../temp-files/integration'); +const RUNTIME_DIR = path.resolve(__dirname, '../../../../microcontroller'); +const BUILD_DIR = path.join(RUNTIME_DIR, 'ports/host/build'); +const PROJECT_ROOT = path.join(TEMP_DIR, 'run-project'); +const SHELL_PATH = path.join(BUILD_DIR, 'shell'); +const RUNTIME_SO_PATH = path.join(BUILD_DIR, 'c-runtime.so'); + +const describeHost = process.platform === 'darwin' ? describe : describe.skip; + +describeHost('project run command (host integration)', () => { + beforeAll(async () => { + spyGlobalSettings('run-integration'); + fs.makeDir(TEMP_DIR); + + if (!fs.exists(SHELL_PATH) || !fs.exists(RUNTIME_SO_PATH)) { + await buildHostRuntime(RUNTIME_DIR, BUILD_DIR); + } + }); + + beforeEach(() => { + deleteGlobalEnv(); + setupGlobalEnvWithHostIntegration(RUNTIME_DIR, BUILD_DIR); + removeDirIfExists(PROJECT_ROOT); + fs.makeDir(PROJECT_ROOT); + mockedCwd.mockReturnValue(PROJECT_ROOT); + }); + + afterAll(() => { + deleteGlobalEnv(); + removeDirIfExists(TEMP_DIR); + }); + + it('runs a program and prints output', async () => { + const exitSpy = mockProcessExit(); + const stdout = captureStdout(); + + createHostProject(PROJECT_ROOT, { + 'src/index.bs': 'console.log("hello from run");', + }, RUNTIME_DIR); + + await handleRunCommand({ withRepl: false, withNotebook: false }); + + expect(exitSpy).toHaveBeenCalledWith(0); + expect(stdout.text()).toContain('hello from run'); + + stdout.restore(); + exitSpy.mockRestore(); + }); + + it('runs a program using the built-in library', async () => { + const exitSpy = mockProcessExit(); + const stdout = captureStdout(); + + createHostProject(PROJECT_ROOT, { + 'src/index.bs': ` +console.log("built-in"); +print("via print"); +console.log(time.now()); + `.trim(), + }, RUNTIME_DIR); + + await handleRunCommand({ withRepl: false, withNotebook: false }); + + expect(exitSpy).toHaveBeenCalledWith(0); + expect(stdout.text()).toContain('built-in'); + expect(stdout.text()).toContain('via print'); + + stdout.restore(); + exitSpy.mockRestore(); + }); + + it('runs a program with user-defined functions and variables', async () => { + const exitSpy = mockProcessExit(); + const stdout = captureStdout(); + + createHostProject(PROJECT_ROOT, { + 'src/index.bs': ` +const message = "hello"; +function greet(): void { + console.log(message); +} +greet(); + `.trim(), + }, RUNTIME_DIR); + + await handleRunCommand({ withRepl: false, withNotebook: false }); + + expect(exitSpy).toHaveBeenCalledWith(0); + expect(stdout.text()).toContain('hello'); + + stdout.restore(); + exitSpy.mockRestore(); + }); + + it('runs a program with a local module import', async () => { + const exitSpy = mockProcessExit(); + const stdout = captureStdout(); + + createHostProject(PROJECT_ROOT, { + 'src/math-utils.bs': ` +export function add(a: integer, b: integer): integer { + return a + b; +} + `.trim(), + 'src/index.bs': ` +import { add } from "./math-utils"; +console.log(add(10, 20)); + `.trim(), + }, RUNTIME_DIR); + + await handleRunCommand({ withRepl: false, withNotebook: false }); + + expect(exitSpy).toHaveBeenCalledWith(0); + expect(stdout.text()).toContain('30'); + + stdout.restore(); + exitSpy.mockRestore(); + }); + + it('runs a program with a package import', async () => { + const exitSpy = mockProcessExit(); + const stdout = captureStdout(); + + createHostProject(PROJECT_ROOT, { + 'src/index.bs': ` +import { mul } from "math-lib"; +console.log(mul(3, 4)); + `.trim(), + }, RUNTIME_DIR, 'test-run', [{ + name: 'math-lib', + sources: { + 'src/index.bs': ` +export function mul(a: integer, b: integer): integer { + return a * b; +} + `.trim(), + }, + }]); + + await handleRunCommand({ withRepl: false, withNotebook: false }); + + expect(exitSpy).toHaveBeenCalledWith(0); + expect(stdout.text()).toContain('12'); + + stdout.restore(); + exitSpy.mockRestore(); + }); + + it('runs a program using inline C', async () => { + const exitSpy = mockProcessExit(); + const stdout = captureStdout(); + + createHostProject(PROJECT_ROOT, { + 'src/index.bs': ` +code\`#include \` + +function pow(x: float, y: float): float { + let result: float; + code\`\${result} = (float)pow(\${x}, \${y});\`; + return result; +} + +console.log(pow(2.0, 3.0)); + `.trim(), + }, RUNTIME_DIR); + + await handleRunCommand({ withRepl: false, withNotebook: false }); + + expect(exitSpy).toHaveBeenCalledWith(0); + expect(stdout.text()).toMatch(/8(\.0+)?/); + + stdout.restore(); + exitSpy.mockRestore(); + }); + + it('runs a program that includes a C file', async () => { + const exitSpy = mockProcessExit(); + const stdout = captureStdout(); + + createHostProject(PROJECT_ROOT, { + 'src/add.c': 'int add(int a, int b) { return a + b; }', + 'src/index.bs': ` +code\`#include "./add.c"\` + +function main(): void { + let result: integer = 0; + code\`\${result} = add(10, 20);\`; + console.log(result); +} + +main(); + `.trim(), + }, RUNTIME_DIR); + + await handleRunCommand({ withRepl: false, withNotebook: false }); + + expect(exitSpy).toHaveBeenCalledWith(0); + expect(stdout.text()).toContain('30'); + + stdout.restore(); + exitSpy.mockRestore(); + }); + + it('runs a program that includes a header file', async () => { + const exitSpy = mockProcessExit(); + const stdout = captureStdout(); + + createHostProject(PROJECT_ROOT, { + 'src/add.h': 'int add(int a, int b);', + 'src/add.c': '#include "add.h"\nint add(int a, int b) { return a + b; }', + 'src/index.bs': ` +code\`#include "add.h"\` + +function main(): void { + let result: integer = 0; + code\`\${result} = add(5, 6);\`; + console.log(result); +} + +main(); + `.trim(), + }, RUNTIME_DIR); + + await handleRunCommand({ withRepl: false, withNotebook: false }); + + expect(exitSpy).toHaveBeenCalledWith(0); + expect(stdout.text()).toContain('11'); + + stdout.restore(); + exitSpy.mockRestore(); + }); + + it('exits with an error when compilation fails', async () => { + const exitSpy = mockProcessExit(); + + createHostProject(PROJECT_ROOT, { + 'src/index.bs': 'this is not valid bluescript', + }, RUNTIME_DIR); + + await handleRunCommand({ withRepl: false, withNotebook: false }); + + expect(exitSpy).toHaveBeenCalledWith(1); + + exitSpy.mockRestore(); + }); +}); diff --git a/lang/src/compiler/board-toolchain/board-toolchain.ts b/lang/src/compiler/board-toolchain/board-toolchain.ts index c1953149..de564456 100644 --- a/lang/src/compiler/board-toolchain/board-toolchain.ts +++ b/lang/src/compiler/board-toolchain/board-toolchain.ts @@ -1,4 +1,4 @@ -import { Package, Project } from '../project'; +import { Project } from '../project'; export type MemoryLayout = { iram:{address:number, size:number}, @@ -30,7 +30,7 @@ export class ShadowMemory { } } -export type ExecutableBinary = { +export type MemoryImage = { iram?: {address: number, data: Buffer}, dram?: {address: number, data: Buffer}, iflash?: {address: number, data: Buffer}, @@ -38,12 +38,16 @@ export type ExecutableBinary = { entryPoints: {isMain: boolean, address: number}[] } -export interface BoardToolchain

{ - memory: ShadowMemory; +export type SharedObject = { + soFile: string, + entryNames: { isMain: boolean, name: string}[], +}; +export type CompileOutput = MemoryImage | SharedObject; + +export interface BoardToolchain

{ get cProlog(): string; get builtinModulePath(): string; - compileC(project: Project

, pkg: P): Promise; - link(project: Project

, entryPoints: string[]): Promise; - extractBinary(elfPath: string, entryPoints: string[]): ExecutableBinary; + compileAndLink(project: P, entryPoints: string[]): Promise; + additionalCompileAndLink(project: P, entryPoints: string[]): Promise; } \ No newline at end of file diff --git a/lang/src/compiler/board-toolchain/esp32-toolchain.ts b/lang/src/compiler/board-toolchain/esp32-toolchain.ts index cd4a4e7b..40d3789a 100644 --- a/lang/src/compiler/board-toolchain/esp32-toolchain.ts +++ b/lang/src/compiler/board-toolchain/esp32-toolchain.ts @@ -1,11 +1,11 @@ import * as path from "path"; import * as fs from "fs"; -import { PackageForEsp32, Project } from "../project"; -import { BoardToolchain, ExecutableBinary, MemoryLayout, ShadowMemory } from "./board-toolchain"; +import { PackageForEsp32, ProjectForEsp32 } from "../project"; +import { BoardToolchain, MemoryImage, MemoryLayout, ShadowMemory } from "./board-toolchain"; import { executeCommand, getErrorMessage } from "../utils"; -import generateMakefile from "./makefile"; -import { ElfReader } from "./elf-reader"; -import generateLinkerScript from "./linker-script"; +import { generateMakefile, esp32MakefilePreset } from "./tools/makefile"; +import { ElfReader } from "./tools/elf-reader"; +import generateLinkerScript from "./tools/linker-script"; export type Esp32ToolchainConfig = { @@ -14,11 +14,12 @@ export type Esp32ToolchainConfig = { espDir: string } -export class Esp32Toolchain implements BoardToolchain { +export class Esp32Toolchain implements BoardToolchain { public memory: ShadowMemory; private config: Esp32ToolchainConfig; private espIdfComponents: EspIdfComponents; + private compiledPackages = new Set(); private definedSymbols: Map; get cProlog() { @@ -41,9 +42,31 @@ export class Esp32Toolchain implements BoardToolchain { this.definedSymbols = new Map(elfReader.readAllSymbols().map(s => [s.name, s])); } - async compileC(project: Project, pkg: PackageForEsp32): Promise { + async compileAndLink(project: ProjectForEsp32, entryPoints: string[]): Promise { + for (const pkg of project.usedDependencies) { + await this.compileC(project, pkg); + this.compiledPackages.add(pkg.name); + } + await this.compileC(project, project.mainPackage); + const elfPath = await this.link(project, entryPoints); + return this.extractBinary(elfPath, entryPoints); + } + + async additionalCompileAndLink(project: ProjectForEsp32, entryPoints: string[]): Promise { + for (const pkg of project.usedDependencies) { + if (!this.compiledPackages.has(pkg.name)) { + await this.compileC(project, pkg); + this.compiledPackages.add(pkg.name); + } + } + await this.compileC(project, project.mainPackage); + const elfPath = await this.link(project, entryPoints); + return this.extractBinary(elfPath, entryPoints); + } + + private async compileC(project: ProjectForEsp32, pkg: PackageForEsp32): Promise { try { - const archivePath = project.archivePath(pkg); + const archivePath = project.archiveFile(pkg); const includeDirs = [ ...this.espIdfComponents.getIncludeDirs(pkg.espIdfComponents), ...this.espIdfComponents.commonIncludeDirs @@ -54,10 +77,10 @@ export class Esp32Toolchain implements BoardToolchain { fs.rmSync(archivePath, { force: true }); } - const makefile = generateMakefile( + const makefile = generateMakefile(esp32MakefilePreset( this.config.compilerToolchainDir, pkg, includeDirs, archivePath - ); + )); project.writeMakefile(pkg, makefile); await executeCommand('make', [], pkg.resolvedDistDir); } catch (error) { @@ -65,10 +88,10 @@ export class Esp32Toolchain implements BoardToolchain { } } - async link(project: Project, entryPoints: string[]): Promise { + private async link(project: ProjectForEsp32, entryPoints: string[]): Promise { try { const cwd = process.cwd(); - const elfPath = project.elfPath(); + const elfPath = project.elfFile(); const archives = this.getArchivesWithEspComponents(project); const linkerscript = generateLinkerScript( @@ -88,7 +111,7 @@ export class Esp32Toolchain implements BoardToolchain { } } - extractBinary(elfPath: string, entryPoints: string[]): ExecutableBinary { + private extractBinary(elfPath: string, entryPoints: string[]): MemoryImage { const elf = new ElfReader(elfPath); const sections = { @@ -120,9 +143,9 @@ export class Esp32Toolchain implements BoardToolchain { } } - private getArchivesWithEspComponents(project: Project): string[] { + private getArchivesWithEspComponents(project: ProjectForEsp32): string[] { const espArchivesFromMain = this.espIdfComponents.getArchiveFilePaths(project.mainPackage.espIdfComponents); - const resultArchives = [project.archivePath(project.mainPackage), ...espArchivesFromMain]; + const resultArchives = [project.archiveFile(project.mainPackage), ...espArchivesFromMain]; const visitedEspArchives = new Set(espArchivesFromMain); @@ -133,14 +156,11 @@ export class Esp32Toolchain implements BoardToolchain { } }; - const reversedPackages = project.dependencies.filter(dep => dep.used).reverse(); - for (const pkg of reversedPackages) { - resultArchives.push(project.archivePath(pkg)); + for (const pkg of project.usedDependencies.reverse()) { + resultArchives.push(project.archiveFile(pkg)); const espArchivesFromPkg = this.espIdfComponents.getArchiveFilePaths(pkg.espIdfComponents); espArchivesFromPkg.forEach(ar => addEspArchive(ar)); } - - // Add common components this.espIdfComponents.commonArchiveFiles.forEach(ar => addEspArchive(ar)); return resultArchives; diff --git a/lang/src/compiler/board-toolchain/host-toolchain.ts b/lang/src/compiler/board-toolchain/host-toolchain.ts new file mode 100644 index 00000000..7f7cd957 --- /dev/null +++ b/lang/src/compiler/board-toolchain/host-toolchain.ts @@ -0,0 +1,106 @@ +import * as path from "path"; +import * as fs from "fs"; +import { BoardToolchain, SharedObject } from "./board-toolchain"; +import { Package, ProjectForHost } from "../project"; +import { generateMakefile, hostMakefilePreset } from "./tools/makefile"; +import { executeCommand, getErrorMessage } from "../utils"; + + +export class HostToolchain implements BoardToolchain { + private runtimeDir: string; + private compileId: number = 0; + private compiledPackages = new Set(); + private generatedSoFiles: string[] = []; + + constructor(runtimeDir: string) { + this.runtimeDir = runtimeDir; + } + + get cProlog() { + return ` +#include +#include "${this.cRuntimeH}" +`; + } + get cRuntimeH() { return path.join(this.runtimeDir, 'core/include/c-runtime.h'); } + get builtinModulePath() { return path.join(this.runtimeDir, 'ports/host/std-module.bs'); } + get runtimeBuildDir() { return path.join(this.runtimeDir, 'ports/host/build'); } + get executableShell() { return path.join(this.runtimeBuildDir, 'shell'); } + get runtimeSo() { return path.join(this.runtimeBuildDir, 'c-runtime.so'); } + + async compileAndLink(project: ProjectForHost, entryPoints: string[]): Promise { + const archiveFiles: string[] = []; + for (const pkg of project.usedDependencies) { + archiveFiles.push(await this.compilePackage(project, pkg)); + this.compiledPackages.add(pkg.name); + } + archiveFiles.push(await this.compilePackage(project, project.mainPackage)); + const soFile = project.soFile(); + await this.link(archiveFiles, entryPoints, soFile); + this.generatedSoFiles.push(soFile); + return { + soFile, + entryNames: entryPoints.map(name => ({isMain: name === project.mainPackage.name, name})), + } + } + + async additionalCompileAndLink(project: ProjectForHost, entryPoints: string[]): Promise { + const archiveFiles: string[] = []; + for (const pkg of project.usedDependencies) { + if (!this.compiledPackages.has(pkg.name)) { + archiveFiles.push(await this.compilePackage(project, pkg)); + this.compiledPackages.add(pkg.name); + } + } + archiveFiles.push(await this.compilePackage(project, project.mainPackage)); + const soFile = project.soFile(this.compileId++); + await this.link(archiveFiles, entryPoints, soFile); + this.generatedSoFiles.push(soFile); + return { + soFile, + entryNames: entryPoints.map(name => ({isMain: name === project.mainPackage.name, name})), + } + } + + private async compilePackage(project: ProjectForHost, pkg: Package): Promise { + try { + const archiveFile = project.archiveFile(pkg); + + // Remove old archive file. + if (fs.existsSync(archiveFile)) { + fs.rmSync(archiveFile, { force: true }); + } + + const makefile = generateMakefile(hostMakefilePreset(pkg, archiveFile)); + project.writeMakefile(pkg, makefile); + await executeCommand('make', [], pkg.resolvedDistDir); + return archiveFile; + } catch (error) { + throw new Error(`Failed to compile package ${pkg.name}: ${getErrorMessage(error)}`, {cause: error}); + } + } + + private linkerSymbolName(sym: string): string { + return process.platform === 'darwin' ? `_${sym}` : sym; + } + + private async link(archiveFiles: string[], entryPoints: string[], outputFile: string): Promise { + try { + const keepEntrySymbols = entryPoints.map( + (sym) => `-Wl,-u,${this.linkerSymbolName(sym)}`, + ); + const args = [ + '-shared', '-fPIC', + '-o', outputFile, + ...archiveFiles, + ...this.generatedSoFiles, + this.runtimeSo, + '-lm', '-ldl', + ...keepEntrySymbols, + ]; + await executeCommand('cc', args); + } catch (error) { + throw new Error(`Failed to link: ${getErrorMessage(error)}`, {cause: error}); + } + } +} \ No newline at end of file diff --git a/lang/src/compiler/board-toolchain/makefile.ts b/lang/src/compiler/board-toolchain/makefile.ts deleted file mode 100644 index 0e035ea4..00000000 --- a/lang/src/compiler/board-toolchain/makefile.ts +++ /dev/null @@ -1,88 +0,0 @@ -import { Package } from "../project"; - - -export default function generateMakefile( - compilerToolchainDir: string, - pkg: Package, - includeDirs: string[], - targetFilePath: string, -) { - return ` -# === Basic settings === -TOOLCHAIN_PREFIX := ${compilerToolchainDir}/xtensa-esp32-elf- -CC := $(TOOLCHAIN_PREFIX)gcc -AR := $(TOOLCHAIN_PREFIX)ar - -# === Directory settings === -SRC_DIR := ${pkg.resolvedSourceDir} -DIST_DIR := ${pkg.resolvedDistDir} -BUILD_DIR := ${pkg.resolvedBuildDir} -PACKAGES_DIR := ${pkg.resolvedPackageDir} - -TARGET := ${targetFilePath} - -# === Check for illegal file name prefixes === -ILLEGAL_PREFIX_FILES := $(shell find $(SRC_DIR) -path $(DIST_DIR) -prune -o -path $(PACKAGES_DIR) -prune -o -type f -name "bs_*.c" -print) -ifneq ($(ILLEGAL_PREFIX_FILES),) - $(error ERROR: You cannot use 'bs_' prefix for C source file names. Please remove or rename the following files: \\ - $(ILLEGAL_PREFIX_FILES)) -endif - -# === Source and object file settings === -ORIG_SOURCES := $(shell find $(SRC_DIR) -path $(DIST_DIR) -prune -o -path $(PACKAGES_DIR) -prune -o -type f -name "*.c" -print) -DIST_SOURCES := $(foreach src,$(ORIG_SOURCES), \\ - $(subst $(SRC_DIR),$(DIST_DIR), $(src)) \\ -) -DIST_SOURCES += $(shell find $(DIST_DIR) -path $(BUILD_DIR) -prune -o -type f -name "bs_*.c" -print) -OBJECTS := $(patsubst $(DIST_DIR)/%.c, $(BUILD_DIR)/%.o, $(DIST_SOURCES)) - -# === Compilation settings === -INCLUDES := ${includeDirs.map(path => `-I ${path}`).join(' ')} -CFLAGS := $(INCLUDES) -O2 -w -fno-common -ffunction-sections -fdata-sections -mtext-section-literals -mlongcalls -fno-zero-initialized-in-bss - - -# ==================================================================== -.PHONY: all - -all: $(TARGET) - -# Copy rules -# -------------------------------------------------------- - -define COPY_RULE_TEMPLATE -$(1): $(2) -\t@echo "Copying $$< to $$@" -\t@mkdir -p $$(dir $$@) -\t@cp $$< $$@ -endef - -$(foreach src,$(ORIG_SOURCES), \\ - $(eval $(call COPY_RULE_TEMPLATE, \\ - $(subst $(SRC_DIR),$(DIST_DIR), $(src)), \\ - $(src) \\ - )) \\ -) - - -# Build rules -# -------------------------------------------------------- - -$(TARGET): $(OBJECTS) -\t@echo "Archiving library: $@" -\t@mkdir -p $(@D) -\t$(AR) rcs $@ $^ - -vpath %.c $(DIST_DIR) - -$(BUILD_DIR)/%.o: $(DIST_DIR)/%.c -\t@echo "Compiling: $< -> $@" -\t@mkdir -p $(@D) -\t$(CC) $(CFLAGS) -c $< -o $@ - - -.PHONY: clean -clean: -\t@echo "Cleaning dist directory..." -\t@rm -rf $(DIST_DIR) -`; -} \ No newline at end of file diff --git a/lang/src/compiler/board-toolchain/elf-reader.ts b/lang/src/compiler/board-toolchain/tools/elf-reader.ts similarity index 100% rename from lang/src/compiler/board-toolchain/elf-reader.ts rename to lang/src/compiler/board-toolchain/tools/elf-reader.ts diff --git a/lang/src/compiler/board-toolchain/elf32.ts b/lang/src/compiler/board-toolchain/tools/elf32.ts similarity index 100% rename from lang/src/compiler/board-toolchain/elf32.ts rename to lang/src/compiler/board-toolchain/tools/elf32.ts diff --git a/lang/src/compiler/board-toolchain/linker-script.ts b/lang/src/compiler/board-toolchain/tools/linker-script.ts similarity index 99% rename from lang/src/compiler/board-toolchain/linker-script.ts rename to lang/src/compiler/board-toolchain/tools/linker-script.ts index 74e26f94..f0a26bba 100644 --- a/lang/src/compiler/board-toolchain/linker-script.ts +++ b/lang/src/compiler/board-toolchain/tools/linker-script.ts @@ -1,4 +1,4 @@ -import { ShadowMemory } from "./board-toolchain"; +import { ShadowMemory } from "../board-toolchain"; export default function generateLinkerScript( diff --git a/lang/src/compiler/board-toolchain/tools/makefile.ts b/lang/src/compiler/board-toolchain/tools/makefile.ts new file mode 100644 index 00000000..6f6b504a --- /dev/null +++ b/lang/src/compiler/board-toolchain/tools/makefile.ts @@ -0,0 +1,164 @@ +import { Package } from "../../project"; + + +type MakefileConfig = { + pkg: Package, + includeDirs: string[], + compileFlags: string[], + outputFile: string, + toolchain: { cc: string; ar: string } +} + + +export function esp32MakefilePreset(toolchainDir: string, pkg: Package, includeDirs: string[], outputFile: string): MakefileConfig { + return { + pkg, includeDirs, + compileFlags: [ + '-O2', '-w', '-fno-common', + '-ffunction-sections', '-fdata-sections', + '-mtext-section-literals', '-mlongcalls', + '-fno-zero-initialized-in-bss', + ], + outputFile, + toolchain: { + cc: `${toolchainDir}/xtensa-esp32-elf-gcc`, + ar: `${toolchainDir}/xtensa-esp32-elf-ar` + } + } +} + +export function hostMakefilePreset(pkg: Package, outputFile: string): MakefileConfig { + return { + pkg, + includeDirs: [], + compileFlags: ['-O2', '-w', '-fPIC', '-DLINUX64'], + outputFile, + toolchain: { + cc: `cc`, + ar: `ar` + } + } +} + +export function generateMakefile(config: MakefileConfig): string { + return [ + // preamble + renderHeader(config), + renderValidation(config), + renderSourceVars(config), + renderCompileFlags(config), + + // body + renderPhonyAll(config), + renderCopyRules(config), + renderBuildRules(config), + renderClean(config) + ].join('\n\n'); +} + +function renderHeader(config: MakefileConfig): string { + const { pkg } = config; + return `# === Basic settings === +CC := ${config.toolchain.cc} +AR := ${config.toolchain.ar} + +# === Directory settings === +SRC_DIR := ${pkg.resolvedSourceDir} +DIST_DIR := ${pkg.resolvedDistDir} +BUILD_DIR := ${pkg.resolvedBuildDir} +PACKAGES_DIR := ${pkg.resolvedPackageDir} + +TARGET := ${config.outputFile}`; +} + +function renderValidation(_config: MakefileConfig): string { + return `# === Check for illegal file name prefixes === +ILLEGAL_PREFIX_FILES := $(shell find $(SRC_DIR) -path $(DIST_DIR) -prune -o -path $(PACKAGES_DIR) -prune -o -type f -name "bs_*.c" -print) +ifneq ($(ILLEGAL_PREFIX_FILES),) + $(error ERROR: You cannot use 'bs_' prefix for C source file names. Please remove or rename the following files: \\ + $(ILLEGAL_PREFIX_FILES)) +endif`; +} + +function renderSourceVars(_config: MakefileConfig): string { + return `# === Source and object file settings === +ORIG_SOURCES := $(shell find $(SRC_DIR) -path $(DIST_DIR) -prune -o -path $(PACKAGES_DIR) -prune -o -type f -name "*.c" -print) +ORIG_HEADERS := $(shell find $(SRC_DIR) -path $(DIST_DIR) -prune -o -path $(PACKAGES_DIR) -prune -o -type f -name "*.h" -print) +DIST_SOURCES := $(foreach src,$(ORIG_SOURCES), \\ + $(subst $(SRC_DIR),$(DIST_DIR), $(src)) \\ +) +DIST_HEADERS := $(foreach hdr,$(ORIG_HEADERS), \\ + $(subst $(SRC_DIR),$(DIST_DIR), $(hdr)) \\ +) +DIST_SOURCES += $(shell find $(DIST_DIR) -path $(BUILD_DIR) -prune -o -type f -name "bs_*.c" -print) +OBJECTS := $(patsubst $(DIST_DIR)/%.c, $(BUILD_DIR)/%.o, $(DIST_SOURCES))`; +} + +function renderCompileFlags(config: MakefileConfig): string { + const extraIncludes = config.includeDirs.map(path => `-I ${path}`).join(' '); + const includes = extraIncludes + ? `-I $(DIST_DIR) -I $(SRC_DIR) ${extraIncludes}` + : `-I $(DIST_DIR) -I $(SRC_DIR)`; + return `# === Compilation settings === +INCLUDES := ${includes} +CFLAGS := $(INCLUDES) ${config.compileFlags.join(' ')}`; +} + +function renderPhonyAll(_config: MakefileConfig): string { + return `# ==================================================================== +.PHONY: all + +all: $(TARGET)`; +} + +function renderCopyRules(_config: MakefileConfig): string { + return `# Copy rules +# -------------------------------------------------------- + +define COPY_RULE_TEMPLATE +$(1): $(2) +\t@echo "Copying $$< to $$@" +\t@mkdir -p $$(dir $$@) +\t@cp $$< $$@ +endef + +$(foreach src,$(ORIG_SOURCES), \\ + $(eval $(call COPY_RULE_TEMPLATE, \\ + $(subst $(SRC_DIR),$(DIST_DIR), $(src)), \\ + $(src) \\ + )) \\ +) + +$(foreach hdr,$(ORIG_HEADERS), \\ + $(eval $(call COPY_RULE_TEMPLATE, \\ + $(subst $(SRC_DIR),$(DIST_DIR), $(hdr)), \\ + $(hdr) \\ + )) \\ +)`; +} + +function renderBuildRules(_config: MakefileConfig): string { + return `# Build rules +# -------------------------------------------------------- + +$(TARGET): $(OBJECTS) | $(DIST_HEADERS) +\t@echo "Archiving library: $@" +\t@mkdir -p $(@D) +\t$(AR) rcs $@ $^ + +vpath %.c $(DIST_DIR) + +$(BUILD_DIR)/%.o: $(DIST_DIR)/%.c +\t@echo "Compiling: $< -> $@" +\t@mkdir -p $(@D) +\t$(CC) $(CFLAGS) -MMD -MP -c $< -o $@ + +-include $(wildcard $(BUILD_DIR)/*.d)`; +} + +function renderClean(_config: MakefileConfig): string { + return `.PHONY: clean +clean: +\t@echo "Cleaning dist directory..." +\t@rm -rf $(DIST_DIR)`; +} diff --git a/lang/src/compiler/compiler-session.ts b/lang/src/compiler/compiler-session.ts index e2b9b2ae..532edd91 100644 --- a/lang/src/compiler/compiler-session.ts +++ b/lang/src/compiler/compiler-session.ts @@ -1,41 +1,34 @@ -import { BoardToolchain, ExecutableBinary } from "./board-toolchain/board-toolchain"; -import { Package, Project } from "./project"; +import { BoardToolchain, CompileOutput } from "./board-toolchain/board-toolchain"; +import { Project } from "./project"; import { TranspilerSession } from "./transpiler-session"; -export class CompilerSession

{ +export class CompilerSession

{ private transpiler: TranspilerSession; - private toolchain: BoardToolchain; - private currentProject: Project

| null = null; + private toolchain: BoardToolchain; + private project: P | null = null; - constructor(toolchain: BoardToolchain

) { + constructor(toolchain: BoardToolchain) { this.transpiler = new TranspilerSession(toolchain.builtinModulePath, toolchain.cProlog); this.toolchain = toolchain; } - public async buildProject(project: Project

): Promise { - this.currentProject = project; + public async buildProject(project: P): Promise { + this.project = project; project.check(); project.clean(); const entryPoints = this.transpiler.transpile(project); - const allPackages = [project.mainPackage, ...project.dependencies.filter(dep => dep.used)]; - for (const pkg of allPackages) { - await this.toolchain.compileC(project, pkg); - } - const elfPath = await this.toolchain.link(project, entryPoints); - return this.toolchain.extractBinary(elfPath, entryPoints); + return this.toolchain.compileAndLink(project, entryPoints); } - public async compileFragment(src: string): Promise { - if (!this.currentProject) { + public async compileFragment(src: string): Promise { + if (!this.project) { throw new Error("Cannot compile fragment before building the workspace."); } - const entryPoints = this.transpiler.transpileFragment(this.currentProject, src); - await this.toolchain.compileC(this.currentProject, this.currentProject.mainPackage); - const elfPath = await this.toolchain.link(this.currentProject, entryPoints); - return this.toolchain.extractBinary(elfPath, entryPoints); + const entryPoints = this.transpiler.transpileFragment(this.project, src); + return this.toolchain.additionalCompileAndLink(this.project, entryPoints); } } diff --git a/lang/src/compiler/project.ts b/lang/src/compiler/project.ts index 1ac1520c..78224c56 100644 --- a/lang/src/compiler/project.ts +++ b/lang/src/compiler/project.ts @@ -67,26 +67,31 @@ export class PackageForEsp32 extends Package { export class Project

{ public readonly mainPackage: P; - public readonly dependencies: (P & { used?: boolean })[]; + public readonly dependencies: Map; + private usedDependenciesMap = new Map(); - constructor(mainPackage: P, dependencies: P[]) { + protected constructor(mainPackage: P, dependencies: Map) { this.mainPackage = mainPackage; this.dependencies = dependencies; } - public static load

( + get usedDependencies() { + return [...this.usedDependenciesMap.values()]; + } + + protected static loadHelper

( mainPackageName: string, packageReader: (name: string) => P ) { const mainPackage = packageReader(mainPackageName); - const dependencies: P[] = []; + const dependencies = new Map(); const tmpQueue = [...mainPackage.dependencies]; const visited = new Set(mainPackage.dependencies); while (tmpQueue.length > 0) { const currName = tmpQueue.shift() as string; const pkg = packageReader(currName); - dependencies.push(pkg); + dependencies.set(pkg.name, pkg); for (const depName of pkg.dependencies) { if (!visited.has(depName)) { @@ -119,7 +124,7 @@ export class Project

{ clean() { this.cleanDistDir(this.mainPackage); - for (const dep of this.dependencies) { + for (const dep of this.dependencies.values()) { this.cleanDistDir(dep); } } @@ -153,27 +158,63 @@ export class Project

{ return filePath; } + addUsedDependency(pkg: P) { + if (pkg.name !== this.mainPackage.name) { + this.usedDependenciesMap.set(pkg.name, pkg); + } + } + + archiveFile(pkg: Package): AbsolutePath { + return path.join(pkg.resolvedBuildDir, `lib${pkg.name}.a`); + } +} + + +export class ProjectForEsp32 extends Project { + private constructor(mainPackage: PackageForEsp32, dependencies: Map) { + super(mainPackage, dependencies); + } + + public static load( + mainPackageName: string, + packageReader: (name: string) => PackageForEsp32, + ): ProjectForEsp32 { + const project = Project.loadHelper(mainPackageName, packageReader); + return new ProjectForEsp32(project.mainPackage, project.dependencies); + } + writeLinkerScript(data: string) { const filePath = path.join(this.mainPackage.resolvedBuildDir, "linkerscript.ld"); fs.writeFileSync(filePath, data); return filePath; } - markDependencyAsUsed(name: string) { - const dependency = this.dependencies.find(dep => dep.name === name); - if (dependency) { - dependency.used = true; - } + elfFile(): AbsolutePath { + return path.join( + this.mainPackage.resolvedBuildDir, + `${this.mainPackage.name}.elf` + ); } +} - archivePath(pkg: P) { - return path.join(pkg.resolvedBuildDir, `lib${pkg.name}.a`); + +export class ProjectForHost extends Project { + private constructor(mainPackage: Package, dependencies: Map) { + super(mainPackage, dependencies); + } + + public static load( + mainPackageName: string, + packageReader: (name: string) => Package, + ): ProjectForHost { + const project = Project.loadHelper(mainPackageName, packageReader); + return new ProjectForHost(project.mainPackage, project.dependencies); } - elfPath() { + soFile(id?: number): AbsolutePath { return path.join( this.mainPackage.resolvedBuildDir, - `${this.mainPackage.name}.elf` + `${this.mainPackage.name}${id ?? ''}.so` ); } } \ No newline at end of file diff --git a/lang/src/compiler/transpiler-session.ts b/lang/src/compiler/transpiler-session.ts index 46ef4785..02eb1bfa 100644 --- a/lang/src/compiler/transpiler-session.ts +++ b/lang/src/compiler/transpiler-session.ts @@ -77,7 +77,7 @@ export class TranspilerSession { return (name: string): GlobalVariableNameTable => { const newPath = this.resolveImport(currentPath, name, project.dependencies); const mod = this.modules.get(newPath.absolutePath); - project.markDependencyAsUsed(newPath.pkg.name); + project.addUsedDependency(newPath.pkg); if (mod) return mod; else { @@ -97,14 +97,14 @@ export class TranspilerSession { } } - private resolveImport(currentPath: PathInPkg, importName: string, dependencies: Package[]): PathInPkg { + private resolveImport(currentPath: PathInPkg, importName: string, dependencies: Map): PathInPkg { if (path.isAbsolute(importName)) { throw new Error("This module system does not support importing from absolute paths."); } else if (importName.startsWith('.')) { // move in package return currentPath.resolve(importName + '.bs'); } else { // move to new package const [pkgName, ...remain] = importName.split('/'); - const pkg = dependencies.find(dep => dep.name === pkgName); + const pkg = dependencies.get(pkgName); if (pkg === undefined) { throw new Error(`Cannot fine package. Package name: ${pkgName}`); } diff --git a/lang/src/index.ts b/lang/src/index.ts index 69eb8145..c4e5ad9f 100644 --- a/lang/src/index.ts +++ b/lang/src/index.ts @@ -1,5 +1,6 @@ export { ErrorLog as CompileError } from './transpiler/utils'; export { CompilerSession } from './compiler/compiler-session'; -export { PackageForEsp32, Project } from './compiler/project'; +export { Package, PackageForEsp32, Project, ProjectForEsp32, ProjectForHost } from './compiler/project'; export { Esp32Toolchain, Esp32ToolchainConfig } from './compiler/board-toolchain/esp32-toolchain'; -export { MemoryLayout, ExecutableBinary } from './compiler/board-toolchain/board-toolchain'; \ No newline at end of file +export { HostToolchain } from './compiler/board-toolchain/host-toolchain'; +export { MemoryLayout, MemoryImage, CompileOutput, SharedObject } from './compiler/board-toolchain/board-toolchain'; \ No newline at end of file diff --git a/lang/tests/compiler/__snapshots__/makefile.test.ts.snap b/lang/tests/compiler/__snapshots__/makefile.test.ts.snap new file mode 100644 index 00000000..31635675 --- /dev/null +++ b/lang/tests/compiler/__snapshots__/makefile.test.ts.snap @@ -0,0 +1,177 @@ +// Jest Snapshot v1, https://goo.gl/fbAQLP + +exports[`generateMakefile generates makefile for esp32 archive file. 1`] = ` +"# === Basic settings === +CC := /opt/esp-toolchain/xtensa-esp32-elf-gcc +AR := /opt/esp-toolchain/xtensa-esp32-elf-ar + +# === Directory settings === +SRC_DIR := /project/myapp/src +DIST_DIR := /project/myapp/dist +BUILD_DIR := /project/myapp/dist/build +PACKAGES_DIR := /project/myapp/packages + +TARGET := /project/myapp/dist/build/libmyapp.a + +# === Check for illegal file name prefixes === +ILLEGAL_PREFIX_FILES := $(shell find $(SRC_DIR) -path $(DIST_DIR) -prune -o -path $(PACKAGES_DIR) -prune -o -type f -name "bs_*.c" -print) +ifneq ($(ILLEGAL_PREFIX_FILES),) + $(error ERROR: You cannot use 'bs_' prefix for C source file names. Please remove or rename the following files: \\ + $(ILLEGAL_PREFIX_FILES)) +endif + +# === Source and object file settings === +ORIG_SOURCES := $(shell find $(SRC_DIR) -path $(DIST_DIR) -prune -o -path $(PACKAGES_DIR) -prune -o -type f -name "*.c" -print) +ORIG_HEADERS := $(shell find $(SRC_DIR) -path $(DIST_DIR) -prune -o -path $(PACKAGES_DIR) -prune -o -type f -name "*.h" -print) +DIST_SOURCES := $(foreach src,$(ORIG_SOURCES), \\ + $(subst $(SRC_DIR),$(DIST_DIR), $(src)) \\ +) +DIST_HEADERS := $(foreach hdr,$(ORIG_HEADERS), \\ + $(subst $(SRC_DIR),$(DIST_DIR), $(hdr)) \\ +) +DIST_SOURCES += $(shell find $(DIST_DIR) -path $(BUILD_DIR) -prune -o -type f -name "bs_*.c" -print) +OBJECTS := $(patsubst $(DIST_DIR)/%.c, $(BUILD_DIR)/%.o, $(DIST_SOURCES)) + +# === Compilation settings === +INCLUDES := -I $(DIST_DIR) -I $(SRC_DIR) -I /opt/esp-idf/components/freertos/include -I /opt/esp-idf/components/esp_common/include +CFLAGS := $(INCLUDES) -O2 -w -fno-common -ffunction-sections -fdata-sections -mtext-section-literals -mlongcalls -fno-zero-initialized-in-bss + +# ==================================================================== +.PHONY: all + +all: $(TARGET) + +# Copy rules +# -------------------------------------------------------- + +define COPY_RULE_TEMPLATE +$(1): $(2) + @echo "Copying $$< to $$@" + @mkdir -p $$(dir $$@) + @cp $$< $$@ +endef + +$(foreach src,$(ORIG_SOURCES), \\ + $(eval $(call COPY_RULE_TEMPLATE, \\ + $(subst $(SRC_DIR),$(DIST_DIR), $(src)), \\ + $(src) \\ + )) \\ +) + +$(foreach hdr,$(ORIG_HEADERS), \\ + $(eval $(call COPY_RULE_TEMPLATE, \\ + $(subst $(SRC_DIR),$(DIST_DIR), $(hdr)), \\ + $(hdr) \\ + )) \\ +) + +# Build rules +# -------------------------------------------------------- + +$(TARGET): $(OBJECTS) | $(DIST_HEADERS) + @echo "Archiving library: $@" + @mkdir -p $(@D) + $(AR) rcs $@ $^ + +vpath %.c $(DIST_DIR) + +$(BUILD_DIR)/%.o: $(DIST_DIR)/%.c + @echo "Compiling: $< -> $@" + @mkdir -p $(@D) + $(CC) $(CFLAGS) -MMD -MP -c $< -o $@ + +-include $(wildcard $(BUILD_DIR)/*.d) + +.PHONY: clean +clean: + @echo "Cleaning dist directory..." + @rm -rf $(DIST_DIR)" +`; + +exports[`generateMakefile generates makefile for host shared library. 1`] = ` +"# === Basic settings === +CC := cc +AR := ar + +# === Directory settings === +SRC_DIR := /project/myapp/src +DIST_DIR := /project/myapp/dist +BUILD_DIR := /project/myapp/dist/build +PACKAGES_DIR := /project/myapp/packages + +TARGET := /project/myapp/dist/build/libmyapp.a + +# === Check for illegal file name prefixes === +ILLEGAL_PREFIX_FILES := $(shell find $(SRC_DIR) -path $(DIST_DIR) -prune -o -path $(PACKAGES_DIR) -prune -o -type f -name "bs_*.c" -print) +ifneq ($(ILLEGAL_PREFIX_FILES),) + $(error ERROR: You cannot use 'bs_' prefix for C source file names. Please remove or rename the following files: \\ + $(ILLEGAL_PREFIX_FILES)) +endif + +# === Source and object file settings === +ORIG_SOURCES := $(shell find $(SRC_DIR) -path $(DIST_DIR) -prune -o -path $(PACKAGES_DIR) -prune -o -type f -name "*.c" -print) +ORIG_HEADERS := $(shell find $(SRC_DIR) -path $(DIST_DIR) -prune -o -path $(PACKAGES_DIR) -prune -o -type f -name "*.h" -print) +DIST_SOURCES := $(foreach src,$(ORIG_SOURCES), \\ + $(subst $(SRC_DIR),$(DIST_DIR), $(src)) \\ +) +DIST_HEADERS := $(foreach hdr,$(ORIG_HEADERS), \\ + $(subst $(SRC_DIR),$(DIST_DIR), $(hdr)) \\ +) +DIST_SOURCES += $(shell find $(DIST_DIR) -path $(BUILD_DIR) -prune -o -type f -name "bs_*.c" -print) +OBJECTS := $(patsubst $(DIST_DIR)/%.c, $(BUILD_DIR)/%.o, $(DIST_SOURCES)) + +# === Compilation settings === +INCLUDES := -I $(DIST_DIR) -I $(SRC_DIR) +CFLAGS := $(INCLUDES) -O2 -w -fPIC -DLINUX64 + +# ==================================================================== +.PHONY: all + +all: $(TARGET) + +# Copy rules +# -------------------------------------------------------- + +define COPY_RULE_TEMPLATE +$(1): $(2) + @echo "Copying $$< to $$@" + @mkdir -p $$(dir $$@) + @cp $$< $$@ +endef + +$(foreach src,$(ORIG_SOURCES), \\ + $(eval $(call COPY_RULE_TEMPLATE, \\ + $(subst $(SRC_DIR),$(DIST_DIR), $(src)), \\ + $(src) \\ + )) \\ +) + +$(foreach hdr,$(ORIG_HEADERS), \\ + $(eval $(call COPY_RULE_TEMPLATE, \\ + $(subst $(SRC_DIR),$(DIST_DIR), $(hdr)), \\ + $(hdr) \\ + )) \\ +) + +# Build rules +# -------------------------------------------------------- + +$(TARGET): $(OBJECTS) | $(DIST_HEADERS) + @echo "Archiving library: $@" + @mkdir -p $(@D) + $(AR) rcs $@ $^ + +vpath %.c $(DIST_DIR) + +$(BUILD_DIR)/%.o: $(DIST_DIR)/%.c + @echo "Compiling: $< -> $@" + @mkdir -p $(@D) + $(CC) $(CFLAGS) -MMD -MP -c $< -o $@ + +-include $(wildcard $(BUILD_DIR)/*.d) + +.PHONY: clean +clean: + @echo "Cleaning dist directory..." + @rm -rf $(DIST_DIR)" +`; diff --git a/lang/tests/compiler/compiler-esp32.test.ts b/lang/tests/compiler/compiler-esp32.test.ts index fc6f433e..c7dd6929 100644 --- a/lang/tests/compiler/compiler-esp32.test.ts +++ b/lang/tests/compiler/compiler-esp32.test.ts @@ -1,7 +1,10 @@ +import * as fs from 'fs'; +import * as path from 'path'; import { getEsp32CompilerConfig, Esp32CompilerTestEnv } from './test-utils'; import { CompilerSession } from '../../src/compiler/compiler-session'; -import { PackageForEsp32, Project } from '../../src/compiler/project'; +import { ProjectForEsp32 } from '../../src/compiler/project'; import { Esp32Toolchain } from '../../src/compiler/board-toolchain/esp32-toolchain'; +import { MemoryImage } from '../../src/compiler/board-toolchain/board-toolchain'; const memoryLayout = { iram: { address: 0x400a0144, size: 10000 }, @@ -12,19 +15,19 @@ const memoryLayout = { const compilerConfig = getEsp32CompilerConfig(); const compile = async (testEnv: Esp32CompilerTestEnv) => { - const project = Project.load( + const project = ProjectForEsp32.load( testEnv.mainPackageName, testEnv.getPackageReader() ); const toolchain = new Esp32Toolchain(compilerConfig, memoryLayout); - const session = new CompilerSession(toolchain); + const session = new CompilerSession(toolchain); await session.buildProject(project); return session; } describe('Test single compile: Compiler for ESP32', () => { - const testEnv = new Esp32CompilerTestEnv(); + const testEnv = new Esp32CompilerTestEnv('compiler-test-esp32'); beforeEach(() => { testEnv.init(); @@ -349,6 +352,22 @@ function foo() { expect(testEnv.resultElfExists()).toBe(true); }); + it('should compile index.bs which includes custom header file.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './add.h', 'int add(int a, int b);'); + testEnv.addSourceFile(testEnv.mainPackageName, './add.c', `#include "add.h"\nint add(int a, int b) {return a + b;}`); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', ` +code\`#include "add.h"\` +function foo() { + code\`add(1, 2);\` +} + `); + + await compile(testEnv); + expect(testEnv.resultElfExists()).toBe(true); + expect(fs.existsSync(path.join(testEnv.root, 'dist/add.h'))).toBe(true); + }); + it('should throw C compilation error.', async () => { testEnv.createMainPackage(); testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', ` @@ -404,9 +423,9 @@ describe('Test additional compile: Compiler for ESP32', () => { testEnv.init(); }); - // afterAll(() => { - // testEnv.delete(); - // }); + afterAll(() => { + testEnv.delete(); + }); it('should throw error if a file with a name consisting only numbers exists in main.', async () => { testEnv.createMainPackage(); @@ -455,6 +474,20 @@ describe('Test additional compile: Compiler for ESP32', () => { expect(binary.entryPoints.length).toBe(2); }); + it('should compile an additional code fragment with a package import.', async () => { + testEnv.createSubPackage('package1'); + testEnv.addSourceFile('package1', + './index.bs', + `export function add(a: integer, b:integer) {return a + b;}` + ); + testEnv.createMainPackage(['package1']); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `1 + 1`); + + const session = await compile(testEnv); + const binary = await session.compileFragment(`import {add} from 'package1';\n add(1, 1);`); + expect(binary.entryPoints.length).toBe(2); + }); + it('should compile several additional code fragments.', async () => { testEnv.createMainPackage(); testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', '1 + 1'); diff --git a/lang/tests/compiler/compiler-host.test.ts b/lang/tests/compiler/compiler-host.test.ts new file mode 100644 index 00000000..980afa91 --- /dev/null +++ b/lang/tests/compiler/compiler-host.test.ts @@ -0,0 +1,496 @@ +import * as path from "path"; +import * as fs from "fs"; +import { ProjectForHost } from "../../src/compiler/project"; +import { HostToolchain } from "../../src/compiler/board-toolchain/host-toolchain"; +import { HostCompilerTestEnv, runtimeDir } from "./test-utils"; +import { SharedObject } from "../../src/compiler/board-toolchain/board-toolchain"; +import { CompilerSession } from "../../src/compiler/compiler-session"; +import { executeCommand } from "../../src/compiler/utils"; + +const runtimeBuildDir = path.join(runtimeDir, 'ports/host/build'); +const builtinModuleC = path.join(runtimeDir, 'ports/host/std-module.c'); +const shellC = path.join(runtimeDir, 'ports/host/shell.c'); +const executableShell = path.join(runtimeBuildDir, 'shell'); +const runtimeSo = path.join(runtimeBuildDir, 'c-runtime.so'); +const runtimeC = path.join(runtimeDir, 'core/src/c-runtime.c'); + +const buildRuntime = async () => { + fs.mkdirSync(runtimeBuildDir, { recursive: true }); + await executeCommand('cc', ["-DLINUX64", "-O2", "-shared", "-fPIC", "-o", runtimeSo, runtimeC, builtinModuleC]); + await executeCommand('cc', ["-DLINUX64", "-O2", "-o", executableShell, shellC, runtimeSo, "-lm", "-ldl"]); +} + +const compile = async (testEnv: HostCompilerTestEnv) => { + const project = ProjectForHost.load( + testEnv.mainPackageName, + testEnv.getPackageReader() + ); + const toolchain = new HostToolchain(runtimeDir); + const session = new CompilerSession(toolchain); + await session.buildProject(project); + return session; +} + + +describe('Test single compile: Compiler for Host', () => { + const testEnv = new HostCompilerTestEnv('compiler-test-host'); + + beforeAll(async () => { + await buildRuntime(); + }) + + beforeEach(() => { + testEnv.init(); + }); + + afterAll(() => { + // testEnv.delete(); + }); + + + it('should compile simple index.bs.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', '1 + 1'); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should throw error if index.bs does not exist.', async () => { + testEnv.createMainPackage(); + + await expect(compile(testEnv)).rejects.toThrow(`Cannot find a module ${testEnv.getSourceFilePath(testEnv.mainPackageName, './index.bs')}`); + }); + + it('should compile index.bs with std function.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', 'print("hello world")'); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs with std object.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', 'console.log("hello world")'); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('can change source directory.', async () => { + testEnv.createMainPackage([], './src', './src/index.bs'); + testEnv.addSourceFile(testEnv.mainPackageName, './src/index.bs', 'print("hello world")'); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('can change entry file.', async () => { + testEnv.createMainPackage([], './src', './src/main.bs'); + testEnv.addSourceFile(testEnv.mainPackageName, './src/main.bs', 'print("hello world")'); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs with a module import.', async () => { + // index.bs <- ./module.bs + + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './module1.bs', `export function add(a: integer, b:integer) {return a + b}`); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `import {add} from './module1';\nadd(1, 2);`); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should throw error if an imported module does not exist.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `import {add} from './module1';\nadd(1, 2);`); + + await expect(compile(testEnv)).rejects.toThrow(`Cannot find a module ${testEnv.getSourceFilePath(testEnv.mainPackageName, './module1.bs')}`); + }); + + it('should throw error if an imported module is imported with absolute path.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './module1.bs', `export function add(a: integer, b:integer) {return a + b}`); + testEnv.addSourceFile(testEnv.mainPackageName, '/index.bs', `import {add} from '${testEnv.getSourceFilePath(testEnv.mainPackageName, './module1.bs')}';\nadd(1, 2);`); + + await expect(compile(testEnv)).rejects.toThrow(`This module system does not support importing from absolute paths.`); + }); + + it('should throw error if an imported module is imported with a path that is not under the source directory.', async () => { + testEnv.createMainPackage([], './src'); + testEnv.addSourceFile(testEnv.mainPackageName, './module1.bs', `export function add(a: integer, b:integer) {return a + b}`); + testEnv.addSourceFile(testEnv.mainPackageName, './src/index.bs', `import {add} from '../module1';\nadd(1, 2);`); + + await expect(compile(testEnv)).rejects.toThrow(`Source file must be under the source dir: module1.bs`); + }); + + it('should compile index.bs with module imports chained.', async () => { + // index.bs <- module1 <- module2 + + testEnv.createMainPackage([]); + testEnv.addSourceFile(testEnv.mainPackageName, + './module2.bs', + `export function add(a: integer, b:integer) {return a + b}` + ); + testEnv.addSourceFile(testEnv.mainPackageName, + './module1.bs', + `import {add} from './module2';\nexport function addMul(a: integer, b:integer) {return add(a, b)*add(a, b)}` + ); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `import {addMul} from './module1';\naddMul(1, 2);`); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs with module imports from dir.', async () => { + // index.bs <- dir/module1 <- dir/module2 + + testEnv.createMainPackage([]); + testEnv.addSourceFile(testEnv.mainPackageName, + './dir/module2.bs', + `export function add(a: integer, b:integer) {return a + b}` + ); + testEnv.addSourceFile(testEnv.mainPackageName, + './dir/module1.bs', + `import {add} from './module2';\nexport function addMul(a: integer, b:integer) {return add(a, b)*add(a, b)}` + ); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `import {addMul} from './dir/module1';\naddMul(1, 2);`); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs with a package import.', async () => { + // index.bs <- package1 + + testEnv.createSubPackage('package1'); + testEnv.addSourceFile('package1', './index.bs', `export function add(a: integer, b:integer) {return a + b}`); + testEnv.createMainPackage(['package1']); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `import {add} from 'package1';\nadd(1, 2);`); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs with a package import 2.', async () => { + // index.bs <- package1/module1 + + testEnv.createSubPackage('package1'); + testEnv.addSourceFile('package1', './module1.bs', `export function add(a: integer, b:integer) {return a + b}`); + testEnv.createMainPackage(['package1']); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `import {add} from 'package1/module1';\nadd(1, 2);`); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs with a package import from different source directory.', async () => { + testEnv.createSubPackage('package1', [], './src'); + testEnv.addSourceFile('package1', './src/index.bs', `export function add(a: integer, b:integer) {return a + b}`); + testEnv.createMainPackage(['package1']); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `import {add} from 'package1';\nadd(1, 2);`); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs with a package import from different source directory and different entry file.', async () => { + testEnv.createSubPackage('package1', [], './src', './src/main.bs'); + testEnv.addSourceFile('package1', './src/main.bs', `export function add(a: integer, b:integer) {return a + b}`); + testEnv.createMainPackage(['package1']); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `import {add} from 'package1';\nadd(1, 2);`); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs with an unused package.', async () => { + testEnv.createSubPackage('package1'); + testEnv.addSourceFile('package1', './index.bs', `export function add(a: integer, b:integer) {return a + b}`); + testEnv.createMainPackage(['package1']); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `1 + 1;`); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs with a package import 2.', async () => { + // index.bs <- package1/module1 + + testEnv.createSubPackage('package1'); + testEnv.addSourceFile('package1', './module1.bs', `export function add(a: integer, b:integer) {return a + b}`); + testEnv.createMainPackage(['package1']); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `import {add} from 'package1/module1';\nadd(1, 2);`); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs with some package imports.', async () => { + // index.bs <- package1 + // <- package2 + + testEnv.createSubPackage('package2'); + testEnv.addSourceFile('package2', './index.bs', `export function mul(a: integer, b:integer) {return a * b}`); + testEnv.createSubPackage('package1'); + testEnv.addSourceFile('package1', './index.bs', `export function add(a: integer, b:integer) {return a + b}`); + testEnv.createMainPackage(['package1', 'package2']); + testEnv.addSourceFile(testEnv.mainPackageName, + './index.bs', +`import {add} from 'package1'; +import {mul} from 'package2' +add(1, 2); +mul(1, 2);` + ); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should throw error if an imported package does not exist.', async () => { + testEnv.createMainPackage(['package1']); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `import {add} from 'package1';\nadd(1, 2);`); + + await expect(compile(testEnv)).rejects.toThrow(`Package package1 is not registered.`); + }); + + it('should compile index.bs with package imports chained.', async () => { + // index.bs <- package1 <- package2 + + testEnv.createSubPackage('package2'); + testEnv.addSourceFile('package2', + './index.bs', + `export function add(a: integer, b:integer) {return a + b}` + ); + testEnv.createSubPackage('package1', ['package2']); + testEnv.addSourceFile('package1', + './index.bs', + `import {add} from 'package2';\nexport function addMul(a: integer, b:integer) {return add(a, b)*add(a, b)}` + ); + testEnv.createMainPackage(['package1']); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `import {addMul} from 'package1';\naddMul(1, 2);`); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs which imports a package with a module import.', async () => { + // index.bs <- package1 <- package1/module1.bs + + testEnv.createSubPackage('package1'); + testEnv.addSourceFile('package1', './module1.bs', `export function mul(a: integer, b:integer) {return a * b}`); + testEnv.addSourceFile('package1', + './index.bs', + `import {mul} from './module1';\nexport function addMul(a: integer, b:integer) {return mul(a, b)+mul(a, b)}` + ); + testEnv.createMainPackage(['package1']); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `import {addMul} from 'package1';\naddMul(1, 2);`); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs which imports a package and a module.', async () => { + // index.bs <- package1 + // <- ./module1.bs + + testEnv.createSubPackage('package1'); + testEnv.addSourceFile('package1', + './index.bs', + `export function add(a: integer, b:integer) {return a + b;}` + ); + testEnv.createMainPackage(['package1']); + testEnv.addSourceFile(testEnv.mainPackageName, './module1.bs', `export function mul(a: integer, b:integer) {return a * b}`); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', ` +import { add } from 'package1'; +// import { mul } from './module1'; + +add(1, 2); +// mul(1,2); + `); + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should treat a class imported via different routes as the same class.', async () => { + // package1 <- package2 + // index.bs <- package1 + // <- package2 + // Shape from package2 in package1 and Shape from package2 in main should be treated as same class. + + testEnv.createSubPackage('package2'); + testEnv.addSourceFile('package2', './index.bs', `export class Shape {constructor() {}}`); + testEnv.createSubPackage('package1', ['package2']); + testEnv.addSourceFile('package1', './index.bs', ` +import { Shape } from 'package2'; +export function getShapeArea(shape: Shape) {return 110;} +`); + testEnv.createMainPackage(['package1', 'package2']); + testEnv.addSourceFile(testEnv.mainPackageName, + './index.bs', +`import { getShapeArea } from 'package1'; +import { Shape } from 'package2'; + +const shape = new Shape(); +getShapeArea(shape); +` + ); + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs which includes c file.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', ` +code\`#include\` +function foo() { + code\`puts("foo");\` +} + `); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs which includes custom c file.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './add.c', 'int add(int a, int b) {return a + b;}') + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', ` +code\`#include "./add.c"\` +function foo() { + code\`add(1, 2);\` +} + `); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + + it('should compile index.bs which includes custom header file.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './add.h', 'int add(int a, int b);'); + testEnv.addSourceFile(testEnv.mainPackageName, './add.c', `#include "add.h"\nint add(int a, int b) {return a + b;}`); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', ` +code\`#include "add.h"\` +function foo() { + code\`add(1, 2);\` +} + `); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + expect(fs.existsSync(path.join(testEnv.root, 'dist/add.h'))).toBe(true); + }); + + it('should throw C compilation error.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', ` +function foo() { + code\`puts("foo");\` +} + `); + + await expect(compile(testEnv)).rejects.toThrow(`do not support implicit function declarations`); + }); + + it('should compile index.bs again after editing file.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', '1 + 1'); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', '1 + 3'); + + await compile(testEnv); + expect(testEnv.resultSharedObjectExists()).toBe(true); + }); + +}); + + +describe('Test additional compile: Compiler for ESP32', () => { + const testEnv = new HostCompilerTestEnv('compiler-test-host'); + + beforeEach(() => { + testEnv.init(); + }); + + afterAll(() => { + testEnv.delete(); + }); + + it('should throw error if a file with a name consisting only numbers exists in main.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', '1 + 1'); + testEnv.addSourceFile(testEnv.mainPackageName, './1.bs', '1 + 1'); + + await expect(compile(testEnv)).rejects.toThrow(`Invalid file name`); + }); + + it('should compile an additional code fragment.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', '1 + 1'); + + const session = await compile(testEnv); + const sharedObjest = await session.compileFragment('1 + 23'); + expect(sharedObjest.entryNames.length).toBe(1); + }); + + it('should compile an additional code fragment with function call.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', 'function add(a, b) {return a + b}'); + + const session = await compile(testEnv); + const sharedObjest = await session.compileFragment('add(2, 3);'); + expect(sharedObjest.entryNames.length).toBe(1); + }); + + it('should compile an additional code fragment with variable access', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', 'let a = 1 + 1;'); + + const session = await compile(testEnv); + const sharedObjest = await session.compileFragment('a += 1;'); + expect(sharedObjest.entryNames.length).toBe(1); + }); + + it('should compile an additional code fragment with a module import.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './module1.bs', `export function add(a: integer, b:integer) {return a + b}`); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `1 + 1`); + + const session = await compile(testEnv); + const sharedObjest = await session.compileFragment(`import {add} from './module1';\n add(1, 1);`); + expect(sharedObjest.entryNames.length).toBe(2); + }); + + it('should compile an additional code fragment with a package import.', async () => { + testEnv.createSubPackage('package1'); + testEnv.addSourceFile('package1', + './index.bs', + `export function add(a: integer, b:integer) {return a + b;}` + ); + testEnv.createMainPackage(['package1']); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', `1 + 1`); + + const session = await compile(testEnv); + const sharedObjest = await session.compileFragment(`import {add} from 'package1';\n add(1, 1);`); + expect(sharedObjest.entryNames.length).toBe(2); + }); + + it('should compile several additional code fragments.', async () => { + testEnv.createMainPackage(); + testEnv.addSourceFile(testEnv.mainPackageName, './index.bs', '1 + 1'); + + const session = await compile(testEnv); + let sharedObjest = await session.compileFragment('function add(a, b) {return a + b}'); + expect(sharedObjest.entryNames.length).toBe(1); + sharedObjest = await session.compileFragment('add(1, 2);'); + expect(sharedObjest.entryNames.length).toBe(1); + }); +}); diff --git a/lang/tests/compiler/makefile.test.ts b/lang/tests/compiler/makefile.test.ts new file mode 100644 index 00000000..beac149c --- /dev/null +++ b/lang/tests/compiler/makefile.test.ts @@ -0,0 +1,43 @@ +import { describe, expect, test } from '@jest/globals'; +import { generateMakefile, esp32MakefilePreset, hostMakefilePreset } from '../../src/compiler/board-toolchain/tools/makefile'; +import { Package } from '../../src/compiler/project'; + +function createTestPackage(name = 'myapp', rootDir = '/project/myapp'): Package { + return new Package( + name, + { + rootDir, + entry: './src/index.bs', + sourceDir: './src', + distDir: './dist', + buildDir: './dist/build', + packageDir: './packages', + }, + [], + ); +} + +describe('generateMakefile', () => { + test('generates makefile for esp32 archive file.', () => { + const pkg = createTestPackage(); + const makefile = generateMakefile(esp32MakefilePreset( + '/opt/esp-toolchain', + pkg, + [ + '/opt/esp-idf/components/freertos/include', + '/opt/esp-idf/components/esp_common/include', + ], + '/project/myapp/dist/build/libmyapp.a', + )); + expect(makefile).toMatchSnapshot(); + }); + + test('generates makefile for host shared library.', () => { + const pkg = createTestPackage(); + const makefile = generateMakefile(hostMakefilePreset( + pkg, + '/project/myapp/dist/build/libmyapp.a', + )); + expect(makefile).toMatchSnapshot(); + }); +}); diff --git a/lang/tests/compiler/test-utils.ts b/lang/tests/compiler/test-utils.ts index d8a79b81..b71470f9 100644 --- a/lang/tests/compiler/test-utils.ts +++ b/lang/tests/compiler/test-utils.ts @@ -5,11 +5,13 @@ import * as fs from 'fs'; import * as path from 'path'; import * as os from 'os'; +export const runtimeDir = path.resolve(__dirname, '../../../microcontroller'); + export function getEsp32CompilerConfig(): Esp32ToolchainConfig { const gccPath = execSync('source ~/esp/esp-idf/export.sh &> /dev/null && which xtensa-esp32-elf-gcc').toString(); const espDir = path.join(os.homedir(), 'esp'); return { - runtimeDir: path.resolve(__dirname, '../../../microcontroller'), + runtimeDir: runtimeDir, compilerToolchainDir: path.resolve(gccPath, '../'), espDir } @@ -17,14 +19,12 @@ export function getEsp32CompilerConfig(): Esp32ToolchainConfig { class CompilerTestEnv

{ readonly root: string; - readonly resultElf: string; public readonly mainPackageName: string; protected packages = new Map(); constructor(name?: string) { this.mainPackageName = name ?? 'compiler-test'; this.root = path.resolve(__dirname, `../../temp-files/${this.mainPackageName}`); - this.resultElf = path.join(this.root, `dist/build/${this.mainPackageName}.elf`); } protected addPackage(pkg: P) { @@ -56,10 +56,6 @@ class CompilerTestEnv

{ } } - public resultElfExists() { - return fs.existsSync(this.resultElf); - } - public init() { this.delete(); fs.mkdirSync(this.root, { recursive: true }); @@ -107,4 +103,50 @@ export class Esp32CompilerTestEnv extends CompilerTestEnv { ) this.addPackage(pkg); } + + public resultElfExists() { + const elfPath = path.join(this.root, `dist/build/${this.mainPackageName}.elf`); + return fs.existsSync(elfPath); + } +} + +export class HostCompilerTestEnv extends CompilerTestEnv { + public createMainPackage(dependencies: string[] = [], srcDir: string = ".", entryFile?: string): void { + const pkg = new Package( + this.mainPackageName, + { + rootDir: this.root, + entry: entryFile ?? path.join(srcDir, 'index.bs'), + sourceDir: srcDir, + distDir: "./dist", + buildDir: "./dist/build", + packageDir: "./packages", + }, + dependencies, + ); + this.addPackage(pkg); + } + + public createSubPackage(name: string, dependencies: string[] = [], srcDir: string = ".", entryFile?: string): void { + const root = path.join(this.root, 'packages', name); + fs.mkdirSync(root, {recursive: true}); + const pkg = new Package( + name, + { + rootDir: root, + entry: entryFile ?? path.join(srcDir, 'index.bs'), + sourceDir: srcDir, + distDir: "./dist", + buildDir: "./dist/build", + packageDir: "./packages", + }, + dependencies, + ) + this.addPackage(pkg); + } + + public resultSharedObjectExists() { + const soPath = path.join(this.root, `dist/build/${this.mainPackageName}.so`); + return fs.existsSync(soPath); + } } \ No newline at end of file diff --git a/microcontroller/core/include/protocol.h b/microcontroller/core/include/protocol.h index 0ee9e121..7e7e8c55 100644 --- a/microcontroller/core/include/protocol.h +++ b/microcontroller/core/include/protocol.h @@ -1,14 +1,12 @@ #ifndef __BS_PROTOCOL__ #define __BS_PROTOCOL__ - #include #include "memory.h" #include "ble.h" - #define CORE_TEXT_SECTION __attribute__((section(".core_text"))) - #define BS_PROTOCL_USE_BLUETOOTH + void CORE_TEXT_SECTION bs_protocol_init(void); void CORE_TEXT_SECTION bs_protocol_write_log(char* message); diff --git a/microcontroller/core/src/protocol.c b/microcontroller/core/src/protocol.c index d15b5f53..a4b546a7 100644 --- a/microcontroller/core/src/protocol.c +++ b/microcontroller/core/src/protocol.c @@ -1,12 +1,11 @@ #include #include - #include "utils.h" #include "memory.h" - #include "../include/protocol.h" #include "../include/main-thread.h" + #define PROTOCOL_LEN 1 typedef enum { @@ -24,6 +23,7 @@ typedef enum { PROTOCOL_END } protocol_t; + static void send_buffer(uint8_t* buffer, uint32_t len) { #ifdef BS_PROTOCL_USE_BLUETOOTH bs_ble_send_buffer(buffer, len); diff --git a/microcontroller/ports/host/comm.c b/microcontroller/ports/host/comm.c new file mode 100644 index 00000000..184bfc73 --- /dev/null +++ b/microcontroller/ports/host/comm.c @@ -0,0 +1,99 @@ +#include +#include +#include +#include +#include "./comm.h" + + +static void comm_send(host_protocol_t protocol, char* payload) { + char line[MAX_LINE_SIZE]; + snprintf(line, PROTO_SIZE, "%02d", protocol); + line[PROTO_SIZE - 1] = ' '; + snprintf((char*)(line + PROTO_SIZE), PAYLOAD_LEN_SIZE, "%04d", (int)strlen(payload)); + line[HEADER_SIZE - 1] = ' '; + strcpy((char*)(line + HEADER_SIZE), payload); + fprintf(stdout, line); + fflush(stdout); +} + +void bs_comm_send_log(char* message) { + comm_send(H_PROTOCOL_LOG, message); +} + +void bs_comm_send_error(char* message) { + comm_send(H_PROTOCOL_ERROR, message); +} + +void bs_comm_send_exectime(float time) { + char* timestr[16]; + snprintf(timestr, sizeof(timestr), "%.4f", time); + comm_send(H_PROTOCOL_EXECTIME, timestr); +} + +void bs_comm_send_loadtime(float time) { + char* timestr[16]; + snprintf(timestr, sizeof(timestr), "%.2f", time); + comm_send(H_PROTOCOL_LOADTIME, timestr); +} + +static void parse_line(char* line, host_protocol_t* protocol, char* payload) { + char protocol_char[PROTO_SIZE]; + protocol_char[0] = line[0]; + protocol_char[1] = line[1]; + protocol_char[2] = NULL; + char payload_len_char[PAYLOAD_LEN_SIZE]; + payload_len_char[0] = line[PROTO_SIZE + 0]; + payload_len_char[1] = line[PROTO_SIZE + 1]; + payload_len_char[2] = line[PROTO_SIZE + 2]; + payload_len_char[3] = line[PROTO_SIZE + 3]; + payload_len_char[4] = NULL; + *protocol = atoi(protocol_char); + int payload_len = atoi(payload_len_char); + for (int i = 0; i < payload_len; i++) { + if (line[HEADER_SIZE + i] == 0x0a || line[HEADER_SIZE + i] == 0x0d) + payload[i] = '\0'; + else + payload[i] = line[HEADER_SIZE + i]; + } +} + +static char* getoneline(char* buffer, int size) { + char* res; + if ((res = fgets(buffer, size, stdin)) == NULL) { + buffer[0] = '\0'; + return NULL; + } + else { + for (int i = 0; i < size; i++) + if (buffer[i] == 0x0a || buffer[i] == 0x0d) + buffer[i] = '\0'; + return res; + } +} + +char* bs_comm_wait_receive(void (*on_load)(char* filename), void (*on_call)(char* funcname)) { + char line[MAX_LINE_SIZE] = {0}; + char* res = getoneline(line, MAX_LINE_SIZE); + if (res == NULL) + return NULL; + + int protocol; + char payload[MAX_PAYLOAD_SIZE] = {0}; + parse_line(line, &protocol, payload); + + switch (protocol) { + case H_PROTOCOL_LOAD: + on_load(payload); + break; + case H_PROTOCOL_CALL: + on_call(payload); + break; + default: + fprintf(stderr, "Error: unknown protocol\n"); + break; + } + return res; +} + + + diff --git a/microcontroller/ports/host/comm.h b/microcontroller/ports/host/comm.h new file mode 100644 index 00000000..e7f5ecb1 --- /dev/null +++ b/microcontroller/ports/host/comm.h @@ -0,0 +1,32 @@ +#ifndef __BS_HOST_COMM__ +#define __BS_HOST_COMM__ + +#include + +#define MAX_PAYLOAD_SIZE 128 +#define PROTO_SIZE 3 +#define PAYLOAD_LEN_SIZE 5 +#define HEADER_SIZE PROTO_SIZE + PAYLOAD_LEN_SIZE +#define MAX_LINE_SIZE HEADER_SIZE + MAX_PAYLOAD_SIZE + +typedef enum { + H_PROTOCOL_NONE = 0, + H_PROTOCOL_LOAD = 1, + H_PROTOCOL_CALL = 2, + + H_PROTOCOL_LOG = 3, + H_PROTOCOL_ERROR = 4, + H_PROTOCOL_EXECTIME = 5, + H_PROTOCOL_LOADTIME = 6, + + H_PROTOCOL_MAX +} host_protocol_t; + +void bs_comm_send_log(char* message); +void bs_comm_send_error(char* message); +void bs_comm_send_exectime(float time); +void bs_comm_send_loadtime(float time); +char* bs_comm_wait_receive(void (*on_load)(char* filename), void (*on_call)(char* funcname)); + + +#endif /* __BS_HOST_COMM__ */ \ No newline at end of file diff --git a/microcontroller/ports/host/shell.c b/microcontroller/ports/host/shell.c new file mode 100644 index 00000000..3d7114d2 --- /dev/null +++ b/microcontroller/ports/host/shell.c @@ -0,0 +1,59 @@ +// Copyright (C) 2024- Shigeru Chiba. All rights reserved. + +#include +#include +#include +#include +#include +#include "../../core/include/c-runtime.h" +#include "./comm.h" + + +extern void bluescript_main0_(); + +void* file_handle; + +static float get_time_ms() { + static struct timespec ts0 = { 0, -1 }; + struct timespec ts; + if (ts0.tv_nsec < 0) + clock_gettime(CLOCK_REALTIME, &ts0); + + clock_gettime(CLOCK_REALTIME, &ts); + return (float)(ts.tv_sec - ts0.tv_sec) * 1000.0 + (float)(ts.tv_nsec - ts0.tv_nsec) / 1000000.0; +} + +static void load(char* filename) { + float start_time = get_time_ms(); + file_handle = dlopen(filename, RTLD_NOW | RTLD_GLOBAL); + bs_comm_send_loadtime(get_time_ms() - start_time); +} + +static int call(char* funcname) { + if (file_handle == NULL) { + fprintf(stderr, "Error: module is not loaded\n"); + return 1; + } + void (*fptr)() = dlsym(file_handle, funcname); + if (fptr == NULL) { + fprintf(stderr, "Error: %s() is not found\n", funcname); + return 1; + } else { + float start_time = get_time_ms(); + int r2 = try_and_catch(fptr); + bs_comm_send_exectime(get_time_ms() - start_time); + return r2; + } +} + +int main() { + gc_initialize(); + bluescript_main0_(); + + while (bs_comm_wait_receive(load, call) != NULL) { + fflush(stdout); + fflush(stderr); + } + + return 0; +} diff --git a/microcontroller/ports/host/std-module.bs b/microcontroller/ports/host/std-module.bs new file mode 100644 index 00000000..6a0c504f --- /dev/null +++ b/microcontroller/ports/host/std-module.bs @@ -0,0 +1,78 @@ +export type integer = number; +export type float = number; +export function code(strings: any, ... keys: any[]) {} + +code` +#include +#include +#include +#include +#include "../../core/include/c-runtime.h" +#include "./comm.h" + + +void send_message(const char* format, ...) { + static char message[MAX_LINE_SIZE]; + va_list list; + va_start(list, format); + vsnprintf(message, MAX_LINE_SIZE, format, list); + bs_comm_send_log(message); +} + +void print_message(value_t m) { + static char buffer[256]; + if (is_int_value(m)) + send_message("%d\n", value_to_int(m)); + else if (is_float_value(m)) + send_message("%f\n", value_to_float(m)); + else if (m == VALUE_NULL || m == VALUE_UNDEF) + send_message("undefined\n"); + else if (m == VALUE_TRUE) + send_message("true\n"); + else if (m == VALUE_FALSE) + send_message("false\n"); + else if (gc_is_string_object(m)) + send_message("'%s'\n", gc_string_to_cstr(m)); + else { + class_object* cls = gc_get_class_of(m); + if (cls == NULL) + send_message("??\n"); + else + send_message("\n", cls->name); + } +} +` + +function print(message: any) { + code`print_message(${message});` +} + +class Console { + log(message: any) { + code`print_message(${message});` + } + + error(message: any) { + code`print_message(${message});` + } +} + + +class Time { + now(): float { + let t: integer = 0 + code` + static struct timespec ts0 = { 0, -1 }; + struct timespec ts; + if (ts0.tv_nsec < 0) + clock_gettime(CLOCK_REALTIME, &ts0); + + clock_gettime(CLOCK_REALTIME, &ts); + ${t} = (int32_t)((ts.tv_sec - ts0.tv_sec) * 1000 + (ts.tv_nsec - ts0.tv_nsec) / 1000000); + ` + return t + } +} + +const console = new Console(); +const time = new Time(); diff --git a/microcontroller/ports/host/std-module.c b/microcontroller/ports/host/std-module.c new file mode 100644 index 00000000..3dbdbf4e --- /dev/null +++ b/microcontroller/ports/host/std-module.c @@ -0,0 +1,123 @@ + +#include +#include +#include +#include +#include "../../core/include/c-runtime.h" +#include "./comm.h" + +void send_message(const char* format, ...) { + static char message[MAX_LINE_SIZE]; + va_list list; + va_start(list, format); + vsnprintf(message, MAX_LINE_SIZE, format, list); + bs_comm_send_log(message); +} + +void print_message(value_t m) { + static char buffer[256]; + if (is_int_value(m)) + send_message("%d\n", value_to_int(m)); + else if (is_float_value(m)) + send_message("%f\n", value_to_float(m)); + else if (m == VALUE_NULL || m == VALUE_UNDEF) + send_message("undefined\n"); + else if (m == VALUE_TRUE) + send_message("true\n"); + else if (m == VALUE_FALSE) + send_message("false\n"); + else if (gc_is_string_object(m)) + send_message("'%s'\n", gc_string_to_cstr(m)); + else { + class_object* cls = gc_get_class_of(m); + if (cls == NULL) + send_message("??\n"); + else + send_message("\n", cls->name); + } +} + + +extern struct func_body _print; +void mth_0_Console(value_t self, value_t _message); +void mth_1_Console(value_t self, value_t _message); +float mth_0_Time(value_t self); +extern CLASS_OBJECT(object_class, 1); +void bluescript_main0_(); +ROOT_SET_DECL(global_rootset0, 2); +static const uint16_t mnames_Console[] = { 8, 9, }; +static const char* const msigs_Console[] = { "(a)v", "(a)v", }; +static const uint16_t plist_Console[] = { }; +CLASS_OBJECT(class_Console, 2) = { + .body = { .s = 0, .i = 0, .cn = "Console", .sc = &object_class.clazz , .an = (void*)0, .pt = { .size = 0, .offset = 0, + .unboxed = 0, .prop_names = plist_Console, .unboxed_types = "" }, .mt = { .size = 2, .names = mnames_Console, .signatures = msigs_Console }, .vtbl = { mth_0_Console, mth_1_Console, }}}; +static const uint16_t mnames_Time[] = { 10, }; +static const char* const msigs_Time[] = { "()f", }; +static const uint16_t plist_Time[] = { }; +CLASS_OBJECT(class_Time, 1) = { + .body = { .s = 0, .i = 0, .cn = "Time", .sc = &object_class.clazz , .an = (void*)0, .pt = { .size = 0, .offset = 0, + .unboxed = 0, .prop_names = plist_Time, .unboxed_types = "" }, .mt = { .size = 1, .names = mnames_Time, .signatures = msigs_Time }, .vtbl = { mth_0_Time, }}}; + +static void fbody_print(value_t self, value_t _message) { + ROOT_SET_N(func_rootset,2,VALUE_UNDEF_2) + func_rootset.values[1] = self; + func_rootset.values[0] = _message; + { + print_message(func_rootset.values[0]);; + } + DELETE_ROOT_SET(func_rootset) +} +struct func_body _print = { fbody_print, "(a)v" }; + +void mth_0_Console(value_t self, value_t _message) { + ROOT_SET_N(func_rootset,2,VALUE_UNDEF_2) + func_rootset.values[0] = self; + func_rootset.values[1] = _message; + { + print_message(func_rootset.values[1]);; + } + DELETE_ROOT_SET(func_rootset) +} + +void mth_1_Console(value_t self, value_t _message) { + ROOT_SET_N(func_rootset,2,VALUE_UNDEF_2) + func_rootset.values[0] = self; + func_rootset.values[1] = _message; + { + print_message(func_rootset.values[1]);; + } + DELETE_ROOT_SET(func_rootset) +} + +value_t new_Console(value_t self) { return self; } + + +float mth_0_Time(value_t self) { + ROOT_SET_N(func_rootset,1,VALUE_UNDEF) + func_rootset.values[0] = self; + { + int32_t _t = 0; + + static struct timespec ts0 = { 0, -1 }; + struct timespec ts; + if (ts0.tv_nsec < 0) + clock_gettime(CLOCK_REALTIME, &ts0); + + clock_gettime(CLOCK_REALTIME, &ts); + _t = (int32_t)((ts.tv_sec - ts0.tv_sec) * 1000 + (ts.tv_nsec - ts0.tv_nsec) / 1000000); + ; + { float ret_value_ = (_t); DELETE_ROOT_SET(func_rootset); return ret_value_; } + } +} + +value_t new_Time(value_t self) { return self; } + + +void bluescript_main0_() { + ROOT_SET_INIT(global_rootset0, 2) + ROOT_SET_N(func_rootset,1,VALUE_UNDEF) + ; + set_global_variable(&global_rootset0.values[0], new_Console(func_rootset.values[0]=gc_new_object(&class_Console.clazz))); + set_global_variable(&global_rootset0.values[1], new_Time(func_rootset.values[0]=gc_new_object(&class_Time.clazz))); + DELETE_ROOT_SET(func_rootset) +} diff --git a/package-lock.json b/package-lock.json index 3ed2cd8f..7ef8ed20 100644 --- a/package-lock.json +++ b/package-lock.json @@ -21540,7 +21540,6 @@ "dependencies": { "@ant-design/colors": "^7.1.0", "@ant-design/icons": "^5.5.1", - "@bscript/lang": "2.0.2", "@codemirror/lang-javascript": "^6.2.4", "@codemirror/view": "^6.38.5", "@testing-library/jest-dom": "^5.16.5", diff --git a/website/docs/reference/bsconfig.md b/website/docs/reference/bsconfig.md index c0f34384..9713a596 100644 --- a/website/docs/reference/bsconfig.md +++ b/website/docs/reference/bsconfig.md @@ -27,12 +27,12 @@ These fields are shared across all supported boards. | Field | Required | Default | Description | | :--- | :---: | :--- | :--- | | `projectName` | Yes | — | Project name. Also used as the main package name during compilation. | -| `boardName` | Yes | — | Target board. Currently supported: `esp32`. | +| `boardName` | Yes | — | Target board. Supported: `esp32`, `host`. | | `version` | No | `"1.0.0"` | Project version string. | | `vmVersion` | No | CLI version | BlueScript runtime version this project targets. | | `srcDir` | No | `"."` | Directory containing BlueScript (`.bs`) and C (`.c`) source files, relative to the project root. | | `entryFile` | No | `"./index.bs"` | Entry BlueScript file executed by `bscript project run`, relative to the project root. | -| `deviceName` | No | `"BLUESCRIPT"` | Bluetooth device name used when scanning for hardware. | +| `deviceName` | No | `"BLUESCRIPT"` | Bluetooth device name used when scanning for hardware. **ESP32 only** — ignored for `host` projects. | | `dependencies` | No | `{}` | Installed package dependencies. Usually managed by `bscript project install`. | ### `srcDir` and `entryFile` @@ -72,7 +72,7 @@ Use `bscript project install` to add packages instead of editing this field by h ## ESP32 fields -When `boardName` is `"esp32"`, the following additional field is available. +When `boardName` is `"esp32"`, the following additional field is available. These fields do not apply to `host` projects. | Field | Required | Default | Description | | :--- | :---: | :--- | :--- | @@ -89,6 +89,24 @@ When `boardName` is `"esp32"`, the following additional field is available. See the [Inline C tutorial](../tutorial/guides/inline-c.md) for usage examples. +## Host example + +A minimal `bsconfig.json` for the host runtime: + +```json title="bsconfig.json (host)" +{ + "projectName": "hello-host", + "boardName": "host", + "version": "1.0.0", + "vmVersion": "2.0.0", + "srcDir": "./src", + "entryFile": "./src/index.bs", + "dependencies": {} +} +``` + +See [Try Without Microcontroller](../tutorial/guides/try-without-microcontroller.md) for setup steps. + ## Generated project layout A project created with `bscript project create` looks like this: diff --git a/website/docs/reference/cli.md b/website/docs/reference/cli.md index 4ec4c9f1..2f5cb92a 100644 --- a/website/docs/reference/cli.md +++ b/website/docs/reference/cli.md @@ -29,7 +29,7 @@ This command generates a new directory containing: | Option | Alias | Description | | :--- | :--- | :--- | -| `--board` | `-b` | Specify the target board (e.g., `esp32`). If omitted, an interactive selection list will appear. | +| `--board` | `-b` | Specify the target board (`esp32` or `host`). If omitted, an interactive selection list will appear. | **Example:** ```bash @@ -38,6 +38,9 @@ bscript project create my-app # Create a project specifically for ESP32 bscript project create my-app --board esp32 + +# Create a project for the host runtime (no hardware) +bscript project create my-app --board host ``` --- @@ -105,17 +108,19 @@ This command runs the compiler locally on your host machine to verify for syntax ### `bscript project run` -Compiles the current project and executes it on a target device via Bluetooth. +Compiles the current project and executes it on the target board. ```bash bscript project run [options] ``` -When you run this command: +When you run this command on an **ESP32** project: 1. The CLI scans for available BlueScript devices over Bluetooth. 2. The project is compiled into native code on your host machine. 3. The code is transferred and executed immediately. +When you run this command on a **host** project, the CLI compiles the project and runs it in a local runtime process on your development machine. No Bluetooth connection is required. + **Options:** | Option | Description | @@ -141,21 +146,23 @@ bscript board setup ``` **Arguments:** -* ``: The target board identifier (e.g., `esp32`). +* ``: The target board identifier (`esp32` or `host`). + +For `esp32`, this downloads ESP-IDF and related tools. For `host`, this builds the local runtime process and requires a C compiler toolchain (`cc` and `make`). See [Try Without Microcontroller](../tutorial/guides/try-without-microcontroller.md). --- ### `bscript board flash-runtime` Flashes the BlueScript Runtime firmware onto the microcontroller. -**Note:** This command requires a physical USB connection to the device. +**Note:** This command requires a physical USB connection to the device. It is **not supported** for `host`. ```bash bscript board flash-runtime [options] ``` **Arguments:** -* ``: The target board identifier. +* ``: The target board identifier (e.g., `esp32`). **Options:** @@ -172,7 +179,7 @@ bscript board flash-runtime esp32 --port /dev/ttyUSB0 ### `bscript board list` -Lists all board architectures currently supported by the installed CLI version. +Lists all board architectures currently supported by the installed CLI version (`esp32` and `host`). ```bash bscript board list @@ -191,7 +198,7 @@ bscript board remove [options] By default, this command asks for confirmation before deleting files. **Arguments:** -* ``: The target board identifier (e.g., `esp32`). +* ``: The target board identifier (`esp32` or `host`). **Options:** @@ -245,4 +252,4 @@ This mode is for language syntax experiments only. Hardware libraries installed | Option | Alias | Description | | :--- | :--- | :--- | -| `--board` | `-b` | Specify the target board (e.g., `esp32`). | +| `--board` | `-b` | Specify the target board (`esp32` or `host`). | diff --git a/website/docs/reference/libraries/builtin.md b/website/docs/reference/libraries/builtin.md index bea9f322..4958da9a 100644 --- a/website/docs/reference/libraries/builtin.md +++ b/website/docs/reference/libraries/builtin.md @@ -82,6 +82,10 @@ console.log(current); Synchronously pauses the program execution for a specified duration. +:::note ESP32 only +`time.delay` is available on **ESP32** only. It is not available on the host runtime. +::: + **Parameters** - `ms` (integer): The number of milliseconds to wait. diff --git a/website/docs/reference/libraries/standard.md b/website/docs/reference/libraries/standard.md index c1eedf9f..d372776b 100644 --- a/website/docs/reference/libraries/standard.md +++ b/website/docs/reference/libraries/standard.md @@ -6,6 +6,10 @@ Core features are kept minimal, while hardware drivers are provided as external To install any of these libraries, use the command: `bscript project install ` +:::note ESP32 only +The libraries listed below are for **ESP32** hardware. They are not available on the host runtime. +::: + ## Available Libraries Currently, the following libraries are available for stable use. diff --git a/website/docs/tutorial/get-started/create-project-and-run.md b/website/docs/tutorial/get-started/create-project-and-run.md index dc28da33..0d8576df 100644 --- a/website/docs/tutorial/get-started/create-project-and-run.md +++ b/website/docs/tutorial/get-started/create-project-and-run.md @@ -59,4 +59,8 @@ We are planning to introduce a **`bscript project deploy`** command. This comman Want to experiment without editing files on every try? After you have a project, run **`bscript project run --with-notebook`** for a browser-based Notebook, or see [REPL & Notebook](../guides/repl.md) for all interactive modes. ::: +:::note No microcontroller? +To run BlueScript on the host runtime without an ESP32, see [Try Without Microcontroller](../guides/try-without-microcontroller.md). +::: + --- diff --git a/website/docs/tutorial/get-started/introduction.md b/website/docs/tutorial/get-started/introduction.md index 43de19ee..01767527 100644 --- a/website/docs/tutorial/get-started/introduction.md +++ b/website/docs/tutorial/get-started/introduction.md @@ -54,9 +54,12 @@ The architecture of BlueScript is based on the research paper *["BlueScript: A D ## Supported Hardware -Currently, BlueScript supports the following platform: -- Espressif ESP32 (Supported) +BlueScript targets microcontroller development first. The primary supported platform is: + +- **Espressif ESP32** — real hardware development over Bluetooth + +You can also run BlueScript on the **host runtime** without a microcontroller. This is useful for language checks and quick experiments. See [Try Without Microcontroller](../guides/try-without-microcontroller.md). :::note Future Roadmap -We plan to support other boards in the future. +We plan to support additional microcontroller boards in the future. ::: \ No newline at end of file diff --git a/website/docs/tutorial/get-started/setup-environment.md b/website/docs/tutorial/get-started/setup-environment.md index c8abd70b..70d00368 100644 --- a/website/docs/tutorial/get-started/setup-environment.md +++ b/website/docs/tutorial/get-started/setup-environment.md @@ -4,9 +4,7 @@ Currently, BlueScript strictly requires **macOS**. Windows and Linux support is under development. ::: -In this guide, we will install the BlueScript CLI and flash the runtime environment to your microcontroller. - -Currently, only **ESP32 development boards** are supported. +In this guide, we will install the BlueScript CLI and flash the runtime environment to your ESP32 microcontroller. ## Prerequisites @@ -51,7 +49,6 @@ Download the necessary environment files for the ESP32 platform: ```bash bscript board setup esp32 ``` -*Note: Currently, only `esp32` is supported.* ### 2. Flash the Runtime @@ -70,3 +67,7 @@ See also [Establish Serial Connection with ESP32](https://docs.espressif.com/pro ::: If the flash is successful, your device is now ready to receive BlueScript code wirelessly! + +:::note No microcontroller? +If you want to try BlueScript without hardware, see [Try Without Microcontroller](../guides/try-without-microcontroller.md). +::: diff --git a/website/docs/tutorial/guides/imports-and-includes.md b/website/docs/tutorial/guides/imports-and-includes.md index 0fd31b24..85ef00c5 100644 --- a/website/docs/tutorial/guides/imports-and-includes.md +++ b/website/docs/tutorial/guides/imports-and-includes.md @@ -1,7 +1,7 @@ # Imports & Includes As your project grows, you will want to split your code into multiple files. -BlueScript handles dependencies differently depending on whether you are loading **BlueScript Modules** or raw **C Source Files**. +BlueScript handles dependencies differently depending on whether you are loading **BlueScript Modules** or local **C / header files**. ## Importing BlueScript Modules @@ -10,7 +10,7 @@ You can create reusable BlueScript code (`.bs` files) and import them into other ### 1. Local Modules To import a module from your own project, use the **relative path** (starting with `./` or `../`). -BlueScript and C source files must be placed under the `srcDir` directory configured in [bsconfig.json](../../reference/bsconfig.md). Relative import paths must resolve to files inside `srcDir`. +BlueScript, C (`.c`), and header (`.h`) files must be placed under the `srcDir` directory configured in [bsconfig.json](../../reference/bsconfig.md). Relative import paths must resolve to files inside `srcDir`. **`src/math-utils.bs`** (The library) ```typescript @@ -45,15 +45,27 @@ When you install an external library (like a driver), you import it by its **Pac import { GPIO } from "gpio"; ``` -## Including C Files +## Including C and Header Files -If you have standalone C source files (`.c`) in your project, you can include them using **Inline C**. +You can use local `.c` and `.h` files in your project via **Inline C** `#include` directives. +Place them under `srcDir`, alongside your `.bs` files. -Unlike standard C compilers, BlueScript's `code` block treats `#include` paths relative to the current file when using quotes. +Unlike system headers, paths in **quotes** are resolved **relative to the current `.bs` file** (same rule as local BlueScript imports). +For details on writing C code inside `code` blocks, see [Inline C](./inline-c.md). + +```text +src/ + index.bs + driver.h # C function declarations + driver.c # C function implementations +``` + +### Option 1: Include a `.c` file directly + +Best for small, self-contained helpers. **`src/native-lib.c`** ```c -// A pure C function int native_multiply(int a, int b) { return a * b; } @@ -61,12 +73,10 @@ int native_multiply(int a, int b) { **`src/index.bs`** ```typescript -// Include the local C file code`#include "./native-lib.c"` export function multiply(a: integer, b: integer): integer { let result = 0; - // Call the function defined in the included C file code`${result} = native_multiply(${a}, ${b});` return result; } @@ -74,10 +84,54 @@ export function multiply(a: integer, b: integer): integer { console.log(multiply(3, 4)); ``` +### Option 2: Use a header and a separate `.c` file (recommended) + +Best when porting existing C code or splitting declarations from implementation. +The build system compiles `.c` files under `srcDir` automatically; include the `.h` from BlueScript so the generated C code sees the function declarations. + +**`src/add.h`** +```c +int add(int a, int b); +``` + +**`src/add.c`** +```c +#include "add.h" + +int add(int a, int b) { + return a + b; +} +``` + +**`src/index.bs`** +```typescript +code`#include "add.h"` + +function main(): void { + let result: integer = 0; + code`${result} = add(10, 20);` + console.log(result); +} + +main(); +``` + +:::note Path rules +- `"./file.c"` or `"file.h"` — project-local, relative to the current `.bs` file +- `` — system or ESP-IDF headers (see [Inline C](./inline-c.md)) +::: + +:::warning File naming +Do not use the `bs_` prefix for C source file names. That prefix is reserved for files generated by the BlueScript compiler. +::: + ### Summary table | Source Type | Syntax | Path Style | Example | | :--- | :--- | :--- | :--- | -| **Local BS Module** | `import { ... }` | Relative | `"./utils"` | +| **Local BS Module** | `import { ... }` | Relative to `.bs` | `"./utils"` | | **Package BS Module** | `import { ... }` | Package Name | `"gpio"` | -| **Local C File** | `code`\`#include ...\` | Relative | `"./driver.c"` | +| **Local C File** | `code`\`#include ...\` | Relative to `.bs` | `"./driver.c"` | +| **Local Header File** | `code`\`#include ...\` | Relative to `.bs` | `"driver.h"` | + +When you include a `.h` file, place the corresponding `.c` implementation under `srcDir`. The build system compiles it automatically; BlueScript only needs the header for declarations. diff --git a/website/docs/tutorial/guides/inline-c.md b/website/docs/tutorial/guides/inline-c.md index 127e395b..cacdc33f 100644 --- a/website/docs/tutorial/guides/inline-c.md +++ b/website/docs/tutorial/guides/inline-c.md @@ -4,10 +4,14 @@ One of BlueScript's most powerful features is **Inline C**. Because BlueScript compiles to native code on the host before transmission, you can embed standard C code directly within your TypeScript-like source files. This allows you to: -* Call native ESP-IDF APIs not yet wrapped by libraries. +* Call native ESP-IDF APIs not yet wrapped by libraries (ESP32 only). * Optimize critical loops for maximum performance. * Port existing C drivers easily. +:::note Board support +Examples that use **ESP-IDF** APIs apply to **ESP32** projects only. Standard C (for example `math.h`) works on both ESP32 and host. +::: + ## Basic Syntax To write C code, use the **`code`** tagged template literal. The content inside the backticks is injected into the generated C source file during compilation. @@ -74,6 +78,8 @@ console.log(getCurrentTime()); If you place a `code` block at the top level of your file (outside of any class or function), it will be placed in the global scope of the generated C file. This is used for `#include` directives or defining C global variables. +To include project-local `.c` or `.h` files, see [Imports & Includes](./imports-and-includes.md). + ```typescript // Global scope: Includes and Helper functions code` diff --git a/website/docs/tutorial/guides/repl.md b/website/docs/tutorial/guides/repl.md index 343b0b26..740c6943 100644 --- a/website/docs/tutorial/guides/repl.md +++ b/website/docs/tutorial/guides/repl.md @@ -14,7 +14,7 @@ The easiest way is the **Notebook**: a browser UI where you run code in cells. Y | :--- | :--- | :--- | | **Notebook** | `bscript project run --with-notebook` | You have a project and want to try code in cells (recommended) | | **Project REPL** | `bscript project run --with-repl` | Same as above, but you prefer the terminal | -| **Global REPL** | `bscript repl -b esp32` | No project yet—language syntax only | +| **Global REPL** | `bscript repl -b esp32` or `bscript repl -b host` | No project yet—language syntax only | | **Normal run** | `bscript project run` | You are writing the full app in `index.bs` | **Notebook vs REPL:** The Notebook supports multi-line cells (**Shift+Enter** to run) and shows output on the side. The REPL accepts **one line per Enter**. @@ -88,10 +88,16 @@ After `index.bs` runs, type one line at the `>` prompt. Installed packages (e.g. ```bash bscript repl -b esp32 +# or, without hardware: +bscript repl -b host ``` Use this for quick syntax checks without a project. **GPIO and other installed libraries are not available.** Exit with **`Ctrl+D`**. +:::note Host runtime +The host runtime must be set up first. See [Try Without Microcontroller](./try-without-microcontroller.md) for setup. +::: + :::note A global Notebook (without a project) is planned for a future release. ::: diff --git a/website/docs/tutorial/guides/try-without-microcontroller.md b/website/docs/tutorial/guides/try-without-microcontroller.md new file mode 100644 index 00000000..06911675 --- /dev/null +++ b/website/docs/tutorial/guides/try-without-microcontroller.md @@ -0,0 +1,83 @@ +--- +sidebar_label: Try Without Microcontroller +--- + +# Try Without Microcontroller + +BlueScript is designed primarily for microcontroller development (ESP32). +If you do not have hardware yet—or want a faster path for language and compiler checks—you can run BlueScript on the **host runtime** on your development machine. + +For the main ESP32 workflow, see [Get Started](../get-started/introduction.md). + +:::danger macOS Only +The host runtime currently requires **macOS**. Windows and Linux support is under development. +::: + +## Prerequisites + +- [Node.js](https://nodejs.org/) v20+ +- C compiler toolchain (`cc` and `make`) + +## Quickstart + +### 1. Install the CLI + +If you have not installed the CLI yet: + +```bash +npm install -g @bscript/cli +``` + +### 2. Set up the host runtime + +```bash +bscript board setup host +``` + +### 3. Create a host project + +```bash +bscript project create hello-host -b host +cd hello-host +``` + +### 4. Write your program + +Edit `src/index.bs`: + +```typescript title="src/index.bs" +console.log("Hello from host runtime!"); +``` + +### 5. Run + +```bash +bscript project run +``` + +You can also start interactive mode: + +```bash +bscript project run --with-repl +# or +bscript project run --with-notebook +``` + +## Notes + +- `bscript board flash-runtime` is for microcontrollers and is not supported for `host`. +- GPIO and other ESP32-specific libraries are not available on host. + +## Host vs ESP32 + +| Topic | Host (`-b host`) | ESP32 (`-b esp32`) | +| :--- | :--- | :--- | +| Main use case | Fast local testing | Real hardware development | +| Runtime location | Local process on the development machine | Runtime on the microcontroller | +| Setup | `bscript board setup host` | `bscript board setup esp32` + `bscript board flash-runtime esp32` | +| Connection during run | Local process | Bluetooth | +| Hardware libraries (GPIO, PWM, I2C) | Not available | Available via packages | +| `flash-runtime` | Not supported | Required for first-time setup | + +Choose **ESP32** if you are building applications for real devices. +Choose **host** if you want a no-hardware path for syntax checks, compiler behavior checks, or CI-like validation. diff --git a/website/sidebars.ts b/website/sidebars.ts index 7e76f68c..54b0bd90 100644 --- a/website/sidebars.ts +++ b/website/sidebars.ts @@ -19,7 +19,8 @@ const sidebars: SidebarsConfig = { 'tutorial/guides/inline-c', 'tutorial/guides/repl', 'tutorial/guides/interrupts', - 'tutorial/guides/imports-and-includes' + 'tutorial/guides/imports-and-includes', + 'tutorial/guides/try-without-microcontroller', ], }, ],