diff --git a/README.md b/README.md index 7a1915f9..bb056c53 100644 --- a/README.md +++ b/README.md @@ -1,59 +1,214 @@ -# Acorn Node +# MEE Node -Acorn node is the main node in the MEE protocol. Learn more: +MEE Node is the main node in the [Modular Execution Environment (MEE)](https://www.biconomy.io/post/modular-execution-environment-supertransactions) protocol. It issues cryptographically signed quotes for supertransactions and executes them across multiple chains. -https://www.biconomy.io/post/modular-execution-environment-supertransactions +## Table of contents -## Development setup +- [Overview](#overview) +- [Architecture](#architecture) +- [Prerequisites](#prerequisites) +- [Dependencies](#dependencies) + - [Redis](#redis) + - [Token Storage Detection Service](#token-storage-detection-service) +- [Quick start](#quick-start) +- [Configuration](#configuration) +- [Running the node](#running-the-node) +- [Docker](#docker) +- [API](#api) +- [Health and operations](#health-and-operations) +- [Further documentation](#further-documentation) +- [Contact](#contact) -### Prerequisites +## Overview -* [bun](https://bun.sh) -* [docker](https://www.docker.com) +The node: -### Installation +- **Quotes** user intents (supertransactions) and returns signed quotes with gas limits, deadlines, and fees. +- **Executes** signed quotes on-chain: it simulates, batches, and submits transactions via worker processes. +- Uses **Redis** for job queues (BullMQ), quote/userOp storage, and caching. +- Uses a **Token Storage Detection** service to resolve ERC20 balance storage slots for simulation. -```bash -bun i -``` +## Architecture + +- **Master process**: Initializes chains, RPC manager, gas manager, batcher, health checks, and spawns workers. +- **API workers** (cluster): Serve HTTP API (quote, execute, info, explorer). +- **Simulator workers** (threads, per chain): Process simulation jobs from the queue. +- **Executor workers** (threads, per chain): Process execution jobs from the queue. + +Quote flow: **Quote API** → **Storage (Redis)** → **Simulator queue** → **Batcher** → **Executor queue** → **Chain RPC**. + +See [docs/architecture.md](docs/architecture.md) for details. + +## Prerequisites + +- [Bun](https://bun.sh) (runtime and package manager) +- [Docker](https://www.docker.com) (optional, for Redis and token-storage service) +- [Rust toolchain](https://rustup.rs) (only if you build the token-storage-detection service from source) + +## Dependencies + +The node requires two external services to run. + +### Redis + +Redis is used for: + +- **Job queues** (BullMQ): simulator and executor queues per chain +- **Storage**: quotes and userOps (by hash), and custom fields +- **Caching**: e.g. token slot detection, price feeds -### Configuration +**Configuration** (see [.env.example](.env.example)): -To see a full node config options, check [.env.example](./.env.example) file. +- `REDIS_HOST` (default: `localhost`) +- `REDIS_PORT` (default: `6379`) -To run a Node, prepare your `.env` file and spin up the node: +**Run Redis locally (Docker):** ```bash -bun run start # ... or `bun run start:dev` to start a node in dev environment +docker run -d --name redis -p 6379:6379 redis:7-alpine ``` -## Live instance +Or use the project’s Compose file (includes Redis Stack): + +```bash +docker compose up -d redis-stack +``` + +**Eviction**: Quote and userOp keys are not set with TTL, so Redis can grow over time. For production, configure an eviction policy (e.g. `maxmemory` + `maxmemory-policy allkeys-lru`). See [docs/dependencies.md](docs/dependencies.md#eviction-policy-recommended) for details. + +See [docs/dependencies.md](docs/dependencies.md#redis) for more detail. + +### Token Storage Detection Service + +A separate HTTP service that returns the **ERC20 balance storage slot** for a given token and chain. The node calls it during simulation to build correct state overrides (e.g. for `balanceOf`). + +**Configuration:** + +- `TOKEN_SLOT_DETECTION_SERVER_BASE_URL` (default: `http://127.0.0.1:5000`) + +The service is implemented in Rust in `apps/token-storage-detection`. It exposes: + +- `GET /{chainId}/{tokenAddress}` → `{ success, msg: { slot } }` + +You can run your own instance or use a hosted one. See [docs/dependencies.md](docs/dependencies.md#token-storage-detection-service) and [apps/token-storage-detection/README.md](apps/token-storage-detection/README.md). -At the moment of updating these docs, there's a single node running at: +## Quick start -https://mee-node.biconomy.io +1. **Clone and install** -There's a roadmap in plan to decentralize the node and let anyone operate their node & provide infra for others to use. + ```bash + git clone + cd mee-node + bun i + ``` + +2. **Start Redis** + + ```bash + docker run -d --name redis -p 6379:6379 redis:7-alpine + ``` + +3. **Start Token Storage Detection** (see [apps/token-storage-detection](apps/token-storage-detection)) + + ```bash + cd apps/token-storage-detection + cp .env.example .env # set RPC URLs for chains you need + cargo run --release --bin token-storage-detection + ``` + + Default: `http://127.0.0.1:5000`. Adjust port in the app’s `.env` if needed (e.g. `SERVER_PORT`). + +4. **Configure the node** + + ```bash + cp .env.example .env + # Set at least: + # - NODE_ID (required) + # - NODE_PRIVATE_KEY (required) + # - REDIS_HOST / REDIS_PORT if not localhost:6379 + # - TOKEN_SLOT_DETECTION_SERVER_BASE_URL if not http://127.0.0.1:5000 + # - CUSTOM_CHAINS_CONFIG_PATH or use built-in chains + ``` + +5. **Run the node** + + ```bash + bun run start # production + bun run start:dev # development (watch mode) + ``` + + API listens on `PORT` (default `4000`). Check [http://localhost:4000/v1/info](http://localhost:4000/v1/info) (or your `PORT`) for version and health. + +## Configuration + +All options are documented in [.env.example](.env.example). Key groups: + +| Area | Main variables | +|------|-----------------| +| **Server** | `PORT`, `NODE_ENV`, `ENV_ENC_PASSWORD` (production/staging secrets) | +| **Node identity** | `NODE_ID`, `NODE_PRIVATE_KEY`, `NODE_NAME`, `NODE_FEE_BENEFICIARY` | +| **Chains** | `CUSTOM_CHAINS_CONFIG_PATH`, batch gas limits, simulator/executor concurrency | +| **Redis** | `REDIS_HOST`, `REDIS_PORT` | +| **Token slot service** | `TOKEN_SLOT_DETECTION_SERVER_BASE_URL` | +| **Workers** | `NUM_CLUSTER_WORKERS`, `MAX_EXTRA_WORKERS`, queue attempts/backoff | +| **Logging** | `LOG_LEVEL`, `PRETTY_LOGS` | + +For production/staging, the node can load encrypted secrets from `keystore/key.enc` (see `ENV_ENC_PASSWORD` and [src/common/setup.ts](src/common/setup.ts)). + +## Running the node + +| Command | Description | +|--------|--------------| +| `bun run start` | Run with Bun (uses `src/main.ts`); cluster + workers. | +| `bun run start:dev` | Watch mode; single process, all modules loaded. | +| `bun run build && bun run start:prod` | Build to `dist/` and run `dist/main.js`. | + +Ensure Redis and the token-storage-detection service are up and reachable; otherwise quote/execute and health may fail. See [docs/operations.md](docs/operations.md) for runbooks. + +## Docker + +- **Node image**: [bcnmy/mee-node](https://hub.docker.com/r/bcnmy/mee-node). Use with your own Redis and token-storage service. +- **Token Storage Detection**: See [apps/token-storage-detection/Dockerfile](apps/token-storage-detection/Dockerfile). Build and run with the same env vars as the Rust app (RPC URLs, optional Redis, etc.). + +Example (node only): + +```bash +docker run -e NODE_ID=... -e NODE_PRIVATE_KEY=... \ + -e REDIS_HOST=host.docker.internal \ + -e TOKEN_SLOT_DETECTION_SERVER_BASE_URL=http://host.docker.internal:5000 \ + -p 4000:4000 bcnmy/mee-node +``` -One can check the node version at any time by accessing `/info` endpoint, as shown [here](https://mee-node.biconomy.io/info). +## API -## Docker image +Public HTTP API (see also live [docs](https://mee-node.biconomy.io/docs)): -Docker repository can be found [here](https://hub.docker.com/r/bcnmy/mee-node). +| Method | Path | Description | +|--------|------|-------------| +| GET | `/v1/info` | Node version, supported chains, health (Redis, token-slot, queues, etc.) | +| GET | `/v1/explorer/:hash` | Get quote by hash (optional `confirmations`) | +| POST | `/v1/quote` | Request a quote (intent → signed quote) | +| POST | `/v1/quote-permit` | Request a quote with permit flow | +| POST | `/v1/exec` | Execute a signed quote | -## API Docs +The **quote** endpoint returns a signed quote (node’s commitment). The **execute** endpoint accepts the user-signed quote, validates it, and runs the intent on the configured chains. -Acorn Node exposes two public endpoints: +## Health and operations -- /v1/quote (POST) -- /v1/execute (POST) +- **`/v1/info`**: Returns node info and health for Redis, token-slot detection, chains, simulator, executor, and workers. +- **Logs**: Structured (e.g. Pino). Level via `LOG_LEVEL`; `PRETTY_LOGS=1` for development. +- **Graceful shutdown**: Use SIGTERM; the process uses `tini` in Docker. -Find request and response examples [here](https://mee-node.biconomy.io/docs). +See [docs/operations.md](docs/operations.md) for runbooks (startup, dependency checks, scaling, troubleshooting). -The quote endpoint accepts user's intent & returns a valid quote cryptographically signed by the Node's private key. This is node's commitment to execute user's intent under certain conditions (gas price, gas limit, execution deadline, etc). +## Further documentation -The execute endpoint accepts quote signed by the end-user. The node will validate that the quote was really issued from its side, and check other parameters (execution deadline). If all match, the node proceeds to fully the intent on all chains, & returns the transaction hash to the user. +- [docs/architecture.md](docs/architecture.md) — Process model, queues, and data flow +- [docs/dependencies.md](docs/dependencies.md) — Redis (including eviction) and Token Storage Detection in detail +- [docs/chain-configuration.md](docs/chain-configuration.md) — Adding and configuring chains: all config fields, price oracles (native + payment), and requirement that any chain referenced by an oracle must be in chain config +- [docs/operations.md](docs/operations.md) — Runbooks and operations +- [.env.example](.env.example) — All configuration options ## Contact -Reach out to us at: connect@biconomy.io +Reach out: connect@biconomy.io diff --git a/apps/token-storage-detection/README.md b/apps/token-storage-detection/README.md index ccf00832..9b4ccd11 100644 --- a/apps/token-storage-detection/README.md +++ b/apps/token-storage-detection/README.md @@ -1,11 +1,58 @@ -# Token Storage Detection server +# Token Storage Detection service -## Usage +HTTP service that returns the **ERC20 balance storage slot** for a given token contract and chain. Used by the MEE Node during simulation to build correct state overrides (e.g. for `balanceOf`). -### Env variables -See `.env.example` for a list of required environment variables. +## API + +- **GET /{chainId}/{tokenAddress}** + - `chainId`: chain id (e.g. `1`, `8453`) + - `tokenAddress`: ERC20 contract address + - Response: `{ success: true, msg: { slot: "0x3" } }` or `{ success: false, error: "..." }` + +## Configuration + +See `.env.example` in this directory. Main options: + +- **Server**: `SERVER_HOST` (default `127.0.0.1`), `SERVER_PORT` (default `3000` in code; `.env.example` uses `5000` to match the node’s default) +- **Chains / RPCs**: For each chain you need, set either `{CHAIN}_RPC` (primary RPC with debug/trace) or `{CHAIN}_FORK_RPC` (e.g. for Anvil fork). Examples: `ETHEREUM_RPC`, `BASE_RPC`, `ETHEREUM_FORK_RPC`, etc. If the RPC does not support the debug/trace APIs required for token detection, use **fork mode** (`{CHAIN}_FORK_RPC`); **Anvil** is a good choice for such chains. +- **Redis** (optional): `REDIS_ENABLED=1`, `REDIS_HOST`, `REDIS_PORT`, `REDIS_PASSWORD`, `REDIS_IS_TLS` for response caching +- **Anvil** (optional): `ANVIL_ENABLED=1` and related options when using fork RPCs +- **Timeouts**: `TIMEOUT_MS`, `LOGGING_ENABLED` + +## Adding a new chain + +Unlike the MEE Node (where adding a standard EVM chain is usually configuration-only), this service requires **code changes**: + +1. Add a variant to the **`Chain` enum** in `src/state.rs`. +2. Extend **`FromStr`** in the same file so the chain id (e.g. `"8453"`) parses to that variant. +3. Set the corresponding **RPC env var** (e.g. `BASE_RPC` or `BASE_FORK_RPC`) in `.env`. + +See the main repo’s [Chain configuration](../../docs/chain-configuration.md) and [Dependencies — Token Storage](../../docs/dependencies.md#adding-new-chains-token-storage-service) for the full picture. + +## Run locally -### Run your own ```bash -$ cargo run --release --bin token-storage-detection +cp .env.example .env +# Set at least one chain RPC, e.g. ETHEREUM_RPC or ETHEREUM_FORK_RPC +cargo run --release --bin token-storage-detection ``` + +By default the node expects this service at `http://127.0.0.1:5000`. Either set `SERVER_PORT=5000` in `.env` or set the node’s `TOKEN_SLOT_DETECTION_SERVER_BASE_URL` to your URL (e.g. `http://127.0.0.1:3000`). + +## Docker + +Build and run with the same env vars (RPCs, optional Redis, `SERVER_PORT`, etc.): + +```bash +docker build -t token-storage-detection . +docker run -p 5000:5000 -e SERVER_PORT=5000 -e ETHEREUM_RPC=... token-storage-detection +``` + +## Operational notes + +- **RPC at boot**: The service builds one RPC provider per configured chain at startup. If **any** chain’s RPC (or Anvil fork) fails during init, the process can **exit** and may restart in a loop until the RPC is fixed. Use stable RPCs; for exotic or unreliable chains, consider a minimal instance with only the chains you need. See [Dependencies — RPC and boot behavior](../../docs/dependencies.md#rpc-and-boot-behavior). +- **When this service fails**: The MEE Node still executes supertransactions using the **default gas limit from the SDK**, which is sufficient for many flows. Complex flows may fail with insufficient gas. When the service is unavailable, the node can fall back to a **Redis-backed cache** of balance storage slots: tokens that were successfully resolved at least once (to detect their storage slot) are cached. This cache is **persistent** (stored in Redis). See [Dependencies — Impact on execution](../../docs/dependencies.md#impact-on-execution-when-the-token-service-fails). + +## Relation to MEE Node + +The MEE Node calls this service when simulating userOps that involve ERC20 balances. If the service is down or returns errors, those simulations can fail. See the main repo’s [docs/dependencies.md](../../docs/dependencies.md#token-storage-detection-service) for details. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 00000000..510861a4 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,77 @@ +# MEE Node architecture + +This document describes how the MEE Node is structured and how data flows from quote to execution. + +## Process model + +The node uses Node.js **cluster** and **worker threads**: + +1. **Primary (master)** + - Runs once. + - Initializes: chains config, RPC manager, gas manager, batcher, health checks. + - Spawns API workers (cluster) and, per chain, simulator and executor workers (threads). + - Pushes config and health results to workers via IPC. + +2. **API workers (cluster)** + - One or more HTTP server processes. + - Handle `/v1/quote`, `/v1/quote-permit`, `/v1/exec`, `/v1/info`, `/v1/explorer/:hash`. + - Receive chain settings, RPC config, gas info, and health results from the master. + +3. **Simulator workers (threads, per chain)** + - Consume jobs from the **simulator queue** for that chain (async batch simulation after quote). + - Run **execution simulation**: they simulate userOps against **on-chain state** (no state overrides). Their role is to confirm that on-chain conditions are met before the execution phase. + +4. **Executor workers (threads, per chain)** + - Consume jobs from the **executor queue** (BullMQ) for that chain. + - Submit signed transactions to the chain RPC. + - Use node-owned EOA wallets (master + optional extra workers from mnemonic/keys). + +Entry points: + +- **Master**: `src/master/bootstrap.ts` +- **API**: `src/api/bootstrap.ts` +- **Simulator**: `src/workers/simulator/main.ts` +- **Executor**: `src/workers/executor/main.ts` + +All started from `src/main.ts` (cluster primary runs master, workers run API). + +## Data flow + +### Quote → storage → simulation → batching → execution + +1. **Quote** — Request comes in. The API runs **pre-simulation** for gas estimation and calldata validity: it fills the on-chain state gap using **state overrides** (e.g. ERC20 balances) and uses the **Token Storage Detection** service to get balance storage slots when needed. Pre-simulation produces gas estimates and validates the batch. The node then stores quote and userOps in Redis and enqueues simulator jobs per chain. + +2. **Simulator** — Workers process simulator jobs: they run **execution simulation** against current on-chain state (no state overrides), so that execution only runs when on-chain conditions are satisfied. They do not use the Token Storage Detection service. The batcher listens for completed jobs. + +3. **Batcher** — Groups simulated userOps per chain into batches under the chain's batch gas limit and enqueues executor jobs. + +4. **Executor** — Workers pick executor jobs, sign and send batch transactions using the node's RPC and EOA, then complete the job. + +5. **Execute** — Client sends the signed quote; node loads from Redis, validates, and execution is driven by the same queues until the execution job is done. + +**Redis** backs queues, quote/userOp storage, and cache. **Token Storage Detection** is used only in the **pre-simulation and gas estimation phase** (in the API during quote), to build state overrides for ERC20 balances; it is not used by simulator workers. + +## Redis usage + +Redis is used for job queues (simulator and executor per chain), quote and userOp storage, and caching. Connection is configured via `REDIS_HOST` and `REDIS_PORT`. + +## Health checks + +The **HealthCheckService** (master) periodically runs: + +- **Redis**: e.g. `CLIENT LIST` to ensure connectivity. +- **Chains / RPC**: per-chain checks. +- **Simulator / Executor**: per-chain queue presence/job counts. +- **Node**: wallet/account status per chain. +- **Token Slot Detection**: per-chain request to the token-storage service (soft: does not mark chain unhealthy). + +Results are sent to API workers. `/v1/info` aggregates them so operators can see status of Redis, token-slot, queues, and chains. + +## Configuration flow + +- **Chains**: Loaded from config (or `CUSTOM_CHAINS_CONFIG_PATH`). Master initializes `ChainsService` and passes chain settings to API workers. +- **RPC**: Master builds RPC chain configs, calls `RpcManagerService.setup()`, then pushes config to API and thread workers (simulator/executor). +- **Gas**: Gas manager runs in master; gas info is synced to API and thread workers. +- **Node wallets**: Node service runs in master; wallet states are pushed to executor workers for signing. + +All of this ensures API and workers see the same chains, RPCs, gas, and wallet state. diff --git a/docs/chain-configuration.md b/docs/chain-configuration.md new file mode 100644 index 00000000..ef4f1713 --- /dev/null +++ b/docs/chain-configuration.md @@ -0,0 +1,200 @@ +# Chain configuration + +This document describes how to add and configure chains for the **MEE Node**: where config lives, every config field, how they affect behavior, and the **price oracle requirement** for native and payment tokens. It also contrasts with the Token Storage Detection service. For a full run-and-maintain tutorial (master EOA, fees, RPC requirements, chain type, payment tokens, arbitrary payment, permit, trusted gas tank), see [Run and maintain the node](run-and-maintain.md). + +--- + +## Overview + +For **standard EVM chains**, adding a chain in the MEE Node is a **configuration-only** change. You add or edit a chain config object (JSON); the node reads it at startup and uses it for RPCs, batching, simulation, execution, gas, and pricing. This doc explains each field and how to add a new chain. + +--- + +## Where chain config is loaded from + +- **`CUSTOM_CHAINS_CONFIG_PATH`** (env): Optional. If set, the node loads from this path (file or directory). +- **Default paths** (if not set): `./config/chains`, then `./chains` (relative to process cwd). + +Config can be: + +- A **directory**: one JSON file per chain, named by chain id (e.g. `1.json`, `8453.json`). +- A **single JSON file**: top-level keys are chain IDs (string), values are chain config objects. + +The node merges env-driven defaults (e.g. `DEFAULT_USER_OPS_BATCH_GAS_LIMIT`) with per-chain JSON; see `.env.example` for global defaults and the table below for per-chain fields. + +**RPC requirements:** For full functionality (simulation, gas estimation), RPCs should support **`debug_traceCall`** and **`eth_feeHistory`**. See [Run and maintain — Chain and RPC requirements](run-and-maintain.md#4-chain-and-rpc-requirements). + +--- + +## Price oracles and required chains + +The node needs a **native coin price** (and optionally **payment token prices**) for fee and payment logic. Prices can come from: + +- **`price.type: "fixed"`** — constant `value` and `decimals` in config. No extra chain or RPC. +- **`price.type: "oracle"`** — price is read from a **Chainlink-style aggregator** (e.g. `latestRoundData`, `decimals`) on a specific chain. You must set **`price.chainId`** and **`price.oracle`** (contract address). + +**Important:** If `price` is `"oracle"` and references a **chainId**, that chain **must exist in your chains config** and have working RPCs. The node calls `RpcManagerService.executeRequest(chainId, ...)` to read the oracle contract. If that chain is not configured, the request will fail and native/payment pricing can break. + +So: **any chain referenced by a price oracle must be present in the chain config**, even if you do not use that chain for execution (e.g. you only use it for ETH/USD price on Ethereum mainnet). + +--- + +## Configuration sources + +The node accepts three configuration sources: + +1. **Environment variables (ENV)** — Node-level settings (e.g. in `.env`). Examples: `NODE_PRIVATE_KEY`, `DEFAULT_USER_OPS_BATCH_GAS_LIMIT`, `DEFAULT_NUM_SIMULATOR_WORKERS_PER_CHAIN`, `MAX_EXTRA_WORKERS`, `CUSTOM_CHAINS_CONFIG_PATH`. These apply across chains unless overridden per chain. +2. **Encrypted keystore (optional)** — In **production** or **staging** (`NODE_ENV=production` or `staging`), the node can load secrets (e.g. `NODE_PRIVATE_KEY`) from an encrypted file at **`keystore/key.enc`**, decrypted at startup using **`ENV_ENC_PASSWORD`**. This keeps the private key encrypted at rest. See [Operations — Configuration](operations.md#configuration) for how to create and use the encrypted keystore. +3. **Chain config (JSON)** — Per-chain settings in the chain config files (directory or single file). Keys in the JSON (e.g. `batcher.batchGasLimit`, `simulator.numWorkers`) override or supplement env-driven defaults for that chain. + +The tables below list **chain config** keys and, where relevant, the **ENV var** that provides the default. Use ENV vars (or the encrypted keystore) to configure the node globally; use chain config JSON to override per chain. + +--- + +## Chain config fields (full reference) + +All fields below are **chain config** (JSON) unless stated otherwise. Optional fields have schema defaults; defaults from ENV are noted. + +### Identification and RPC + +| Chain config key | Type | Required | Default | Description / impact | +|------------------|------|----------|--------|----------------------| +| **`chainId`** | string (numeric) | Yes | — | Chain id (e.g. `"1"`, `"8453"`). Must match the key in the config file/directory. | +| **`name`** | string | Yes | — | Human-readable name (logs, errors, gas estimator). | +| **`rpcs`** | string[] | Yes | — | RPC URLs for this chain. Used by RPC manager for simulation and execution. At least one required. | + +--- + +### Contracts + +| Chain config key | Type | Required | Default | Description / impact | +|------------------|------|----------|--------|----------------------| +| **`contracts.entryPointV7`** | address | No | `0x0000000071727De22E5E9d8BAf0edAc6f37da032` | EntryPoint v7 contract. Used for simulation, execution, and paymaster deployment. | +| **`contracts.pmFactory`** | address | No | `0x000000005824a1ED617994dF733151D26a4cf03d` | Node paymaster factory. Used to deploy and fund the node paymaster. | +| **`contracts.disperse`** | address | No | `0xd15fE25eD0Dba12fE05e7029C88b10C25e8880E3` | Disperse contract used when funding multiple worker EOAs. | + +Override only if your chain uses different deployments. + +--- + +### Batcher and gas limits + +| Chain config key | ENV var (default) | Type | Required | Default | Description / impact | +|------------------|-------------------|------|----------|--------|----------------------| +| **`batcher.batchGasLimit`** | `DEFAULT_USER_OPS_BATCH_GAS_LIMIT` | bigint/number | No | `8_000_000` | Max gas per batch of userOps submitted in one transaction. Set via ENV or chain config. | + +--- + +### Gas and chain type (gas estimator, L1/L2) + +| Chain config key | Type | Required | Default | Description / impact | +|------------------|------|----------|--------|----------------------| +| **`type`** | `"evm"` \| `"optimism"` \| `"arbitrum"` | No | `"evm"` | Chain type for gas estimation. L2s use different fee logic (e.g. L1 fee). | +| **`eip1559`** | boolean | No | `false` | Whether the chain supports EIP-1559. Affects fee calculation. | +| **`gasPriceMode`** | `"standard"` \| `"fast"` \| `"rapid"` | No | `"standard"` | Aggressiveness of gas price (more = higher fee, faster inclusion). | +| **`l1ChainId`** | string | No | — | For L2s, the L1 chain id (e.g. `"1"` for Ethereum). Used to fetch L1 gas price and L1 fee. | +| **`feeHistoryBlockTagOverride`** | string | No | — | Override block tag for `eth_feeHistory` (e.g. `"latest"`, `"pending"`). | +| **`minMaxFeePerGas`** | bigint/number (wei) | No | — | Optional minimum `maxFeePerGas` (wei) for legacy chains; some RPCs (e.g. BSC) reject txs below this. | + +--- + +### Native token price (critical for fees and payment) + +| Chain config key | Type | Required | Default | Description / impact | +|------------------|------|----------|--------|----------------------| +| **`price`** | object | Yes | — | Native coin price source. | +| **`price.type`** | `"fixed"` \| `"oracle"` | Yes | — | See below. | +| **`price.value`** | number/bigint | If `type === "fixed"` | — | Fixed price (e.g. USD per native token). | +| **`price.decimals`** | number | If `type === "fixed"` | — | Decimals for the fixed price. | +| **`price.chainId`** | string | If `type === "oracle"` | — | **Chain where the oracle contract lives.** This chain **must** be in your chains config with valid RPCs. | +| **`price.oracle`** | address | If `type === "oracle"` | — | Chainlink-style aggregator contract (must expose `latestRoundData` and `decimals`). | + +If you use **`price.type: "oracle"`**, the chain in **`price.chainId`** must be configured; otherwise oracle reads will fail and native token pricing will break. + +--- + +### Paymaster funding (node operations) + +| Chain config key | Type | Required | Default | Description / impact | +|------------------|------|----------|--------|----------------------| +| **`paymasterFunding`** | string (ether) | No | `"0.025"` | Amount of native token sent to the paymaster **only during first deployment** (when the paymaster contract for this chain and master EOA does not exist yet). Not used for top-ups. | +| **`paymasterFundingThreshold`** | string (ether) | No | `"0"` | Balance below which the node considers the paymaster unhealthy. Operator must top up the paymaster when balance falls below this (see [Operations — Master EOA and paymaster funding](operations.md#master-eoa-and-paymaster-funding)). | +| **`paymasterInitCode`** | hex | No | — | Optional custom paymaster init code. | + +For how the master EOA, paymaster deployment, and worker balances interact, see [Operations — Master EOA and paymaster funding](operations.md#master-eoa-and-paymaster-funding). + +--- + +### Confirmation and execution + +| Chain config key | Type | Required | Default | Description / impact | +|------------------|------|----------|--------|----------------------| +| **`waitConfirmations`** | number | No | `3` | Number of block confirmations before considering a tx confirmed. Used by executor and explorer. | +| **`waitConfirmationsTimeout`** | number (ms) | No | `60000` (60 s) | Timeout when waiting for confirmations. | + +--- + +### Simulator (per-chain) + +Configure via **ENV vars** (node-level defaults) or override per chain in chain config JSON. Omit a chain config key to use the ENV default. + +| ENV var (node-level default) | Chain config key (per-chain override) | Type | Default | Description / impact | +|------------------------------|----------------------------------------|------|--------|----------------------| +| **`DEFAULT_NUM_SIMULATOR_WORKERS_PER_CHAIN`** | `simulator.numWorkers` | number | `1` | Number of simulator thread workers for this chain. | +| **`DEFAULT_SIMULATOR_WORKER_CONCURRENCY`** | `simulator.workerConcurrency` | number | `10` | Concurrency per simulator worker. | +| **`DEFAULT_SIMULATOR_STALLED_JOBS_RETRY_INTERVAL`** | `simulator.stalledJobsRetryInterval` | number (ms) | `5000` (5 s) | Interval for retrying stalled simulation jobs. | +| **`DEFAULT_SIMULATOR_RATE_LIMIT_MAX_REQUESTS_PER_INTERVAL`** | `simulator.rateLimitMaxRequestsPerInterval` | number | `100` | Rate limit: max requests per interval. | +| **`DEFAULT_SIMULATOR_RATE_LIMIT_DURATION`** | `simulator.rateLimitDuration` | number (s) | `1` | Rate limit interval in seconds. | +| **`DEFAULT_SIMULATOR_TRACE_CALL_RETRY_DELAY`** | `simulator.traceCallRetryDelay` | number (ms) | `2000` | Delay before retrying a failed trace/simulation call. | + +--- + +### Executor (per-chain) + +Configure via **ENV vars** (node-level) or override per chain in chain config JSON. Worker count is also capped by **`MAX_EXTRA_WORKERS`** (env). Omit a chain config key to use the schema or ENV default. + +| ENV var (node-level default) | Chain config key (per-chain override) | Type | Default | Description / impact | +|------------------------------|----------------------------------------|------|--------|----------------------| +| *(chain config only)* | `executor.pollInterval` | number (ms) | `1000` | Poll interval when waiting for transaction receipt. | +| **`DEFAULT_EXECUTOR_STALLED_JOBS_RETRY_INTERVAL`** | `executor.stalledJobsRetryInterval` | number (ms) | `5000` (5 s) | Interval for retrying stalled execution jobs. | +| **`DEFAULT_EXECUTOR_RATE_LIMIT_MAX_REQUESTS_PER_INTERVAL`** | `executor.rateLimitMaxRequestsPerInterval` | number | `100` | Rate limit: max requests per interval. | +| **`DEFAULT_EXECUTOR_RATE_LIMIT_DURATION`** | `executor.rateLimitDuration` | number (s) | `1` | Rate limit interval in seconds. | +| *(chain config only)* | `executor.workerFunding` | string (ether) | `"0.001"` | Target balance used when funding worker EOAs (e.g. via disperse). | +| *(chain config only)* | `executor.workerFundingThreshold` | string (ether) | `"0"` | **Minimum native balance** each worker (or master when used as worker) must have to be considered healthy. Limits the maximum executable call gas limit for that chain; set high enough for the largest transaction you expect. | +| **`MAX_EXTRA_WORKERS`** (max cap) | `executor.workerCount` | number | `1` | Number of worker EOAs to use for execution on this chain. **0** = use master only; otherwise between **1** and **`MAX_EXTRA_WORKERS`**. Workers are derived from `NODE_ACCOUNTS_MNEMONIC` or `NODE_ACCOUNTS_PRIVATE_KEYS`. | + +--- + +### Gas limit overrides (per-chain) + +Chain config keys that override the global gas estimator defaults for this chain. All are optional. Omit to use the global default. + +| Chain config key | Default | Description | +|------------------|---------|-------------| +| **`gasLimitOverrides.paymasterVerificationGasLimit`** | — | Paymaster verification gas. | +| **`gasLimitOverrides.senderCreateGasLimit`** | — | Gas for sender contract creation (when initCode is set). | +| **`gasLimitOverrides.baseVerificationGasLimit`** | — | Base verification gas. | +| **`gasLimitOverrides.fixedHandleOpsGas`** | — | Fixed gas for handleOps. | +| **`gasLimitOverrides.perAuthBaseCost`** | — | Per-signature/auth base cost (e.g. EIP-7702). | + +--- + +### Payment tokens + +| Chain config key | Type | Required | Default | Description / impact | +|------------------|------|----------|--------|----------------------| +| **`paymentTokens`** | array | Yes (non-empty for execution chains) | — | List of supported payment tokens (e.g. USDC) for this chain. | +| **`paymentTokens[].name`** | string | Yes | — | Token name. | +| **`paymentTokens[].address`** | address | Yes | — | Token contract address. | +| **`paymentTokens[].symbol`** | string | Yes | — | Token symbol. | +| **`paymentTokens[].price`** | object | Yes | — | Same shape as native `price`: `{ type: "fixed", value, decimals }` or `{ type: "oracle", chainId, oracle }`. | +| **`paymentTokens[].permitEnabled`** | boolean | No | `false` | Whether ERC-20 Permit (signature-based approval) is supported; set `true` for quote-permit flows. | + +If **any** payment token uses **`price.type: "oracle"`** with a **`chainId`**, that chain must be present in your chains config (same rule as for native price). + +--- + +### Arbitrary token support + +**Arbitrary token payment** lets users pay fees with tokens that are not in the chain's `paymentTokens` list. To support it you must configure at least one **payment provider** via environment variables: **LiFi** (`LIFI_API_KEY`) or **Gluex** (`GLUEX_API_KEY`, `GLUEX_PARTNER_UNIQUE_ID`). The node uses these providers to obtain **swap calldata** (route/quote). This calldata is **not** executed by the node to perform a swap. It is used only to: (1) **validate** that the token is swappable (there is liquidity), and (2) **determine how much** of the token is required to cover fees (exchange rate). The user's payment in the arbitrary token is **received at the node's fee receiver address** (`NODE_FEE_BENEFICIARY`). The node does not swap the token on your behalf. **Operator responsibilities** when enabling arbitrary payment: periodically **swap** the tokens received at the fee receiver; **rebalance** the node portfolio across chains and ensure **paymasters on supported chains are well funded** with native token. See [Operations — Master EOA and paymaster funding](operations.md#master-eoa-and-paymaster-funding). + diff --git a/docs/dependencies.md b/docs/dependencies.md new file mode 100644 index 00000000..b83ddcca --- /dev/null +++ b/docs/dependencies.md @@ -0,0 +1,164 @@ +# MEE Node dependencies + +The node depends on two external services: **Redis** and the **Token Storage Detection** service. Both must be running and reachable for normal operation. + +## Redis + +### Role + +- **Job queues (BullMQ)** + Simulator and executor queues are backed by Redis. Each supported chain has its own simulator and executor queue. If Redis is down or unreachable, no simulation or execution jobs are processed. + +- **Quote and UserOp storage** + When a quote is created (e.g. via `/v1/quote`), the quote and its userOps are stored in Redis. The execute endpoint and internal batching load this data by quote hash / userOp hash. Keys are namespaced (e.g. `storage:quote:...`, `storage:user-op:...`). + +- **Caching** + The node uses Redis for generic cache (e.g. token slot results, price data) via `StorageService.getCache` / `setCache` with TTL. + +### Configuration + +| Variable | Default | Description | +|------------|-------------|--------------------| +| `REDIS_HOST` | `localhost` | Redis host | +| `REDIS_PORT` | `6379` | Redis port | + +Defined in `src/modules/core/redis/redis.config.ts`. Used by `RedisService`, which is shared by storage, queues, and health check. + +### Running Redis + +- **Docker (plain Redis)** + ```bash + docker run -d --name redis -p 6379:6379 redis:7-alpine + ``` + +- **Project Compose** + The repo’s `compose.yml` includes a `redis-stack` service (ports 6379 and 8001). + ```bash + docker compose up -d redis-stack + ``` + +- **Production** + Use a managed Redis (e.g. AWS ElastiCache, Redis Cloud) or your own cluster. Ensure the node’s `REDIS_HOST` / `REDIS_PORT` (and any network/firewall) allow connections. + +### Eviction policy (recommended) + +The node does **not** set TTL on quote and userOp keys; only the generic cache layer uses TTL when `setCache(..., { ttl })` is called. Over time, `storage:quote:*` and `storage:user-op:*` keys can grow. To avoid unbounded memory use, configure Redis with an **eviction policy** so older or less-used keys can be evicted when memory is limited. + +**Suggested configuration** (in `redis.conf` or via server args): + +- **maxmemory**: Set a limit (e.g. `maxmemory 2gb`). +- **maxmemory-policy**: Use a policy that fits your workload, for example: + - `volatile-lru` — evict least recently used keys among those that have a TTL (cache keys; quote/userOp keys have no TTL so they are not evicted by this policy). + - `allkeys-lru` — evict least recently used keys across all keys. Use this if you want quote/userOp data to be evictable under memory pressure (execution of very old quotes may then fail if data was evicted). + - `allkeys-lru` is a good default for open-source deployments where controlling memory is important; be aware that evicted quote/userOp data cannot be recovered. + +Example for a dedicated Redis used by the node: + +```conf +maxmemory 2gb +maxmemory-policy allkeys-lru +``` + +Alternatively, use a policy like `volatile-ttl` if you only want to evict cache keys that have TTL and you accept that quote/userOp keys never expire (ensure `maxmemory` is large enough). + +### Health check + +The master runs a periodic health check that uses Redis (e.g. `CLIENT LIST`). If Redis is unhealthy, the node reports it and chain health is considered degraded (all chains depend on Redis for queues and storage). The result is exposed on `/v1/info` under the `redis` module. + +--- + +## Token Storage Detection service + +### Role + +During **simulation**, the node needs correct state overrides for ERC20 balances. The storage layout of `balanceOf` (the slot used for `mapping(address => uint256)`) is token- and sometimes chain-specific. The **Token Storage Detection** service returns the balance storage slot for a given token contract and chain so the node can build the right overrides. + +Used in: + +- `TokenSlotDetectionService.getBalanceStorageSlot(tokenAddress, accountAddress, chainId)` +- Called from the simulation path when building state overrides for tokens (e.g. payment tokens). + +If the service is down or returns errors, simulations that need token balance overrides can fail; the node treats this as a **soft** health check (it does not mark the chain as unhealthy, but quote/simulation may still fail for affected requests). + +### API contract + +- **Base URL**: Configured by `TOKEN_SLOT_DETECTION_SERVER_BASE_URL` (default `http://127.0.0.1:5000`). +- **Endpoint**: `GET /{chainId}/{tokenAddress}` + - `chainId`: chain id (e.g. `1`, `8453`). + - `tokenAddress`: ERC20 contract address. +- **Response**: + - Success: e.g. `{ success: true, msg: { slot: "0x3" } }` (slot in hex). + - Error: e.g. `{ success: false, error: "SlotNotFound" }`. + +The node hashes the slot with the account address for mapping storage (see `getBalanceStorageSlot` in `token-slot-detection.service.ts`). + +### Configuration + +| Variable | Default | Description | +|-------------------------------------|---------------------------|--------------------------------| +| `TOKEN_SLOT_DETECTION_SERVER_BASE_URL` | `http://127.0.0.1:5000` | Base URL of the detection service | + +No auth is configured in the node; the service is assumed to be internal or network-protected. + +### Running the service + +The service lives in **`apps/token-storage-detection`** (Rust). It needs RPC URLs for each chain it should support. + +1. **From source** + ```bash + cd apps/token-storage-detection + cp .env.example .env + # Set *_RPC or *_FORK_RPC for the chains you need (e.g. ETHEREUM_RPC, BASE_RPC) + cargo run --release --bin token-storage-detection + ``` + By default it listens on `SERVER_HOST:SERVER_PORT` (e.g. `127.0.0.1:3000`). The repo’s `.env.example` for the **node** uses port 5000; either set `SERVER_PORT=5000` in the token-storage app or set `TOKEN_SLOT_DETECTION_SERVER_BASE_URL` to the actual URL (e.g. `http://127.0.0.1:3000`). + +2. **Docker** + Use the Dockerfile in `apps/token-storage-detection`. Build and run with the same env vars (RPCs, optional Redis, `SERVER_PORT`, etc.). Expose the chosen port and point the node’s `TOKEN_SLOT_DETECTION_SERVER_BASE_URL` at it. + +3. **Optional Redis** + The token-storage app can use Redis for caching (see its `.env.example`: `REDIS_ENABLED`, `REDIS_HOST`, etc.). This is independent of the node’s Redis; the node only talks to the service over HTTP. + +### Adding new chains (Token Storage service) + +Unlike the MEE Node, where adding a standard EVM chain is usually a matter of **chain configuration** (see [Chain configuration](chain-configuration.md)), the Token Storage Detection service currently requires **code changes** to support a new chain: + +1. **Add the chain to the `Chain` enum** in `apps/token-storage-detection/src/state.rs`. +2. **Implement `FromStr`** in the same file so that the chain id (e.g. `"8453"`) maps to the new enum variant. +3. **Set the RPC env var** for that chain (e.g. `BASE_RPC` or `BASE_FORK_RPC` in `.env`). + +A future improvement would be to support a **chain config file or env-based chain list** so new chains can be added without code changes; until then, document this manual process for operators. The MEE Node only needs its chain config updated; the token server must be updated and redeployed for the same chain. + +### RPC and boot behavior + +The Token Storage service builds one RPC provider per configured chain at **startup**. If any chain uses an **unreliable or failing RPC** (or a Fork RPC that fails during Anvil spawn), the service can **panic or return an error** during `make_app_http_providers` and **exit**. The process may then restart (e.g. under Kubernetes) and keep crashing until the RPC is fixed or that chain is removed from config. + +**Implications:** + +- For open-source or multi-tenant use, one bad RPC can prevent the whole service from starting. +- Prefer stable RPCs; if you use an exotic chain or unreliable endpoint, consider running a minimal token-storage instance with only the chains you need, or fixing the service to **skip or retry** failing chains at boot instead of exiting (future improvement). + +### Impact on execution when the token service fails + +Token Storage Detection is used for **simulation** (balance overrides). It is **not** required for execution itself: + +- If the token service is down or returns errors, the node may still **execute** supertransactions: the **default gas limit from the SDK** is used, which is sufficient for many flows. +- The node can fall back to a **Redis-backed cache** of balance storage slots: only tokens that were successfully resolved at least once (to detect their slot) are cached. This cache is **persistent** (stored in Redis). +- **Complex flows** (e.g. with custom token logic or higher gas needs) may **fail with insufficient gas** when token slot detection is unavailable and the token is not in the cache, because simulation cannot refine gas estimates. + +So: token service failure does not block execution, but can reduce reliability for complex or first-time tokens. Health checks and monitoring for the token service are still recommended. + +### Health check + +The master runs a soft health check per chain by calling the token-storage service (e.g. for a supported payment token). Result is shown under the `token-slot-detection` module in `/v1/info`. Failures do not mark the chain as unhealthy but indicate that simulation may fail for tokens that need slot detection. + +--- + +## Summary + +| Dependency | Purpose | Required for | Node env / config | +|---------------------------|----------------------------------|-------------------------|--------------------------| +| **Redis** | Queues, quote/userOp storage, cache | All quote/exec and queues | `REDIS_HOST`, `REDIS_PORT` | +| **Token Storage Detection** | ERC20 balance slot for simulation | Correct simulation overrides | `TOKEN_SLOT_DETECTION_SERVER_BASE_URL` | + +Both should be running before starting the node. See [operations.md](operations.md) for startup order and troubleshooting. diff --git a/docs/operations.md b/docs/operations.md new file mode 100644 index 00000000..e70126c4 --- /dev/null +++ b/docs/operations.md @@ -0,0 +1,162 @@ +# MEE Node operations + +Runbooks and operational notes for running and managing the MEE Node. For a step-by-step tutorial (why the master EOA exists, where fees go, what is auto-funded, chain and RPC requirements, chain type and payment tokens, arbitrary payment, permit, trusted gas tank, and connecting with the SDK), see [Run and maintain the node (tutorial)](run-and-maintain.md). + +## Startup order + +1. **Redis** + Start Redis and ensure it is reachable on `REDIS_HOST:REDIS_PORT`. + +2. **Token Storage Detection service** + Start the service and ensure it listens on the URL set in `TOKEN_SLOT_DETECTION_SERVER_BASE_URL`. Configure RPCs for all chains you will support in the node. + +3. **Node** + Set at least `NODE_ID`, `NODE_PRIVATE_KEY`, and any chain/RPC config (e.g. `CUSTOM_CHAINS_CONFIG_PATH`). Ensure the **master EOA** is funded on each chain you run (for paymaster deployment and, if applicable, top-ups). See [Master EOA and paymaster funding](#master-eoa-and-paymaster-funding). Then start the node: + - `bun run start` (production) + - `bun run start:dev` (development) + +If Redis or the token-storage service is down, the node may start but quotes/simulations/executions and health will be affected. Check `/v1/info` after startup. + +## Checking health + +- **GET /v1/info** + Returns node version, supported chains, and health for: + - Redis + - Token Slot Detection (per chain, soft) + - Chains (RPC, etc.) + - Simulator / Executor (queues) + - Node (wallets) + - Workers + +Use this to confirm both dependencies and internal components are healthy. + +## Configuration + +The node supports **three ways** to supply configuration and secrets (e.g. the master EOA private key): + +1. **Environment variables** — Set `NODE_PRIVATE_KEY` and other vars in `.env` or in your process environment (e.g. systemd, Kubernetes secrets). Plain text; suitable for local or controlled environments. +2. **Environment variables only** — Same as above but without a `.env` file (e.g. all vars from the orchestrator). The node still requires `NODE_PRIVATE_KEY` to be set. +3. **Encrypted keystore (optional)** — In **production** or **staging** (`NODE_ENV=production` or `NODE_ENV=staging`), the node can load secrets from an **encrypted file** at **`keystore/key.enc`**. The file is decrypted at startup using **`ENV_ENC_PASSWORD`** (the password you used to encrypt the file). Decrypted keys (e.g. `NODE_PRIVATE_KEY`) are then available to the node. This keeps the private key encrypted at rest; only `ENV_ENC_PASSWORD` needs to be provided at runtime (e.g. via a secure secret manager). The node uses `@chainlink/env-enc`: create the encrypted file with the env-enc CLI (e.g. `npx env-enc set` after `npx env-enc set-pw`), then place the output file at **`keystore/key.enc`** in the working directory. Ensure **`ENV_ENC_PASSWORD`** is set when starting the node so the keystore can be decrypted. If the keystore is present and the password is correct, the decrypted variables override or supplement those from `.env`. + +- **Chains**: Configure via built-in chain list or `CUSTOM_CHAINS_CONFIG_PATH`. RPC URLs and batch gas limits are critical for simulator and executor behavior. + +- **Workers**: + - `NUM_CLUSTER_WORKERS`: number of API processes. + - `MAX_EXTRA_WORKERS`: extra EOA accounts from mnemonic for execution. + Tune based on load and RPC limits. + +## Master EOA and paymaster funding + +The node uses a **master EOA** (from `NODE_PRIVATE_KEY`) and optionally **worker EOAs**. Workers are derived from **`NODE_ACCOUNTS_MNEMONIC`** or **`NODE_ACCOUNTS_PRIVATE_KEYS`** (mnemonic is the easiest). Two parameters control how many workers are used: + +- **`MAX_EXTRA_WORKERS`** (env) — Global maximum number of worker EOAs that can be used on **any** chain. +- **`executor.workerCount`** (per chain in chain config) — Number of workers to use on that chain. Can be from **0** (use master only) up to **`MAX_EXTRA_WORKERS`**. More workers allow higher throughput (more transactions in parallel). + +All funding for paymaster deployment and for workers comes from the **master EOA**. This section describes what is funded at boot and what you must maintain. + +### Master EOA balance (per chain) + +The **master EOA** must hold enough **native token on each chain** where the node runs to cover: + +1. **Deploying the paymaster contract** — If the paymaster for that chain and master EOA is not yet deployed, the node deploys it at startup. Deployment costs gas. +2. **Initial paymaster funding** — The amount in chain config (`paymasterFunding`) is sent to the paymaster **only during that first deployment**. If the paymaster contract already exists, the node **does not** send `paymasterFunding` again. You must **manually** track paymaster balance and top it up when needed. +3. **Funding workers** — See [Funding workers (how and when)](#funding-workers-how-and-when) below. + +### Funding workers (how and when) + +Workers are funded **at every node boot**, per chain, **after** the paymaster is deployed (if needed) and workers are whitelisted. The logic is implemented in the node’s `fundWorkers` step. + +**When:** On each startup, for each chain, the node runs in order: (1) deploy and fund paymaster (if not deployed), (2) whitelist workers on the paymaster, (3) **fund workers**. So worker funding runs every time the node starts, for every chain that has `executor.workerCount` > 0. + +**Who is funded:** The first **`executor.workerCount`** worker EOAs (derived from `NODE_ACCOUNTS_MNEMONIC` or `NODE_ACCOUNTS_PRIVATE_KEYS`), up to at most **`MAX_EXTRA_WORKERS`** and the number of accounts you have. If `workerCount` is 0, no workers are funded (the master EOA is the only executor). + +**Target balance:** Each of these workers is topped up to **`executor.workerFunding`** (chain config; default in the schema is **0.001** native token, e.g. 0.001 ETH). The node reads each worker’s current balance; for any worker below `workerFunding`, it computes the shortfall. + +**How much is sent (disperse):** The master EOA calls the **disperse** contract’s **`disperseEther`** with a list of worker addresses and a list of amounts. For each worker, the amount is **max(0, workerFunding − current balance)**. Only workers with a positive shortfall are included. The **total value** of the tx is the **sum of those amounts**; that is exactly how much native token the master EOA sends in that one transaction. + +**Example (one chain):** `workerFunding` = 0.001 ETH, 3 workers, balances 0, 0, 0.0005 ETH. Shortfalls: 0.001, 0.001, 0.0005 → **total 0.0025 ETH** from master EOA in one disperse tx (plus gas for that tx). After the tx, each worker has 0.001 ETH. + +**Example (workers already topped up):** Same config, but all 3 workers already have ≥ 0.001 ETH. All shortfalls are 0 → no disperse tx is sent; the node logs “No workers require funding. Skipping funding.” + +### How much the master EOA needs (per chain, at boot) + +Reserve enough native token on each chain so the master EOA can cover: + +| Item | When | Amount (examples) | +|------|------|--------------------| +| Paymaster deploy + initial funding | Only if paymaster not yet deployed | **Gas for 1 tx** (deploy) + **`paymasterFunding`** (e.g. **0.025** ETH from chain config default). So e.g. ~0.03 ETH on first run. | +| Worker funding | Every boot if `workerCount` > 0 and any worker below target | **Gas for 1 disperse tx** + **sum of (workerFunding − balance)** for each worker below `workerFunding`. With default **workerFunding = 0.001** and 3 workers at zero: **~0.003 ETH** + gas. | +| Paymaster already deployed | Every boot when no workers need funding | **0** (no paymaster or worker tx). | + +**Concrete example (first boot, one chain):** Paymaster not deployed, 3 workers, all at 0, defaults (`paymasterFunding` 0.025, `workerFunding` 0.001). Master EOA needs: **~0.025 ETH** (paymaster deploy + fund) + **~0.003 ETH** (disperse to 3 workers) + **gas for 2 txs** (deploy + disperse). So on the order of **~0.03 ETH** plus gas. Exact gas depends on the chain. + +**Later boots (same chain):** Paymaster already deployed → no paymaster tx. If workers are still at or above 0.001, no disperse. If you restarted and workers were drained: again **disperse shortfalls only** (e.g. 3 × 0.001 = 0.003 ETH) + gas for 1 tx. + +### Tracking and topping up paymaster balance + +Paymaster balance is **not** topped up automatically after the first deploy. Monitor **`/v1/info`**: the node reports paymaster balance and healthy/unhealthy per chain. When balance falls below `paymasterFundingThreshold`, the node marks it unhealthy and workers cannot relay until you top up. + +**To top up:** Go to the **official EntryPoint v7** contract address for that chain (see your chain config or a block explorer). Call **`deposit()`** (or the equivalent that credits the paymaster), with the **paymaster contract address as the beneficiary**. You can send native coin from any account; the deposit is credited to the paymaster. Repeat as needed to keep the node operational. + +### Worker EOA balance and paymaster refund + +Each **worker EOA** (or the master when used as the only worker) needs two things to relay: + +1. **Minimum native balance** — Set in chain config as `executor.workerFundingThreshold`. The worker must have at least this much native token to be able to submit an EVM transaction. This minimum **limits the maximum executable call gas limit** for that chain: if the threshold is too low, large transactions will fail. Set it high enough for the biggest transaction you expect to execute on that chain. +2. **Sufficient paymaster balance** — The worker pays gas via the paymaster (userOps). After a successful execution, the worker is **refunded** from the paymaster balance. So the worker’s own native balance stays roughly constant (or increases slightly if the refund exceeds the actual cost). If the paymaster balance is too low, refunds fail and workers cannot continue relaying. + +In short: keep the **master EOA** funded for one-time deploy and any manual top-ups you do; keep **paymaster** balance above threshold via manual top-ups; and set **workerFundingThreshold** high enough so workers can execute your largest expected transactions. + +## Logs + +- **Level**: `LOG_LEVEL` (e.g. `info`, `debug`). +- **Format**: JSON by default; set `PRETTY_LOGS=1` for development. +- **Callers**: Optional `LOG_CALLERS` for file/line in log lines. + +Use logs to debug quote/execution flow, queue delays, and RPC or token-storage errors. + +## Graceful shutdown + +The process handles SIGTERM (e.g. when using `tini` in Docker). Allow a few seconds for in-flight requests and job processing to finish before forcing kill. + +## Dependency failures + +### Redis unreachable + +- **Symptom**: Health check fails for Redis; queues do not advance; new quotes may not be stored. +- **Actions**: + - Verify Redis is running and reachable from the node (firewall, `REDIS_HOST`/`REDIS_PORT`). + - Restart Redis if needed; the node will reconnect when Redis is back. + +### Token Storage Detection unreachable or errors + +- **Symptom**: Simulations that need token balance overrides can fail (e.g. "Token overrides failed" or "SlotNotFound"). Health may show token-slot-detection as unhealthy for some chains. The node still executes using default gas limits and can fall back to a **Redis-backed cache** of slots for tokens that were successfully detected at least once (persistent in Redis). +- **Actions**: + - Ensure the service is running and `TOKEN_SLOT_DETECTION_SERVER_BASE_URL` is correct. + - Ensure the service has RPCs configured for the chains you use (or use fork mode / Anvil for chains where the RPC does not support token detection). + - Check the service logs for RPC or slot-detection errors. + +## Scaling + +- **API**: Increase `NUM_CLUSTER_WORKERS` for more concurrent HTTP handlers. +- **Execution**: Increase `MAX_EXTRA_WORKERS` (and provide `NODE_ACCOUNTS_MNEMONIC` or `NODE_ACCOUNTS_PRIVATE_KEYS`) to add more EOA workers for executor jobs. +- **Redis**: Use a Redis cluster or managed service for high availability and throughput. +- **Token Storage**: Run multiple instances behind a load balancer if needed; the node uses a single base URL. + +## Docker + +- **Node**: Use the official image with env vars for `NODE_ID`, `NODE_PRIVATE_KEY`, Redis, and `TOKEN_SLOT_DETECTION_SERVER_BASE_URL`. For Redis/token-storage on the host, use `host.docker.internal` (or equivalent) as hostname. +- **Token Storage**: Build from `apps/token-storage-detection/Dockerfile` and run with the same env vars as when running from source (RPCs, optional Redis, port). +- **Redis**: Use any Redis 7 image or the repo’s `compose.yml` for local dev. + +## Troubleshooting checklist + +1. Redis up and reachable? (`REDIS_HOST`, `REDIS_PORT`). +2. Token Storage Detection up and URL correct? (`TOKEN_SLOT_DETECTION_SERVER_BASE_URL`). +3. Node identity set? (`NODE_ID`, `NODE_PRIVATE_KEY`). +4. Chains and RPCs configured? (built-in or `CUSTOM_CHAINS_CONFIG_PATH`). +5. Master EOA and paymaster: master EOA funded on each chain? Paymaster balance above threshold? See [Master EOA and paymaster funding](#master-eoa-and-paymaster-funding). +6. `/v1/info`: any module unhealthy? +7. Logs: any repeated errors (RPC, token-slot, queue, or storage)? + +For architecture and dependency details, see [architecture.md](architecture.md) and [dependencies.md](dependencies.md). diff --git a/docs/run-and-maintain.md b/docs/run-and-maintain.md new file mode 100644 index 00000000..2db6d246 --- /dev/null +++ b/docs/run-and-maintain.md @@ -0,0 +1,155 @@ +# Run and maintain the MEE Node — tutorial + +This guide walks through running and maintaining a MEE Node step by step: why the master EOA exists, where fees go, what is auto-funded, chain and RPC requirements, chain config (including type and payment tokens), arbitrary payment, permit tokens, trusted gas tank (sponsored execution), and how to monitor the node and connect with the SDK. + +--- + +## 1. Why the master EOA exists and needs to be funded + +The **master EOA** is the node’s primary account (from **`NODE_PRIVATE_KEY`**). The private key can be supplied via environment variable or, in production/staging, from an **encrypted keystore** (`keystore/key.enc`) using **`ENV_ENC_PASSWORD`** — see [Operations — Configuration](operations.md#configuration). It is used to: + +- **Deploy** the node paymaster contract on each chain (once per chain when not yet deployed). +- **Fund** the paymaster with native token **only at first deployment** (see [Operations — Master EOA and paymaster funding](operations.md#master-eoa-and-paymaster-funding)). +- **Fund worker EOAs** at boot via the disperse contract (one tx per chain to top all workers up to `executor.workerFunding`). + +So the master EOA must hold enough **native token on each chain** where the node runs to cover deployment gas, initial paymaster funding (only when deploying), and worker funding at each boot. After that, paymaster balance is **not** auto-topped; the operator tops it up manually. + +--- + +## 2. Where the node receives fees and in which token + +Execution fees are sent to the **fee beneficiary** address, configured by **`NODE_FEE_BENEFICIARY`** (defaults to the master EOA if unset). Fees can be received as: + +- **Native token** — When the user pays in the chain’s native coin (e.g. ETH). +- **Configured payment tokens** — When the user pays in a token listed in the chain’s `paymentTokens` (e.g. USDC). The payment userOp transfers that token to the fee beneficiary. +- **Arbitrary tokens** — When arbitrary payment is enabled (see section 7). The user pays in a token that is not in `paymentTokens`; the token is still received at the fee beneficiary. Arbitrary tokens are **not** random mock or dead/illiquid meme tokens: only tokens **supported by the configured swap providers** (LiFi, Gluex) are accepted, so the node stays safe against unsupported or illiquid tokens. The node does not swap received tokens; the operator is responsible for swapping and rebalancing. + +--- + +## 3. What balances are automatically funded at first boot + +On **each startup**, for each configured chain the node: + +1. **Paymaster** — If the paymaster contract for that chain and master EOA is **not** deployed, the node deploys it and sends `paymasterFunding` (from chain config, e.g. 0.025 ETH) in the same tx. If the paymaster **is** already deployed, no funding is sent; you must top up manually. +2. **Workers** — If `executor.workerCount` > 0, the node funds worker EOAs from the master EOA via the **disperse** contract: **one transaction per chain**. Each worker is topped up to **`executor.workerFunding`** (default 0.001 native token). The amount sent in that tx is the **sum of shortfalls** (workerFunding − current balance) for each worker below target; workers already at or above target get 0. Workers are derived from `NODE_ACCOUNTS_MNEMONIC` or `NODE_ACCOUNTS_PRIVATE_KEYS`; the number funded is up to `min(workerCount, MAX_EXTRA_WORKERS, number of accounts)`. + +So: **paymaster** is only funded when it is first deployed; **workers** are funded every boot (to `workerFunding` each). Master EOA must have enough for deploy + paymasterFunding (first time only) and for the disperse sum + gas. For a detailed breakdown with examples (how much the master EOA needs, disperse math), see [Operations — Funding workers (how and when)](operations.md#funding-workers-how-and-when). + +--- + +## 4. Chain and RPC requirements + +Chains and RPCs used by the node must support: + +- **`debug_traceCall`** — Used for simulation (e.g. tracing handleOps). If the RPC does not support it, simulation may fail; for the Token Storage Detection service, use **fork mode** (e.g. Anvil) for such chains. See [Token Storage Detection README](../apps/token-storage-detection/README.md). +- **`eth_feeHistory`** — Used by the gas manager for EIP-1559 chains to get base fee and priority fee. Configurable via `feeHistoryBlockTagOverride` (e.g. `"latest"`, `"pending"`). +- Standard calls: `eth_getCode`, `eth_getBalance`, `eth_call`, contract reads (e.g. EntryPoint `balanceOf`), and for L2s the appropriate L1 fee estimation (e.g. Optimism `estimateL1Gas`, Arbitrum oracle). + +Ensure your RPC endpoints support these; otherwise quotes, simulation, or execution may fail. + +--- + +## 5. Chain ID and type (evm, optimism, arbitrum) — and how L1/L2 gas is calculated + +- **`chainId`** — Set in chain config to the chain’s numeric id (e.g. `"1"`, `"8453"`). Must match the key in your config file/directory. +- **`type`** — One of `"evm"` | `"optimism"` | `"arbitrum"`. Default is `"evm"`. + +**Difference between types:** + +- **`evm`** — Standard EVM chain. Gas is estimated using L2 fee only (gas price × gas limit). No L1 component. +- **`optimism`** — Optimism-style L2. Total cost includes **L2 gas** (base fee + priority fee) plus **L1 fee**. The node fetches L1 gas price from the L1 chain (`l1ChainId`), then uses the RPC’s `estimateL1Gas` (or equivalent) to get the L1 fee for the call data. Set **`l1ChainId`** (e.g. `"1"` for Ethereum). +- **`arbitrum`** — Arbitrum-style L2. Similarly, total cost = L2 gas + **L1 component**. The node uses the chain’s Arbitrum gas oracle contract to compute the L1 cost (e.g. `gasEstimateL1Component`). Set **`l1ChainId`** for the L1 chain. + +So: for L2s, set `type` and `l1ChainId` correctly so the gas estimator can compute L1 + L2 fees; for standard L1/EVM chains use `type: "evm"` (or omit). + +--- + +## 6. Role of payment tokens + +**Payment tokens** (in chain config `paymentTokens`) define **which tokens the node accepts as fee payment** on that chain. Each entry specifies the token contract (name, address, symbol), its price (fixed or oracle), and optionally **`permitEnabled`** (see section 8). Only tokens listed here can be used for fee payment unless **arbitrary token** support is enabled (section 7). The node uses this list to validate payment and to compute fee amounts in the chosen token. + +--- + +## 7. Activating arbitrary payment token support + +To allow users to pay fees in **any liquid token** (even if not in `paymentTokens`): + +1. Configure **at least one payment provider** via env: + - **LiFi**: `LIFI_API_KEY` + - **Gluex**: `GLUEX_API_KEY`, `GLUEX_PARTNER_UNIQUE_ID` +2. The node uses these providers to get **swap calldata** (route/quote). This is used only to (1) **validate** that the token is swappable (has liquidity) and (2) **determine how much** token is required to cover fees (exchange rate). Only tokens **supported by the configured swap providers** (LiFi, Gluex) are accepted—not random mock or dead/illiquid tokens—so the node is safe against unsupported or illiquid assets. The node **does not** execute the swap; the user’s payment in the arbitrary token is **received at the fee beneficiary** (`NODE_FEE_BENEFICIARY`). +3. As operator you must **periodically swap** received tokens and **rebalance** the node (e.g. keep paymasters funded with native token). See [Chain configuration — Arbitrary token support](chain-configuration.md#arbitrary-token-support). + +--- + +## 8. Permit-enabled tokens (ERC‑2612 / ERC‑20 Permit) + +For tokens that support **ERC‑20 Permit** (signature-based approval without a separate `approve` tx), set **`permitEnabled: true`** on the payment token in chain config. Then: + +- The SDK will request **quote-permit** by default for that token. +- Users can **gaslessly** approve spending from their EOA via a signature; the supertransaction can execute and pay fees in one flow without a prior approval transaction. + +So: for any payment token that implements Permit, set `permitEnabled: true` so the node and SDK use the permit flow. + +--- + +## 9. TRUSTED_GAS_TANK — sponsored supertransactions + +**`TRUSTED_GAS_TANK_ADDRESS`** (env) is the address of a **trusted gas tank**. When the **payment userOp** of a supertransaction is a **sponsored** payment (userOps indicate sponsorship) and the **sender** of that payment userOp is this trusted gas tank address: + +- The node treats it as **trusted sponsorship**. +- **Simulation of the payment userOp is skipped** (signature is still verified against the expected gas tank owner). +- The node **executes all other userOps** in the supertransaction **without requiring fees from the user** — the “payment” is considered covered by the trusted tank. + +So: set `TRUSTED_GAS_TANK_ADDRESS` to the gas tank contract (or EOA) that is allowed to sponsor supertransactions. When a payment userOp from that address is marked sponsored and its signature is valid, the node will execute the batch without expecting the user to pay fees. This is used for fully sponsored flows (e.g. Biconomy-hosted gas tank). + +--- + +### External gas tank sponsorship + +**External gas tank sponsorship** is when a **third party** (e.g. a dapp or relayer) runs their own **gas tank** (nexus smart account) and **sponsors transactions on behalf of their users**, while the **node still gets paid** for execution. The node is compensated by the gas tank; this is different from **trusted** sponsorship where the node sponsors without being paid. + +**Full flow (overview):** + +1. The **app or third party** requests a quote from the node, specifying **`paymentInfo.sender`** as the **nexus gas tank account** address (and the chosen payment token). The node returns a quote that includes the supertransaction userOps: **`userOps[0]`** is the **payment userOp** (fee payment from the gas tank to the node). +2. **Two signatures are required** before execution: + - **Gas tank private key** must sign the **payment userOp** (`userOps[0]` in the supertransaction userOps list from the generated quote). + - **End user** must sign the **quote** by signing the **supertransaction hash** — best done using the utility functions provided in the **AbstractJS SDK** (e.g. for quote signing and execution). +3. Once both signatures are attached, the client submits the signed quote to the node’s execute endpoint; the node runs the supertransaction and receives fees from the gas tank. + +**Requirements:** The token must be (1) **accepted by this node** (listed in the chain’s payment tokens or supported as arbitrary payment), and (2) the **balance** must exist on the nexus gas tank account, and the **nexus account must be deployed** on the given chain. + +**Concrete example:** For a full implementation of how third parties can set up an external gas tank and use it with the node (including quote, signing, and execution), see the **[MEE self-hosted sponsorship starter kit](https://github.com/bcnmy/mee-self-hosted-sponsorship-starter-kit)**. + +**Difference from trusted gas tank:** With **`TRUSTED_GAS_TANK_ADDRESS`**, the node **unconditionally** executes and does **not** require fees from the user (trusted mode). With **external gas tank sponsorship**, the node **does** require payment: the payment userOp moves funds from the third party’s gas tank to the node, so the node is compensated and the third party runs their own sponsorship flow. + +--- + +## 10. Monitoring the node and connecting with @biconomy/abstractjs + +**Monitoring:** + +- **GET /v1/info** — Aggregated health: Redis, Token Slot Detection, chains (RPC, paymaster, workers), simulator/executor queues, node wallets. Use this to confirm the node and dependencies are healthy and to track paymaster and worker balances. +- **Logs** — Use `LOG_LEVEL` (e.g. `debug`) and your logging stack to inspect quote, simulation, and execution flow. + +**When the node is fully set up and healthy:** Check that `/v1/info` shows all modules (Redis, chains, token-slot, simulator/executor, node wallets) in a healthy state. Then clients can use your node for quotes and execution by pointing the AbstractJS SDK at your node URL. + +**Connecting to your node via AbstractJS:** + +To connect to your MEE Node (e.g. after it is running and healthy), use the **`@biconomy/abstractjs`** SDK and pass your node’s base URL as the **`url`** option when creating the MEE client. All quote, quote-permit, and execute flows then go to your node instead of the default Biconomy network. + +```ts +import { createMeeClient } from "@biconomy/abstractjs"; + +const meeClient = await createMeeClient({ + account: orchestrator, // your orchestrator/smart account + url: "https://your-mee-node-url", // your MEE node base URL (e.g. https://mee.example.com) + // apiKey: "optional-for-rate-limiting" +}); +``` + +Use the client for quote, quote-permit, and exec as usual; traffic is sent to your node. Without `url`, the SDK uses the default Biconomy network. + +--- + +For startup order, dependency failures, and paymaster top-up details, see [Operations](operations.md). For chain config fields and payment-token/oracle setup, see [Chain configuration](chain-configuration.md).