Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions apps/be/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Backend API: Postgres + pgvector, NEP-413 login, run/event storage + SSE, encryp
- **Paid data (#150, `src/shop.ts`):** `GET /internal/v1/shop/{item}` answers 402 with a price and `SHOP_PAY_TO`; with `?tx=` it serves a live NEAR RPC report only for this Mind's own `sent` payment row (to `SHOP_PAY_TO`, at least the price, last hour), confirmed by RPC `tx` (success, signer = the Mind's wallet), and records the hash in `shop_purchases` (009) so it is used once. Plain `rpc.testnet.near.org` forgets transactions after a few days, hence the one-hour window. `SHOP_PAY_TO` must be on the paying wallet's address list. Tests use the fake OutLayer's `/rpc` (`tx`, `status`, `gas_price`).
- **Several sellers (#176, Best Price Buyer):** `SHOP_SELLERS` (`account:multiplier,…`) lists sellers `s1`, `s2`, … by their place in the list; empty means `SHOP_PAY_TO` alone as `s1`. A bad list stops the backend at start. `?quotes=1` lists them cheapest first with `onYourList`, read from the paying wallet's own OutLayer policy (`addressRuleOf` in `wallets.ts`), so the Mind only picks sellers OutLayer would let it pay. A 402 without `seller` offers the cheapest. A payment counts for the seller its `wallet_payments.to_account` names, at that seller's price; with `?seller=` it must be that seller. Every seller must also be on the demo wallet's list for try-outs, which is a policy the demo wallet's owner signs in Meteor. Intended (11 Nov, not set yet): `SHOP_SELLERS="lin1er.testnet:1.0,testing7.testnet:0.8"`.
- **Negotiation (agents track, `src/shop.ts`):** `?seller=sN&offer=<NEAR>` asks one seller to sell for less. Each seller has a floor, a share of its price (third `SHOP_SELLERS` field `account:multiplier:floor`, default 0.8): at or above it the offer is `accepted` (never above the list price), below it the seller `countered`s once, halfway to its price but not under the floor. Stateless; the paid step carries the same `offer` and checks the payment against the agreed price (an offer under the floor never lowers it). The engine's `make_offer` and `buy_data.offerNear` call exactly this; keep the shape (`docs/api-contract.md`).
- **Payments CSV (realworld track):** `GET /v1/minds/{id}/payments.csv` (`paymentsCsv` in `src/payments.ts`), owner only. "Approved by" is read from the run events of the call that made the payment (`tool.result.result.paymentId` → same `toolCallId`'s `tool.called.approvedBy` / `tool.approval_required`), in one query with CTEs: per-row subqueries made it 3× slower under load. Cells that start like a formula get a leading `'`.
- **Payments CSV (realworld track):** `GET /v1/minds/{id}/payments.csv` (`paymentsCsv` in `src/payments.ts`), owner only. "Approved by" is read from the run events of the call that made the payment (`tool.result.result.paymentId` → same `toolCallId`'s `tool.called.approvedBy` / `tool.approval_required`), in one query with CTEs: per-row subqueries made it 3× slower under load. Cells that start like a formula get a leading `'`. **Payments Excel (#298):** `GET /v1/minds/{id}/payments.xlsx` (`paymentsXlsx`), the same rows (`paymentRows`) as a real workbook from `src/xlsx.ts`, a small writer with no dependency (stored zip, `zlib.crc32`). Cells are typed (date serial in UTC with a date format, amounts as numbers, the explorer link as a hyperlink, text as inline strings, which are never formulas), with a bold frozen header and a filter. It exists because a comma CSV opens in one column in Excel on a comma-decimal locale (Indonesia), and no single CSV separator fits every locale. An amount a JS number can't hold exactly stays text.
- **Meta-transaction relayer (realworld track, `src/relay.ts`):** `POST /v1/relay { signedDelegate }` (NEP-366, base64 Borsh). The relayer account (`RELAYER_ACCOUNT_ID`, `RELAYER_PRIVATE_KEY`) pays gas **and any inner deposit** (docs.near.org "Meta Transactions", read 13 Nov), and the registry refunds storage to the caller, so only `unequip_skill` / `set_wallet` with exactly 1 yoctoNEAR are relayed; `register_mind` and `equip_skill` need storage (and a price) and are not. Everything that can be refused is refused before sending (sender = signed-in account, receiver = `REGISTRY_CONTRACT_ID`, gas, ed25519 signature over sha256(NEP-461 prefix + delegate), expiry ≤ 1000 blocks, full-access key, nonce) so a bad request costs no gas. 10 per owner per hour, in memory. RPC: `NEAR_RPC_URL`, default `https://test.rpc.fastnear.com`. `borsh` 2.0.0 is a direct dependency for decoding (same version near-api-js uses). Not run on testnet yet: no relayer account is funded; a wallet that signs delegate actions without sending them (Meteor?) is not proven either.
- **Mind as Seller (#181, `src/offers.ts`, migration 011):** a Mind publishes an offer with its own wallet token (`PUT /internal/v1/offers/{item}`, stored in `mind_offers`); another Mind buys it with `GET /internal/v1/minds/{sellerMindId}/offers/{item}[?tx=]`, the same 402 and receipt check as the shop but `payTo` is the seller Mind's own wallet address and the RPC `tx` must show receiver = that address; hashes are used once (`offer_purchases`). Try-out and playground Minds can neither sell nor be bought from (the shared demo wallet is nobody's own): `403 demo_not_allowed` / `404 no_offer`. The buyer's wallet must list the seller's 64-hex address, or OutLayer refuses the payment (`policy_denied`).
- **Telegram (#168, `src/telegram.ts`, migration 010):** one bot per Mind, made by the owner with @BotFather. `PUT /v1/minds/{id}/telegram { token }` checks it with `getMe`, refuses a bot another Mind uses (`409 telegram_bot_in_use`), stores it AES-GCM with AAD `telegram:{mindId}`, and either sets a webhook (`TELEGRAM_WEBHOOK_BASE` + `/v1/telegram/{mindId}/webhook`, with a random `secret_token`) or, locally, deletes it so `TELEGRAM_POLLING=true` can poll `getUpdates`. The answer has a one-time `t.me/<bot>?start=<code>`: the private chat that sends that code becomes the owner; every other chat is ignored without a reply. `POST /v1/telegram/{mindId}/webhook` has no session: the `X-Telegram-Bot-Api-Secret-Token` header must match (`401` otherwise), it answers `200` at once and handles the update after; `last_update_id` drops redelivered updates. An owner message runs as a chat through `runTelegramChat` (`src/server.ts`: `relayRun` into an in-memory sink, so events, threads, the live view and try-out limits are the same as the app), in one thread per bot (`thread_id`), in UTC (no time zone from Telegram). **Streaming (#222, `LiveReply` in `src/telegram.ts`):** the sink passes each relayed event to Telegram as it is written (`memorySink(onEvent)`; `relayRun` and the app path are unchanged). "typing…" is sent every 4 s until the first text shows and again while a tool runs (Telegram clears it after 5 s). The first text is sent at once, then the same message is edited as it grows, at most once per 1.5 s (`telegramTiming`), each time the whole text so far through the converter (a half-written mark stays literal until its pair arrives, so it is always valid HTML); past 4096 characters it continues in a new message. `run.finished` triggers the last edit with the complete formatted reply, so the owner does not wait for `run.verified`. An edit answered "message is not modified" is ignored. Approvals still come as their own messages after the reply. The reply after an approval button press is not streamed yet (sent whole when the resumed run ends). **Rich text (#221, `src/telegramFormat.ts`):** every bot message is sent with `parse_mode: "HTML"`. The Mind's Markdown is converted by our own small converter (no library): bold, italic, strikethrough, inline and fenced code (`<pre><code class="language-x">`), links `[text](url)` and bare URLs (http/https only; any other scheme stays visible as text), headings as bold lines, lists as "•"/"1." lines, blockquotes, tables as one line per row; all other text is escaped (`&lt; &gt; &amp; &quot;` only). `telegramChunks` splits at 4096 between blocks or lines, never inside a tag, so each message is valid on its own. A `400` "can't parse entities" resends that message as plain text (`plainOf`). Approval cards (#208) use the same path, but only the title is bold and every other word is escaped, never read as Markdown: their words come from tool arguments (an email subject, an event title someone else wrote), so a stranger cannot inject a link into a message that asks the owner to approve. **Approvals happen in Telegram too (#208, Axel's call 11 Nov; replaces "never in Telegram"):** each approval a Telegram run asks for is its own message (`askOwner`) with the app card's checks or details (`src/approvals.ts`, ported from apps/fe `wallet/data/checks.ts` and `runs/data/approvalDetails.ts`; keep the words in step, and give every new approval tool a case in both, or Telegram shows only its name; long values are cut by `clip` to stay under Telegram's 4096 characters) and buttons Approve / Decline / Open in app (`/minds/{id}/chat/{threadId}`); no Approve when the policy would refuse or the wallet holds less than the payment (`next: 'fund'`, #228; an unfunded wallet's account isn't on chain yet and OutLayer reads its balance as `"0"`, spike 01, so it counts as 0), a button to `/minds/{id}/wallet` instead. Order of reasons: frozen, limit, today's limit, list, balance. Today's limit (#149) is OutLayer's rule `spentTodayNear + amount > dailyNear` (a payment exactly on the cap goes through; the day starts 00:00 UTC), and a `policy_denied` message names it too, since a refused payment doesn't count toward the day. A wallet read that fails still shows no checks and keeps Approve (OutLayer decides). `callback_data` is an opaque one-time code in `telegram_approvals` (013); only `callback_query.from.id` = the owner chat id counts, everyone else gets "Only the owner can approve"; `answerCallbackQuery` is always called; the first press uses up both codes. A press resumes through `resumeRun` (`src/server.ts`), the same function as the app's approve/decline route, then edits the message to the result (never "rejected on-chain", no tx link for a refusal) and sends the Mind's reply and any next approval. `resumeRun` accepts only a call of the run's current pause (`isAwaiting` in `src/runs.ts`: at most one `run.finished` after its `tool.approval_required`) and claims the run (`awaiting_approval` → `running` in one update, undone if the engine refuses), so the app and Telegram cannot resume one run twice. `allowed_updates` is `["message", "callback_query"]`, and every start re-runs `setWebhook` for all bots (`registerTelegramWebhooks`) so bots linked before #208 get presses; a URL button Telegram refuses (`400`) is resent as a link in the text. `DELETE` deletes the webhook (best effort) and the row. Polling skips a bot that errors, so one bad row cannot stop the others. Tests use `test/fake-telegram.ts` (also `answerCallbackQuery`, `editMessageText`, `editMessageReplyMarkup`, `refuseLocalUrls`, `refuseHtmlOnce`, `telegramHtmlError`, which refuses HTML Telegram would refuse, `deleteMessage`, and "message is not modified"; every call has its time `at`) and the fake engine's `fake.scripts` (scripted replies with slow tools, text pacing and verification delay; `fake.timings`) with fresh bot ids per run; the fake engine's `pay <n> NEAR to <account>[ and …]` asks for `send_payment` one pause at a time and pays through the backend's internal route (`fake.backendUrl`). A labelled link shows its host after it (`[Your bank](http://evil.example/login)` reads "Your bank (evil.example)"), unless the label names it; the plain-text fallback drops that note since it shows the whole URL (Lin1er, #223).
Expand All @@ -57,7 +57,7 @@ Backend API: Postgres + pgvector, NEP-413 login, run/event storage + SSE, encryp

## Commands

- `bun run dev` (port 4000, applies migrations at start), `bun run migrate`, `bun run test:unit` (offline), `bun run test` (unit + integration: local Postgres + fake engine), `bun run load:shop` (the shop's quotes and offers and the payments CSV under load, fake engine and fake OutLayer; fails on any wrong answer), `bun run load` (fake engine and fake Telegram; fails if a run loses events, if a paused run answered twice at once is resumed other than once, or if a Telegram reply streamed alongside the app traffic does not end with the full text), `bun run typecheck`.
- `bun run dev` (port 4000, applies migrations at start), `bun run migrate`, `bun run test:unit` (offline), `bun run test` (unit + integration: local Postgres + fake engine), `bun run load:shop` (the shop's quotes and offers and the payments CSV and Excel file under load, fake engine and fake OutLayer; fails on any wrong answer), `bun run load` (fake engine and fake Telegram; fails if a run loses events, if a paused run answered twice at once is resumed other than once, or if a Telegram reply streamed alongside the app traffic does not end with the full text), `bun run typecheck`.
- Postgres: `docker compose up -d postgres` from the repo root; `docker compose stop` when done. If port 55432 is taken by another project, run it on another host port with a compose override (`ports: !override ["55433:5432"]`) and pass `DATABASE_URL=…:55433/hellonear` to `bun run test` / `bun run load`.
- Schema changes: add a new `migrations/NNN_name.sql` (never edit an applied one) and update `docs/api-contract.md` §5.
- Tests never call NEAR AI or the NEAR RPC: `test/fake-engine.ts` stands in for the engine and `setProvider()` fakes the access-key lookup; signatures are real. Run `bun run test` and `bun run load` before every push/merge that touches `apps/be`, and report results in the PR. CI: `.github/workflows/be.yml`.
Expand Down
10 changes: 8 additions & 2 deletions apps/be/load/shop.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Short local load test of the paths the track work changed: the paid-data shop's price list and negotiation
// (`/internal/v1/shop/{item}?quotes=1` and `?seller=&offer=`, with the run's wallet token, as the engine calls them) and
// the payments CSV (`GET /v1/minds/{id}/payments.csv`, each Mind holding 200 payments with their runs' events). Every
// the payments CSV and Excel file (`GET /v1/minds/{id}/payments.csv` and `.xlsx`, each Mind holding 200 payments with their runs' events). Every
// answer must be 200 with the right shape: an offer is accepted or countered at the expected price, and the CSV has one
// line per payment. Fake engine and fake OutLayer, local Postgres. Never point this at a public endpoint.
// Run: bun run load:shop (env: VUS=10, DURATION_S=10)
Expand Down Expand Up @@ -58,7 +58,7 @@ async function user() {
}

const users = await Promise.all(Array.from({ length: VUS }, user))
const latencies: Record<string, number[]> = { quotes: [], offer: [], csv: [] }
const latencies: Record<string, number[]> = { quotes: [], offer: [], csv: [], xlsx: [] }
let errors = 0
const fail = (what: string) => { if (errors++ < 5) console.error(what) }
async function timed(name: string, f: () => Promise<boolean>) {
Expand Down Expand Up @@ -93,6 +93,12 @@ await Promise.all(users.map(async (u, n) => {
const lines = text.trimEnd().split('\r\n')
return r.status === 200 && lines.length === PAYMENTS_PER_MIND + 1 && lines.filter((l) => l.includes(',your rule,')).length === PAYMENTS_PER_MIND / 2
})
if (i % 3 === 1) await timed('xlsx', async () => {
const r = await fetch(`${base}/v1/minds/${u.mindId}/payments.xlsx`, { headers: { cookie: u.cookie } })
const buf = Buffer.from(await r.arrayBuffer())
// A zip (PK) holding one row per payment plus the header.
return r.status === 200 && buf.subarray(0, 2).toString() === 'PK' && (buf.toString('latin1').match(/<row r="/g)?.length ?? 0) === PAYMENTS_PER_MIND + 1
})
}
}))
const seconds = (Date.now() - started) / 1000
Expand Down
49 changes: 37 additions & 12 deletions apps/be/src/payments.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { newId } from './crypto.ts'
import { pool, query } from './db.ts'
import { HttpError, UPSTREAM_ERROR } from './http.ts'
import { keyOf, outlayerRaw, toNear, toYocto } from './wallets.ts'
import { xlsx } from './xlsx.ts'

export const sendPaymentSchema = z.object({
to: z.string().toLowerCase().regex(/^(?=.{2,64}$)(([a-z\d]+[-_])*[a-z\d]+\.)*([a-z\d]+[-_])*[a-z\d]+$/, 'a NEAR account id'),
Expand Down Expand Up @@ -125,7 +126,9 @@ export function csvCell(v: string): string {
const safe = /^[=+\-@\t\r]/.test(v) ? `'${v}` : v
return /[",\r\n]/.test(safe) ? `"${safe.replace(/"/g, '""')}"` : safe
}
export async function paymentsCsv(mindId: string, limit = 10_000): Promise<string> {
export interface PaymentRow { at: Date; status: string; direction: 'out'; amountNear: string; counterparty: string; purpose: string; approvedBy: string; txHash: string; explorer: string; note: string }
// The rows behind both downloads (CSV and Excel), newest first.
export async function paymentRows(mindId: string, limit = 10_000): Promise<PaymentRow[]> {
// One pass: the payment calls in this Mind's runs, then who approved each, then the rows (no per-row subqueries).
const rows = await query<{ id: string; created_at: Date; status: PaymentStatus; amount_yocto: string; to_account: string; tx_hash: string | null; message: string | null; shop_item: string | null; offer_item: string | null; approved_by: string | null }>(
`with calls as (
Expand All @@ -146,18 +149,40 @@ export async function paymentsCsv(mindId: string, limit = 10_000): Promise<strin
left join (select distinct on (payment_id) payment_id, item from shop_purchases where mind_id = $1) s on s.payment_id = p.id
left join (select distinct on (payment_id) payment_id, item from offer_purchases where buyer_mind_id = $1) o on o.payment_id = p.id
where p.mind_id = $1 order by p.created_at desc limit $2`, [mindId, limit])
const lines = [CSV_HEADER, ...rows.map((p) => [
p.created_at.toISOString().replace('T', ' ').slice(0, 19),
CSV_STATUS[p.status] ?? p.status,
'out',
toNear(BigInt(p.amount_yocto)),
p.to_account,
p.shop_item ? `data report: ${p.shop_item}` : p.offer_item ? `bought from another Mind: ${p.offer_item}` : '',
p.approved_by ?? '',
p.tx_hash ?? '',
p.tx_hash ? `https://testnet.nearblocks.io/txns/${encodeURIComponent(p.tx_hash)}` : '',
p.message ?? '',
return rows.map((p) => ({
at: p.created_at,
status: CSV_STATUS[p.status] ?? p.status,
direction: 'out' as const,
amountNear: toNear(BigInt(p.amount_yocto)),
counterparty: p.to_account,
purpose: p.shop_item ? `data report: ${p.shop_item}` : p.offer_item ? `bought from another Mind: ${p.offer_item}` : '',
approvedBy: p.approved_by ?? '',
txHash: p.tx_hash ?? '',
explorer: p.tx_hash ? `https://testnet.nearblocks.io/txns/${encodeURIComponent(p.tx_hash)}` : '',
note: p.message ?? '',
}))
}

export async function paymentsCsv(mindId: string, limit = 10_000): Promise<string> {
const lines = [CSV_HEADER, ...(await paymentRows(mindId, limit)).map((p) => [
p.at.toISOString().replace('T', ' ').slice(0, 19), p.status, p.direction, p.amountNear, p.counterparty, p.purpose, p.approvedBy, p.txHash, p.explorer, p.note,
])]
// A BOM so Excel opens it as UTF-8; CRLF line ends per RFC 4180.
return '' + lines.map((l) => l.map(csvCell).join(',')).join('\r\n') + '\r\n'
}

// The same rows as an Excel workbook (#298): typed cells, so the columns, the amounts and the dates come out right in
// any locale, which a CSV can't do (Excel on a comma-decimal locale splits on `;`). An amount that a JS number can't
// hold exactly (more than ~15 significant digits) stays text rather than being rounded.
export async function paymentsXlsx(mindId: string, limit = 10_000): Promise<Buffer> {
const amount = (s: string) => (String(Number(s)) === s ? Number(s) : s)
return xlsx({
name: 'Payments',
header: CSV_HEADER,
widths: [20, 30, 10, 14, 28, 34, 12, 48, 60, 40],
rows: (await paymentRows(mindId, limit)).map((p) => [
p.at, p.status, p.direction, amount(p.amountNear), p.counterparty, p.purpose, p.approvedBy, p.txHash,
p.explorer ? { text: p.explorer, link: p.explorer } : '', p.note,
]),
})
}
Loading
Loading