diff --git a/README.md b/README.md index f8d0f28..47f8cc8 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,28 @@ +
+ # EditAI -**An AI video editor you talk to.** Say "remove the silences" or "caption every clip", and an agent -makes the edit on your real timeline: it reads the project, decides which cuts to make, and applies -them. Anything destructive stops and asks you first. +**An AI harness for video editing. You describe the edit; an agent makes it on your real timeline.** + +[**Live site**](https://editai-agent.vercel.app) · +[Quick start](#getting-started) · +[The 16 tools](#the-16-timeline-tools) · +[Add your own MCP](#reaching-past-the-timeline) · +[Review evidence](#code-review-evidence-qodo) [![License: MIT](https://img.shields.io/badge/license-MIT-green?style=flat)](LICENSE) +[![Built on TrueForge](https://img.shields.io/badge/harness-TrueForge-7c5cff?style=flat)](https://trueforge.dev) +[![MCP](https://img.shields.io/badge/protocol-MCP-7c5cff?style=flat)](https://modelcontextprotocol.io) +[![Tests](https://img.shields.io/badge/tests-31%20passing-3aa39b?style=flat)](#tests) +[![Reviewed by Qodo](https://img.shields.io/badge/reviewed%20by-Qodo-e0a63b?style=flat)](#code-review-evidence-qodo) + +
+ +--- + +Say "remove the silences" or "caption every clip", and an agent makes the edit on your real +timeline: it reads the project, decides which cuts to make, and applies them. Anything destructive +stops and asks you first. The tedious parts of editing are the ones a machine should do. Cutting dead air out of a twenty-minute take is thirty minutes of scrubbing; here it is one sentence and one approval click. @@ -12,39 +30,57 @@ Captioning every clip means transcribing each one by hand; here a sub-agent hand parallel and the captions land on their own track, timed. The agent does the mechanical work, and you keep the decisions: it proposes, you approve, and every edit is undoable. -### What it can do today +**Three layers, one conversation.** Depending on what you ask for, the same agent drives your own +editing UI, the editor in this repo, or the ffmpeg encode underneath it. You do not choose the +layer; the request does. -| Ask for this | What the agent does | -| --- | --- | -| *"Remove the silences"* | Finds every silent range on the voice track and ripple-deletes it across all tracks, closing the gaps. Verified: 24s → 21.1s, exactly the 2.9s of silence. | -| *"Caption every video clip"* | Fans out one sub-agent per clip to transcribe in parallel, merges the results, and lays timed captions on a new track. | -| *"Cut the intro to 3 seconds"* | Trims the clip, keeping the media in sync by moving its source offset. | -| *"Duck the music under the voiceover"* | Sets clip volume. | -| *"Split this at 15 seconds"* | Cuts a clip in two, both halves still frame-accurate. | -| *"Export it at 1080p"* | Renders, after you approve. | +--- -### The agent tools +## Watch it work -The timeline is exposed to the agent as **16 [MCP](https://modelcontextprotocol.io) tools**, not as -a prompt describing a timeline. The agent calls real functions against real state: +The one operation worth showing end to end, because the numbers are checkable: + +``` +you ▸ remove the silences -- **Read:** `get_project`, `transcribe_clip`, `find_silences`, `detect_beats`, `list_changes` -- **Write:** `split_clip`, `trim_clip`, `move_clip`, `set_volume`, `add_text`, `add_captions`, `undo` -- **Destructive:** `delete_clip`, `ripple_delete`, `remove_silences` -- **Gated:** `export_project` +agent → find_silences(min_duration: 0.5, track_id: "a1") + ← 3 ranges: 3.2-4.1, 9.6-10.4, 16.8-18.0 (2.9s total) -Every tool validates its input and returns errors to the model as data, so a stale clip id becomes -a correction the agent recovers from rather than a failed turn. The destructive four are published -with MCP's `destructiveHint` annotation, which is what makes the harness stop and ask you before -they run. + "Three silent ranges, 2.9s in all. This will cut every track. Remove them?" -Because it is MCP, the same agent can reach anything else that speaks MCP: web search, your issue -tracker, an internal API you wrap yourself. Connectors attach by name and authorize in chat. +you ▸ [Approve] + +agent → remove_silences(min_duration: 0.5) + ← duration 24.0s → 21.1s, every track cut, gaps closed + + "Cut 2.9s of dead air across every track. Say undo to put it back." +``` + +24.0 down to 21.1 is exactly the 2.9 seconds of silence, with picture, sound and captions still in +sync. There is an animated version of this on the [live site](https://editai-agent.vercel.app). + +## What you can ask for + +| Ask | What the agent actually does | Tools | +| --- | --- | --- | +| *"Remove the silences"* | Finds every silent range on the voice track, tells you what will go, then ripple-deletes it across all tracks, closing the gaps. Verified: 24s → 21.1s. | `find_silences` → `remove_silences` | +| *"Caption every video clip"* | Fans out one sub-agent per clip to transcribe in parallel, merges the results (including sentences that straddle a cut), and lays timed captions on their own track. | `transcribe_clip` ×N → `add_captions` | +| *"Cut this to the beat"* | Reads the tempo off the music track and splits on the beat grid. Both halves stay frame-accurate. | `detect_beats` → `split_clip` | +| *"Trim the intro to 3 seconds"* | Trims the clip and moves its source offset by the same amount, so the picture does not jump. | `trim_clip` | +| *"Duck the music under the voiceover"* | Sets clip volume where the voice track is speaking and restores it where it is not. | `get_project` → `set_volume` | +| *"Grade it warmer and add grain"* | Writes the ffmpeg filter graph, runs it in the media sandbox, then probes the output to confirm it matches the intent. | `probe_media` → `run_ffmpeg` | +| *"Kill the room tone"* | Denoises the voice track in the sandbox, leaving the original file untouched beside it. | `run_ffmpeg` (`afftdn`) | +| *"Put their logo in the corner"* | Searches the live web through Bright Data, scrapes the asset, and brings it into the project. | `search_engine` → `scrape_as_markdown` | +| *"Export it at 1080p"* | Renders, after you approve. | `export_project` | + +Motion graphics, transitions and animation work the same way: either as an ffmpeg filter graph in +the media sandbox, or by attaching an MCP server that specialises in them. See +[Reaching past the timeline](#reaching-past-the-timeline). ## How it works -EditAI is built on [TrueForge](https://trueforge.dev), an open-source agent harness. The harness -runs the agent loop; EditAI supplies the domain. +EditAI is built on [TrueForge](https://trueforge.dev), TrueFoundry's open-source agent harness. The +harness runs the agent loop; EditAI supplies the domain. ``` browser harness domain @@ -53,7 +89,15 @@ runs the agent loop; EditAI supplies the domain. │ (apps/web) │ + SSE │ agent loop │ │ 16 timeline │ │ │ │ approvals │ │ tools │ │ timeline ◄──┼───────────┼──────────────┼── SSE ───┤ project store │ -└──────────────┘ └──────────────┘ └──────────────────┘ +└──────────────┘ └──────┬───────┘ └──────────────────┘ + │ MCP + ┌─────────────┼──────────────┐ + ▼ ▼ ▼ + ┌──────────────┐ ┌─────────┐ ┌──────────────┐ + │ ffmpeg │ │ bright- │ │ anything │ + │ sandbox │ │ data │ │ else that │ + │ (container) │ │ (web) │ │ speaks MCP │ + └──────────────┘ └─────────┘ └──────────────┘ ``` The editor never calls a model. It creates a session, streams turn events, renders tool calls and @@ -68,22 +112,135 @@ server's event stream, so an edit the agent makes shows up in the UI as it happe | Sessions that survive a reload | `apps/web/src/components/editor/use-assistant.ts` replays turns and re-attaches to a running one | | Live web research | `bright-data` connector: `search_engine` and `scrape_as_markdown`, attached deferred so it costs no context until a task needs it (see [docs/brightdata.md](docs/brightdata.md)) | | Any model provider | `apps/agent/scripts/setup.ts` registers whichever API keys are present, including any OpenAI-compatible endpoint | -| Sandboxed execution | Two layers: the harness sandbox (Daytona, configured automatically when `DAYTONA_API_KEY` is set) for general code, and `packages/ffmpeg-sandbox` for media work | +| Sandboxed execution | Two layers: the harness sandbox (Daytona) for general code, and `packages/ffmpeg-sandbox` for media work | +| Domain know-how | `skills/video-editing/SKILL.md`, loaded by the agent whenever a task touches ffmpeg | + +## The 16 timeline tools + +The timeline is exposed to the agent as **16 [MCP](https://modelcontextprotocol.io) tools**, not as +a prompt describing a timeline. The agent calls real functions against real state. + +| Tool | Kind | Arguments | What it does | +| --- | --- | --- | --- | +| `get_project` | read | | Tracks, clips, media metadata, exports. Call it first; ids change after edits. | +| `list_changes` | read | `limit` | Recent edits, oldest first. | +| `transcribe_clip` | read | `clip_id` | Speech inside one clip, as timed segments in timeline seconds. | +| `find_silences` | read | `min_duration`, `track_id` | Silent ranges on a track. Preview only. | +| `detect_beats` | read | `track_id` | Beat timestamps derived from the track's tempo. | +| `split_clip` | write | `clip_id`, `at` | Cuts a clip in two. Returns both halves. | +| `trim_clip` | write | `clip_id`, `start?`, `end?` | New in/out points, keeping media in sync via `sourceOffset`. | +| `move_clip` | write | `clip_id`, `start?`, `track_id?` | New start time, or another track of the same kind. | +| `set_volume` | write | `clip_id`, `volume` | Clip volume, 0 to 100. | +| `add_text` | write | `text`, `start`, `duration`, `track_id` | A title or caption on a text track. | +| `add_captions` | write | `segments[]`, `track_label` | Timed captions on the captions track, creating it if needed. | +| `undo` | write | | Reverts the most recent change. | +| `delete_clip` | **destructive** | `clip_id` | Removes a clip, leaving a gap. | +| `ripple_delete` | **destructive** | `start`, `end` | Removes a range from every track and closes the gap. | +| `remove_silences` | **destructive** | `min_duration`, `track_id` | Ripple-deletes every silence over the threshold. | +| `export_project` | **approval** | `format`, `resolution` | Renders the timeline to a file. | + +Every tool validates its input with zod and returns errors to the model **as data**, so a stale clip +id becomes a correction the agent recovers from rather than a failed turn. + +The four gated tools are published with MCP's `destructiveHint` annotation, and the agent declares +`require_approval_for_tools: ["@destructive", "export_project"]`. That turns the annotation into a +pause: the harness stops the turn, the editor shows the tool and its arguments, and the run only +continues once a person allows or denies it. + +Three semantics worth knowing before you read the code: + +- **`sourceOffset` keeps media in sync.** Trimming a clip's start moves its offset into the source + file by the same amount, so the picture does not jump. +- **Ripple delete is the interesting operation.** Removing a range cuts every track, splits any clip + straddling the range, and shifts everything after it left. `remove_silences` applies it once per + silence, from the end backwards, so earlier ranges stay valid. +- **Captions merge across clip boundaries.** Fanning captioning out per clip means a sentence + spanning a cut is reported twice, clamped to each side; `add_captions` merges those back into one. + +See [apps/agent/README.md](apps/agent/README.md) for the full tool reference and timeline semantics. + +## Reaching past the timeline + +Because everything is MCP, the same agent can reach anything else that speaks MCP: web search, a +motion-graphics server, your issue tracker, an internal API you wrap yourself. **Adding a capability +is a name in a list and a restart, not a release.** + +```bash +# keyless, the default +EDITAI_CONNECTORS=exa bun run setup + +# header auth +BRIGHT_DATA_MCP_HEADER="Authorization: Bearer " \ + EDITAI_CONNECTORS=exa,bright-data bun run setup + +# OAuth: dynamic client registration, nothing to configure here. +# The first time the agent reaches for it, the turn pauses with an authorize +# URL and the editor shows a Connect button. Verified against Linear. +EDITAI_CONNECTORS=exa,linear bun run setup + +# your own server, attached the same way +EDITAI_CONNECTORS=exa,bright-data,motion-graphics bun run setup +``` + +Extra connectors attach **read-only and deferred**, so a connector you rarely use costs nothing in +context until the agent actually reaches for it. The tools go live on the agent's next turn, in the +same conversation. + +**Bright Data** is the one wired up and verified: `search_engine`, `search_engine_batch`, +`scrape_as_markdown`, `scrape_batch` and `ask_brightdata_assistant`. It is what lets you say "put +their logo in the corner" and get the current logo rather than a model's memory of one. +Setup details and the auth gotcha are in [docs/brightdata.md](docs/brightdata.md). + +## Where the code runs + +Generated code never runs on the host. Two layers cover the two kinds of work. + +**Harness sandbox (Daytona).** `setup.ts` registers the provider when `DAYTONA_API_KEY` is present, +and the agent's `exec` tool then runs in a remote sandbox. Verified end to end against the running +harness: + +``` +sandbox.created sandbox_id: v1:daytona:default.e17058ee-... +exec python3 -c "print(sum(int(x)**2 for x in range(1,101)))" +tool.response {"success":true,"response":{"exitCode":0,"result":"338350\n"}} +``` + +**Media sandbox (`packages/ffmpeg-sandbox`).** ffmpeg and ffprobe are not safe to point at +agent-supplied arguments on the host, so every invocation runs in a throwaway container: +`--network none`, capped memory and CPU, `--pids-limit 256`, a non-root user, a single mounted +workspace, and a hard timeout. It exposes four tools: + +| Tool | What it does | +| --- | --- | +| `list_media` | What is in the workspace | +| `probe_media` | Duration, resolution, codecs, fps | +| `run_ffmpeg` | Runs ffmpeg with an argument array | +| `run_python` | Glue work, parsing, arithmetic | + +`run_python` is annotated `destructiveHint: true`, so the harness shows the script and waits for +approval before it runs: a script with the workspace mounted read-write can delete the source media, +and the container is not a defence against that. + +The split is deliberate. The harness sandbox is for computation the agent should not estimate; the +media sandbox is for work that must reach the media files. ## Getting started -You need [Bun](https://bun.sh) and Node 22.14+ (for the harness). +You need [Bun](https://bun.sh) and Node 22.14+ (for the harness). Docker is needed only for the +media sandbox. ```bash +git clone https://github.com/deonmenezes/edit-ai.git +cd edit-ai bun install -# 1. the harness +# 1. the harness (separate terminal) npx @truefoundry/trueforge@latest # http://localhost:8790 # 2. the timeline tools cd apps/agent && bun run start # http://localhost:8941 -# 3. wire them together (any one key is enough) +# 3. wire them together (any one model key is enough) ANTHROPIC_API_KEY=sk-... bun run setup # 4. the editor @@ -93,24 +250,48 @@ cd ../.. && bun run dev:web # http://localhost:5173 Then ask for an edit: "Remove the silences", "Caption every video clip". Without the harness running, the editor still loads with a sample timeline; the assistant panel -says it is offline. +says it is offline. `setup` is idempotent, so rerun it after changing `agent.json` or adding a key. -## Stack +### Environment -- [TanStack Start](https://tanstack.com/start) + React 19, Vite, Tailwind CSS v4, shadcn/ui -- [TrueForge](https://trueforge.dev) agent harness, [MCP](https://modelcontextprotocol.io) tools -- Cloudflare Workers (via Wrangler), Bun + Turborepo monorepo +| Variable | Effect | +| --- | --- | +| `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` / `GEMINI_API_KEY` | Registers that provider with every model in the TrueForge catalog. | +| `NVIDIA_API_KEY` | Registers NVIDIA NIM as a custom OpenAI-compatible provider. | +| `OPENAI_COMPATIBLE_BASE_URL` + `_API_KEY` + `_MODELS` (+ `_NAME`) | Registers any other OpenAI-compatible endpoint: vLLM, Ollama, a gateway. | +| `DAYTONA_API_KEY` | Configures the harness sandbox and enables it on the agent. | +| `EDITAI_MODEL` | Pins the agent's model instead of picking the best configured one. | +| `EDITAI_CONNECTORS` | Extra MCP servers to attach, comma separated. Defaults to `exa`. | +| `_MCP_HEADER` | Credential for a header-auth connector, e.g. `BRIGHT_DATA_MCP_HEADER="Authorization: Bearer ..."`. | +| `TRUEFORGE_BASE_URL` | Defaults to `http://localhost:8790`. | +| `EDITAI_AGENT_PORT` | Defaults to `8941`. | + +At least one model key is required. With none set, `setup` stops and tells you. `setup` refuses to +POST keys over plaintext HTTP to anything but localhost. + +Copy [`apps/agent/.env.example`](apps/agent/.env.example) to `apps/agent/.env` to start from a +documented set. ## Layout ``` apps/ - web/ the editor: timeline, preview, assistant panel - agent/ MCP server exposing the timeline, plus the agent definition + web/ the editor: timeline, preview, assistant panel + agent/ MCP server exposing the timeline, plus the agent definition + src/tools.ts the 16 tools + src/project.ts the timeline model: split, trim, ripple delete, captions + scripts/setup.ts registers models, connectors, sandbox and the agent +packages/ + ffmpeg-sandbox/ containerised ffmpeg/ffprobe/python MCP server +skills/ + video-editing/ ffmpeg recipes and rules the agent loads on demand +docs/ + brightdata.md connector setup and verification +site/ the landing page (static, deployed to Vercel) +tests/ the Qodo verdict parser's fixtures and tests +.github/workflows/ CI, and the Qodo merge gate ``` -See [apps/agent/README.md](apps/agent/README.md) for the tool reference and timeline semantics. - ## Scripts | Command | What it does | @@ -118,73 +299,65 @@ See [apps/agent/README.md](apps/agent/README.md) for the tool reference and time | `bun run dev` | Run every app in dev mode | | `bun run dev:web` | Run only the web app | | `bun run build` | Build every app | +| `bun run test` | Run the Bun test suites | | `bun run deploy` | Build and deploy to Cloudflare | -## Sandboxed execution +## Tests -Generated code never runs on the host. Two layers cover the two kinds of work. +**31 tests**, all passing. -**Harness sandbox (Daytona).** `setup.ts` registers the provider when -`DAYTONA_API_KEY` is present, and the agent's `exec` tool then runs in a remote -sandbox. Verified end to end against the running harness: +| Suite | Count | Covers | +| --- | --- | --- | +| `apps/agent/test/project.test.ts` | 17 | Split and trim invariants, ripple-delete arithmetic across tracks, silence removal, transcript windowing, caption merging, undo, and the clip-id and trim-bound regressions Qodo surfaced. | +| `apps/web/.../transcript.test.ts` | 6 | Transcript windowing and caption rendering in the editor. | +| `tests/test_qodo_verdict.py` | 8 | The merge gate's verdict parser, against the real Qodo comment bodies that broke it. | +```bash +bun run test # the Bun suites (agent + web) +cd apps/agent && bun test # the timeline model +python3 -m pytest tests/ # the Qodo verdict parser ``` -sandbox.created sandbox_id: v1:daytona:default.e17058ee-... -exec python3 -c "print(sum(int(x)**2 for x in range(1,101)))" -tool.response {"success":true,"response":{"exitCode":0,"result":"338350\n"}} -``` -**Media sandbox (`packages/ffmpeg-sandbox`).** ffmpeg and ffprobe are not safe to -point at agent-supplied arguments on the host, so every invocation runs in a -throwaway container: `--network none`, capped memory and CPU, `--pids-limit 256`, -a non-root user, a single mounted workspace, and a hard timeout. `run_python` -there is annotated `destructiveHint: true`, so the harness shows the script and -waits for approval before it runs, because a script with the workspace mounted -read-write can delete the source media and the container is not a defence -against that. - -The split is deliberate: the harness sandbox is for computation the agent should -not estimate, and the media sandbox is for work that must reach the media files. - -## Qodo Code Review Evidence - -### Merging on the review - -Qodo posts its verdict as an issue comment. It publishes no check run, no commit -status and no approving review, so GitHub's own auto-merge has nothing to gate -on. `.github/workflows/qodo-automerge.yml` is that missing gate. - -It listens for edited comments as well as created ones, because Qodo posts a -placeholder and then edits the verdict into that same comment: a gate watching -only for new comments never sees a verdict at all. On an edited event -`comment.user` is still the bot even when a person did the editing, so the -sender is checked too. - -It fails closed in every direction, because the first draft did not and Qodo -said so. It reads only the structured counter chips, never the prose: Qodo -quotes findings and diff hunks verbatim, so the words "no issues found" appear -inside reviews that are *not* clean, and any substring test on the comment body -is forgeable by the pull request's own content. It binds the verdict to the -commit Qodo footers in the comment and refuses to merge when that is no longer -the head, since a review applies to one revision and `issue_comment` runs give a -job no link to the pull request head. It treats a failure to read check state as -an error rather than as an absence of failures. And it merges with -`--match-head-commit`, so a push racing the merge is rejected by GitHub instead -of slipping in. +## Code review evidence (Qodo) + +Every pull request here is reviewed by [Qodo](https://qodo.ai). **Eight findings across three +reviews: seven were real and are fixed, one was checked and rejected with a proof.** + +### The merge gate + +Qodo posts its verdict as an issue comment. It publishes no check run, no commit status and no +approving review, so GitHub's own auto-merge has nothing to gate on. +[`.github/workflows/qodo-automerge.yml`](.github/workflows/qodo-automerge.yml) is that missing gate. + +It listens for edited comments as well as created ones, because Qodo posts a placeholder and then +edits the verdict into that same comment: a gate watching only for new comments never sees a verdict +at all. On an edited event `comment.user` is still the bot even when a person did the editing, so +the sender is checked too. + +It fails closed in every direction, because the first draft did not and Qodo said so: + +- It reads only the **structured counter chips**, never the prose. Qodo quotes findings and diff + hunks verbatim, so the words "no issues found" appear inside reviews that are *not* clean, and any + substring test on the comment body is forgeable by the pull request's own content. +- It **binds the verdict to the commit** Qodo footers in the comment and refuses to merge when that + is no longer the head, since a review applies to one revision and `issue_comment` runs give a job + no link to the pull request head. +- It treats a **failure to read check state as an error**, not as an absence of failures. +- It merges with **`--match-head-commit`**, so a push racing the merge is rejected by GitHub instead + of slipping in. ### PR #3, first review -[PR #3](https://github.com/deonmenezes/edit-ai/pull/3) was reviewed with Qodo Merge, which raised -three issues. All three were real and all three are fixed in the PR: +[PR #3](https://github.com/deonmenezes/edit-ai/pull/3) raised three issues. All three were real and +all three are fixed: 1. **Duplicate clip ids after repeated ripple deletes** (`apps/agent/src/project.ts`). The - right-hand half of a split clip took the id `${c.id}r`, so a clip cut more than once produced - the same id twice. Reproduced on the sample project: `removeSilences` left three clips sharing - `c6r` and three sharing `c7r`, which breaks clip lookup, deletion and React keys. Ids are now - allocated from the set of ids in use. Covered by three regression tests. + right-hand half of a split clip took the id `${c.id}r`, so a clip cut more than once produced the + same id twice. Reproduced on the sample project: `removeSilences` left three clips sharing `c6r` + and three sharing `c7r`, which breaks clip lookup, deletion and React keys. Ids are now allocated + from the set of ids in use. Covered by three regression tests. 2. **Export announced before the file existed.** `exportProject` committed the record, which - notifies SSE subscribers synchronously, and only then wrote the file. The write now happens - first. + notifies SSE subscribers synchronously, and only then wrote the file. The write now happens first. 3. **The session-restore effect leaked its stream on unmount**, calling `setState` on a gone component and leaving the connection open. Its cleanup now aborts the controller. @@ -201,10 +374,10 @@ rejected: - **Real:** `setup.ts` POSTs API keys to the harness, so it now refuses to do that over plaintext HTTP to anything but localhost. - **False positive:** the reviewer called the source-media bound in `trimClip` - (`end - (c.start - c.sourceOffset)`) wrong and predicted a clip could be extended to 28s instead - of 18s. The expression expands to `c.sourceOffset + (end - c.start)`, which is the correct source - time, and the code accepts exactly up to the limit and rejects one frame past it. Two tests now - pin that boundary so the correct form is not "fixed" into a broken one later. + (`end - (c.start - c.sourceOffset)`) wrong and predicted a clip could be extended to 28s instead of + 18s. The expression expands to `c.sourceOffset + (end - c.start)`, which is the correct source + time, and the code accepts exactly up to the limit and rejects one frame past it. Two tests now pin + that boundary so the correct form is not "fixed" into a broken one later. ### PR #2 review @@ -216,14 +389,33 @@ rejected: workspace is exactly what it is meant to reach. The tool is now registered with `destructiveHint: true`, so the harness stops and shows the script for approval first, the same gate the timeline's `delete_clip` and `ripple_delete` use. -- **The path guard rejected valid code.** Scanning a Python source string for `..` or a URL - rejected correct programs without adding a boundary. Path validation now applies only to - path-like arguments; for code payloads the container is the boundary. +- **The path guard rejected valid code.** Scanning a Python source string for `..` or a URL rejected + correct programs without adding a boundary. Path validation now applies only to path-like + arguments; for code payloads the container is the boundary. + +## Stack + +- [TanStack Start](https://tanstack.com/start) + React 19, Vite, Tailwind CSS v4, shadcn/ui +- [TrueForge](https://trueforge.dev) agent harness, [MCP](https://modelcontextprotocol.io) tools +- Cloudflare Workers (via Wrangler), Bun + Turborepo monorepo +- Landing page: hand-written static HTML in `site/`, deployed to Vercel + +## Known limits + +Worth stating plainly, because the demo does not make them obvious: +- `export_project` writes a JSON description of the render rather than encoding video. Wiring it to + ffmpeg is the obvious next step and does not change the agent-facing contract. +- Transcripts, silences and tempo come from media metadata in the sample project rather than from + running ASR and onset detection over real files. +- Motion graphics have no dedicated timeline tool yet. Today they go through the ffmpeg sandbox or + an attached MCP server. ## Contributing -Issues and pull requests are welcome. See [CONTRIBUTING.md](.github/CONTRIBUTING.md) for setup and guidelines, and open an issue first for anything larger than a bug fix. +Issues and pull requests are welcome. See [CONTRIBUTING.md](.github/CONTRIBUTING.md) for setup and +guidelines, and open an issue first for anything larger than a bug fix. Every PR is reviewed by Qodo +before it can merge. ## License diff --git a/site/.gitignore b/site/.gitignore new file mode 100644 index 0000000..e985853 --- /dev/null +++ b/site/.gitignore @@ -0,0 +1 @@ +.vercel diff --git a/site/README.md b/site/README.md new file mode 100644 index 0000000..e4d7e00 --- /dev/null +++ b/site/README.md @@ -0,0 +1,37 @@ +# site + +The EditAI landing page: + +One hand-written `index.html` with inline CSS and JS. No build step, no dependencies, no +framework. Open the file to work on it. + +```bash +python3 -m http.server 4477 --directory site # http://localhost:4477 +``` + +## The hero timeline + +The timeline in the hero is not a screenshot. It renders the same sample project the agent server +ships with (`apps/agent/src/project.ts`: `intro.mp4`, `b-roll.mp4`, `talking-head.mp4`, the +voiceover and its three silences) and performs a real ripple delete on it: `mapTime()` in the page +script applies the same rule the server does, so 24.0s becomes 21.1s and clips straddling a silence +get shorter rather than merely shifting. + +**If the sample project changes, change `SILENCES` and `LANES` in `index.html` to match.** The page +claims those are real numbers, so they have to stay real. + +## Colours + +Every colour is lifted from the editor rather than invented: the well, panel and foreground greys +from `apps/web/src/styles.css`, the violet `#7c5cff` from the favicon, and the three clip colours +(`#3aa39b` video, `#e0a63b` text, `#5fae63` audio) that the timeline paints tracks with. + +## Deploying + +```bash +cd site +vercel deploy --prod --scope deonmenezes-projects +``` + +Production aliases: `editai-agent.vercel.app` (canonical), `edit-ai-video.vercel.app`, +`edit-ai-lemon.vercel.app`. diff --git a/site/index.html b/site/index.html new file mode 100644 index 0000000..42e10f6 --- /dev/null +++ b/site/index.html @@ -0,0 +1,950 @@ + + + + + +EditAI: talk to your timeline + + + + + + + + + + + + + + + + + + +
+ + + EditAI + + 00:00:00:00 + + + + + GitHub + +
+ + + + +
+
+ + +
+
+ 00:00:00 + Cold open + +
+ +

Talkto yourtimeline

+ +

+ EditAI is an AI harness for video editing. Ask for the cut, the motion + graphic, the filter or the noise gate. An agent makes the edit on your real timeline + through 16 MCP tools, and stops for your approval before anything destructive. +

+ + +
+
+ + + ⌘⏎ +
+ +
+
V1
+
A1
+
T1
+
+ +
+ Waiting for an instruction + + + 24.0s +
+
+ +

+ Real behaviour, real numbers: find_silences then + ripple_delete across every track, 24.0s down to 21.1s, which is + exactly the 2.9 seconds of dead air. Nothing is cut until you approve it, and every edit undoes. +

+
+ + +
+
+ 00:00:14 + The handoff + +
+

Connect the MCP.
The agent takes the room.

+

+ Point your client at EditAI's MCP server and an agent comes up inside a sandbox on + TrueForge, TrueFoundry's open-source agent harness. From there it drives the + edit at whichever layer the job needs. +

+ +
+
+ Layer one +

Your editing UI

+

Bring your own front end. The timeline is exposed as tools, not as a prompt describing a + timeline, so any MCP client can drive it.

+
any MCP client
+
+
+ Layer two +

Our editor

+

A full timeline, preview and assistant panel. The agent's edits stream in over SSE, so + the tracks redraw as the work happens.

+
apps/web · TanStack Start
+
+
+ Layer three +

The encode itself

+

When a job needs the real file, the agent writes the ffmpeg filter graph and runs it in a + locked-down container.

+
ffmpeg · ffprobe · python
+
+
+ +
# 1 · the harness
+npx @truefoundry/trueforge@latest          → localhost:8790
+
+# 2 · the timeline tools
+cd apps/agent && bun run start            → localhost:8941
+
+# 3 · wire them together (any one key is enough)
+ANTHROPIC_API_KEY=sk-... bun run setup
+
+# 4 · the editor
+bun run dev:web                           → localhost:5173
+
+ + +
+
+ 00:00:36 + Say it + +
+

Ask in a sentence.
Watch the tracks move.

+

+ Every ask below runs real tools against real state. Clip ids, source offsets and frame + boundaries are the agent's problem, not yours. +

+ +
+
+

▸Remove the silences

+

Finds every silent range on the voice track, tells you what will go, and + ripple-deletes it across all tracks so the gaps close.

+

find_silences → remove_silences

+
+
+

▸Caption every clip

+

Fans out one sub-agent per clip to transcribe in parallel, merges the + results, and lays timed captions on their own track.

+

transcribe_clip × N → add_captions

+
+
+

▸Cut this to the beat

+

Reads the tempo off the music track and splits on the beat grid, both + halves still frame-accurate.

+

detect_beats → split_clip

+
+
+

▸Duck the music under the voiceover

+

Sets clip volume where the voice track is speaking and puts it back + where it is not.

+

get_project → set_volume

+
+
+

▸Grade it warmer and add grain

+

Writes the ffmpeg filter graph, runs it in the media sandbox, then probes + the output to confirm it is what you asked for.

+

probe_media → run_ffmpeg

+
+
+

▸Kill the room tone

+

Denoises the voice track in the sandbox and leaves the original file + untouched next to it.

+

run_ffmpeg · afftdn

+
+
+ +

+ Sixteen timeline tools in all. Five read, seven write and undo, three destructive, one gated + export. The destructive ones carry MCP's destructiveHint, which is + what makes the harness stop and ask you first. +

+
+ + +
+
+ 00:00:58 + Pull it in + +
+

Need a logo?
It goes and gets one.

+

+ The agent reaches the live web through Bright Data, so a logo, a product shot + or a reference frame is one sentence away from being on your timeline. The connector attaches + deferred: it costs no context until a task actually needs it. +

+
+
+ Find it +

search_engine

+

Live search results, not a training-set memory of what a brand looked like two years ago.

+
+
+ Take it +

scrape_as_markdown

+

Pulls the page down clean, so the agent can lift the asset and the copy around it.

+
+
+
+ + +
+
+ 00:01:16 + Add a tool mid-call + +
+

A new skill is a
line in a config.

+ +
+
+ On the call + Bro, can you add an MCP for motion graphics? I want to use it in this video. +
+
+ You + Sent to your agent. +
+
+ +

+ Every capability here is an MCP server, so adding one is a name in a list and a restart, not a + release. The agent picks up the new tools on its next turn and starts using them in the same + conversation. +

+ +
EDITAI_CONNECTORS=exa,bright-data,motion-graphics bun run setup
+
+✓ motion-graphics   attached   read-only, deferred
+✓ agent updated     14 connectors → tools live on the next turn
+ +

+ Catalog servers attach by name, OAuth ones authorise in chat, and anything you host yourself + attaches the same way. Read-only and deferred by default, so a connector you rarely use does + not crowd the agent's context. +

+
+ + +
+
+ 00:01:34 + Where it runs + +
+

Nothing the agent
writes runs on your box.

+

+ Two sandboxes, because the two kinds of work need different boundaries. +

+
+
+ Daytona +

Harness sandbox

+

For the computation the agent should not estimate: arithmetic, parsing, anything it + would otherwise guess at. Runs remotely, registered the moment the key is present.

+
exit 0 · 338350
+
+
+ Container +

Media sandbox

+

Every ffmpeg call runs in a throwaway container: no network, capped memory and CPU, 256 + pids, non-root, one mounted workspace, hard timeout. Python there asks for approval first, + because a script with the workspace mounted can delete your source media.

+
--network none · --pids-limit 256
+
+
+
+ + +
+
+ 00:01:52 + Reviewed by Qodo + +
+

Every line here was
reviewed by Qodo.

+

+ Not a badge. Eight findings across three reviews, each one either fixed or answered with a + proof, and a merge gate that reads Qodo's structured verdict rather than its prose. +

+ +
+ 8Findings raised + 7Real, fixed + 1Checked, rejected + 5Regression tests added +
+ +
+
+ PR #3 + Duplicate clip ids after repeated ripple deletes. A clip + cut twice produced the same id twice, breaking lookup, deletion and React keys. The test + suite had missed it because the existing test asserted the buggy id as correct. + FIXED +
+
+ PR #3 + Export announced before the file existed. The record was + committed, notifying subscribers, and only then was the file written. + FIXED +
+
+ PR #3 + Session restore leaked its stream on unmount. It set state + on a gone component and left the connection open. + FIXED +
+
+ PR #3 + Setup posted API keys over plaintext HTTP. It now refuses + to do that to anything but localhost. + FIXED +
+
+ PR #3 + The trim bound was called wrong, and was not. The source + expression expands to the correct source time; the code accepts exactly up to the limit + and rejects one frame past it. Two tests now pin that boundary. + REJECTED +
+
+ PR #2 + Sandboxed Python could delete the source media. The + argument guard could not catch it, and the container is no defence, since the workspace is + exactly what it is meant to reach. The tool now asks for approval first. + FIXED +
+
+ +

+ Qodo publishes no check run and no approving review, so GitHub's own auto-merge has nothing to + gate on. The workflow in this repo is that missing gate: it reads only the structured counter + chips, never the prose, because Qodo quotes findings verbatim and a pull request can put the + words "no issues found" into its own diff. It binds the verdict to the reviewed commit and + merges with --match-head-commit, so a push racing the merge is + rejected rather than slipped in. +

+
+ + +
+

And the video you
just watched?

+

It cut itself.

+ +
+ + + +
+
+ + + + diff --git a/site/vercel.json b/site/vercel.json new file mode 100644 index 0000000..92bf592 --- /dev/null +++ b/site/vercel.json @@ -0,0 +1,13 @@ +{ + "$schema": "https://openapi.vercel.sh/vercel.json", + "cleanUrls": true, + "headers": [ + { + "source": "/(.*)", + "headers": [ + { "key": "X-Content-Type-Options", "value": "nosniff" }, + { "key": "Referrer-Policy", "value": "strict-origin-when-cross-origin" } + ] + } + ] +}