Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ Spaces, specialist agents, and conversations that move between text, calls, and

---

![Live specialist conversation](docs/demos/specialist-chat.gif)
![Ask Scout to browse, review its draft, and save a Space page](docs/demos/chat-to-space.gif)

_A live conversation with Scout, with the updated Dot avatars and separate Dots, Spaces, and recent chats. [Watch the MP4](docs/demos/specialist-chat.mp4)._
_Ask → browse → approve → save. A live computer view and a human review card appear right in chat, then the approved draft becomes an editable Space page. [Watch the 23-second video](docs/demos/chat-to-space.mp4). Enlarged for readability; idle time is trimmed and playback is accelerated._

## Overview

Expand Down Expand Up @@ -48,6 +48,10 @@ See [Computer setup](docs/COMPUTERS.md) to build the pinned services and connect

_Ask Scout to open a website, summarize it, save notes, and verify the file. Every computer action in this demo is requested through chat; CopilotKit tool renderers show the live browser, saved file, and terminal output inline. [Watch the MP4](docs/demos/computer-chat.mp4)._

### Review before saving

Ask a Dot to show a draft before saving it. A CopilotKit human-in-the-loop card pauses the conversation for **Approve & save** or **Decline**. Approval creates the page in an authorized Space and returns a link; retries recover the same saved page. The agent continues after your decision.

### Text and calls

A continuous conversation keeps the Dot's avatar and status above the messages, with text and call controls close at hand. Work updates, source links, and call receipts appear in the timeline; a side panel shows results or the agent's computer.
Expand Down
26 changes: 19 additions & 7 deletions docs/demos/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,37 @@

Recorded from the local OpenDots app on September 29, 2026. The recordings use a dedicated `opendots` Intelligence project, a live `gpt-5.4-mini` model, and example content in a separate Launch studio Space. No customer data or credentials are shown.

| Recording | Flow | Duration |
| -------------------------------------------- | --------------------------------------------------------------------------------------------- | ---------- |
| [Specialist chat](specialist-chat.mp4) | Continue a launch-planning conversation with Scout and receive a live follow-up response | 16 seconds |
| [Spaces and page chat](spaces-page-chat.mp4) | Open a Space and page, ask about the saved brief, then open the same conversation under Scout | 24 seconds |
| Recording | Flow | Duration | Framing |
| -------------------------------------------- | ----------------------------------------------------------------------------------- | ---------- | ------- |
| [Chat to Space](chat-to-space.mp4) | Ask Scout to browse, review the proposed draft, approve it, and open the saved page | 23 seconds | 150% |
| [Specialist chat](specialist-chat.mp4) | Continue launch planning with Scout and receive a live follow-up response | 13 seconds | 150% |
| [Spaces and page chat](spaces-page-chat.mp4) | Open a Space and page, ask about the saved brief, then continue under Scout | 19 seconds | 135% |
| [Computer chat](computer-chat.mp4) | Browse a website, save notes, and verify the file through chat | 34 seconds | 150% |

The recordings show the updated plush avatars and separate Dots, Spaces, and recent-chat navigation. Space-access settings preserve existing grants. The computer demo uses Scout with browser, file, and shell access enabled for its own container.

The README embeds compact GIF versions. MP4 versions are included for playback and reuse. These are screen-capture walkthroughs, with pauses between recording segments omitted; no assistant responses are fabricated or replaced. The captures are sampled at four frames per second, so they are intended to show workflows rather than animation performance. They have no audio.
The README embeds compact GIF versions. MP4 versions are included for playback and reuse. These are screen-capture walkthroughs. The hero omits four seconds of idle footage and plays at 1.2×; the other clips play at 1.25×. Pauses between capture segments are also omitted. These edits show the workflow, not a model-latency benchmark. No assistant responses or tool results are fabricated or replaced. Each video is cropped around its active components at the listed magnification relative to the original 1280-pixel-wide capture; GIFs use the same framing. The captures are sampled at four frames per second, so they are intended to show workflows rather than animation performance. They have no audio.

## Chat-to-Space hero

[Chat to Space](chat-to-space.mp4) follows one live natural-language request:

> Visit https://www.copilotkit.ai in your computer. Draft “Launch in three steps”: one goal and three milestones, under 80 words. Show it for my approval before saving in Launch studio.
Scout opens the website in its existing OpenBot computer. CopilotKit `useRenderTool` displays the browser inline, and `useHumanInTheLoop` presents the draft with **Approve & save** and **Decline**. The recording shows approval, the saved-page receipt, and the finished Space page. The page also survived a reload after recording.

Approval uses an owner-authenticated route, checks the Dot's current Space access, and deduplicates saves by thread and tool call. Reconnecting recovers an existing approval before offering a new decision. Slack does not receive this web-only review tool.

## Chat-driven computer demo

[Computer chat](computer-chat.mp4) (42 seconds) uses the existing CopilotKit `useAgent` / `useCopilotKit` chat and server tools. The OpenBot computer is provisioned before recording. Only these two natural-language messages drive the workflow:
[Computer chat](computer-chat.mp4) (34 seconds) uses the existing CopilotKit `useAgent` / `useCopilotKit` chat and server tools. The OpenBot computer is provisioned before recording. Only these two natural-language messages drive the workflow:

1. “Open https://www.copilotkit.ai in your computer and tell me what it offers in two short bullets.”
2. “Save those notes as copilotkit-notes.md on your computer. Use your terminal to verify the file, then tell me where it is and how many words it contains.”

Scout navigates, summarizes the page, writes the file, and executes a terminal check. Its real reply reports `/workspace/copilotkit-notes.md` and 41 words; an independent container command confirmed both. No manual browser navigation, file editing, or terminal command entry appears in this recording. CopilotKit `useRenderTool` and `CopilotChatToolCallsView` render the computer directly inside the conversation: a live browser card, a file receipt, and terminal output. The side panel stays closed. Tool results and responses are not scripted.

The final recording was captured after correcting host-to-supervisor networking and documenting snapshot recovery following a computer restart. Earlier failed takes are not part of the clip. Between the two chat requests, recording pauses were omitted; response generation within the captured segments is shown as recorded.
The final recording was captured after correcting host-to-supervisor networking and documenting snapshot recovery following a computer restart. Earlier failed takes are not part of the clip. Between the two chat requests, recording pauses were omitted; the revised clip accelerates the captured footage to 1.25× without changing its sequence.

## What was verified

Expand Down
Binary file added docs/demos/chat-to-space.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/demos/chat-to-space.mp4
Binary file not shown.
Binary file modified docs/demos/computer-chat.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/demos/computer-chat.mp4
Binary file not shown.
Binary file modified docs/demos/spaces-page-chat.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/demos/spaces-page-chat.mp4
Binary file not shown.
Binary file modified docs/demos/specialist-chat.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/demos/specialist-chat.mp4
Binary file not shown.
20 changes: 18 additions & 2 deletions src/client/Chat.tsx
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
import { PageReviewCard } from './PageReviewCard';
import { pageReviewSchema, pageReviewTool } from '../shared/page-review';
import { contextualMessage, type PageContext } from './page-context';
import { api } from './api';
import type { Page } from '../server/pages';
import { useEffect, useRef, useState } from 'react';
import {
CopilotChatToolCallsView,
useRenderTool,
useHumanInTheLoop,
useAgent,
useCopilotKit,
} from '@copilotkit/react-core/v2';
Expand Down Expand Up @@ -163,6 +166,17 @@ export function Chat({
useEffect(() => {
if (paused && voice.status !== 'idle') void voice.end();
}, [paused]);
useHumanInTheLoop(
{
name: pageReviewTool.name,
description: pageReviewTool.description,
parameters: pageReviewSchema,
render: (props) => (
<PageReviewCard {...props} threadId={thread.id} onSaved={onSaved} />
),
},
[thread.id, onSaved],
);
const computerCalls = agent.messages.flatMap((message) =>
message.role === 'assistant' ? (message.toolCalls ?? []) : [],
);
Expand Down Expand Up @@ -200,8 +214,10 @@ export function Chat({
['user', 'assistant'].includes(message.role) &&
((typeof message.content === 'string' && message.content.trim()) ||
(message.role === 'assistant' &&
message.toolCalls?.some((call) =>
call.function.name.startsWith('computer_'),
message.toolCalls?.some(
(call) =>
call.function.name.startsWith('computer_') ||
call.function.name === pageReviewTool.name,
))),
);
return (
Expand Down
200 changes: 200 additions & 0 deletions src/client/PageReviewCard.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,200 @@
import { useEffect, useRef, useState } from 'react';
import { Check, FileText, ArrowUpRight } from 'lucide-react';
import ReactMarkdown from 'react-markdown';
import { pageReviewSchema } from '../shared/page-review';
import { decidePageReview, restorePageReview } from './page-review-decision';
import { computerToolResult } from './ComputerToolCard';
import { openPageLink } from './page-navigation';
import type { Page } from '../server/pages';
export function PageReviewCard({
args,
status,
result,
respond,
threadId,
toolCallId,
onSaved,
}: {
args: unknown;
status: string;
result?: unknown;
respond?: (result: unknown) => Promise<void>;
threadId: string;
toolCallId: string;
onSaved: () => void;
}) {
const draft = pageReviewSchema.safeParse(args);
const outcome = computerToolResult(result);
const [savedPage, setSavedPage] = useState<Page>();
const [error, setError] = useState('');
const [busy, setBusy] = useState(false);
const [receiptReady, setReceiptReady] = useState(false);
const [restoreAttempt, setRestoreAttempt] = useState(0);
const pending = useRef(false);
const finished = status === 'complete';
const recordedApproval = outcome.approved === true;
const saved = !!savedPage || recordedApproval;
const pageId =
savedPage?.id ?? (typeof outcome.pageId === 'string' ? outcome.pageId : '');
const spaceId =
savedPage?.spaceId ??
(typeof outcome.spaceId === 'string' ? outcome.spaceId : '');
useEffect(() => {
if (recordedApproval) return;
let active = true;
setReceiptReady(false);
setError('');
void restorePageReview(threadId, toolCallId)
.then((page) => {
if (!active) return;
setSavedPage(page ?? undefined);
setReceiptReady(true);
})
.catch((cause) => {
if (active)
setError(
cause instanceof Error
? cause.message
: 'Could not restore this review.',
);
});
return () => {
active = false;
};
}, [threadId, toolCallId, recordedApproval, restoreAttempt]);
const decide = async (approved: boolean) => {
if (!respond || !receiptReady || pending.current) return;
pending.current = true;
setBusy(true);
setError('');
try {
const page = await decidePageReview(threadId, toolCallId, args, approved);
if (!page) {
await respond({
approved: false,
message: 'The owner declined this draft. Do not save it.',
});
return;
}
setSavedPage(page);
onSaved();
await respond({
approved: true,
pageId: page.id,
spaceId: page.spaceId,
url: `/#/spaces/${page.spaceId}/pages/${page.id}`,
});
} catch (cause) {
setReceiptReady(false);
setError(
cause instanceof Error
? cause.message
: 'Could not save the approved draft.',
);
} finally {
pending.current = false;
setBusy(false);
}
};
return (
<section className="page-review-card" aria-label="Review page draft">
<header>
<FileText size={17} />
<strong>
{saved
? 'Saved to your Space'
: !receiptReady
? 'Checking saved review…'
: finished
? 'Review ended'
: 'Ready for your review'}
</strong>
<span>
{saved
? 'Approved'
: !receiptReady
? 'Checking'
: finished
? 'Not saved'
: 'You decide'}
</span>
</header>
<div className="page-review-body">
<h3>{draft.success ? draft.data.title : 'Preparing your draft…'}</h3>
{draft.success && (
<ReactMarkdown
components={{
img: ({ alt }) => <span>{alt}</span>,
a: ({ href, children }) => (
<a href={href} target="_blank" rel="noreferrer">
{children}
</a>
),
}}
>
{draft.data.content}
</ReactMarkdown>
)}
</div>
{error && <p role="alert">{error}</p>}
{!receiptReady && error && (
<button
type="button"
onClick={() => setRestoreAttempt((attempt) => attempt + 1)}
>
Retry review
</button>
)}
<footer>
{saved && pageId && spaceId && (
<button
type="button"
className="review-primary"
onClick={() =>
openPageLink(
`/#/spaces/${encodeURIComponent(spaceId)}/pages/${encodeURIComponent(pageId)}`,
)
}
>
Open page <ArrowUpRight size={15} />
</button>
)}
{!finished && respond && receiptReady && (
<>
<button
type="button"
disabled={busy || (!saved && !draft.success)}
className="review-primary"
onClick={() => void decide(true)}
>
<Check size={15} />
{busy
? 'Saving…'
: saved
? 'Continue conversation'
: 'Approve & save'}
</button>
{!saved && (
<button
type="button"
disabled={busy}
onClick={() => void decide(false)}
>
Decline
</button>
)}
</>
)}
{!saved && (
<small>
{!receiptReady
? 'Checking whether this draft was already saved.'
: finished
? 'No page was saved.'
: 'Nothing is saved until you approve.'}
</small>
)}
</footer>
</section>
);
}
26 changes: 26 additions & 0 deletions src/client/page-review-decision.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import { pageReviewSchema } from '../shared/page-review';
import type { Page } from '../server/pages';
import { api } from './api';

const reviewPath = (threadId: string) =>
`/conversations/${encodeURIComponent(threadId)}/reviewed-page`;

export function restorePageReview(threadId: string, toolCallId: string) {
return api<Page | null>(
`${reviewPath(threadId)}/${encodeURIComponent(toolCallId)}`,
);
}

export async function decidePageReview(
threadId: string,
toolCallId: string,
args: unknown,
approved: boolean,
): Promise<Page | null> {
// A previous save may have committed even if its response never arrived.
const previous = await restorePageReview(threadId, toolCallId);
if (previous) return previous;
if (!approved) return null;
const draft = pageReviewSchema.parse(args);
return api<Page>(reviewPath(threadId), 'POST', { ...draft, toolCallId });
}
Loading
Loading