-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathuser-guide.html
More file actions
327 lines (314 loc) · 15.2 KB
/
Copy pathuser-guide.html
File metadata and controls
327 lines (314 loc) · 15.2 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>BunnyRunner User Guide</title>
<style>
:root {
--bg: #0d1117;
--fg: #c9d1d9;
--accent: #FF6B9D;
--accent2: #4ADE80;
--muted: #8b949e;
--card: #161b22;
--border: #30363d;
--warn: #FDE047;
}
* { box-sizing: border-box; }
body {
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif;
background: var(--bg);
color: var(--fg);
line-height: 1.7;
margin: 0;
padding: 0;
}
header {
background: linear-gradient(135deg, #1f2937 0%, #111827 100%);
border-bottom: 1px solid var(--border);
padding: 2rem 1rem;
text-align: center;
}
header h1 { margin: 0; color: var(--accent); }
header p { color: var(--muted); margin: 0.5rem 0 0; }
.container {
max-width: 800px;
margin: 0 auto;
padding: 2rem 1rem;
}
h2 {
color: var(--accent);
border-bottom: 1px solid var(--border);
padding-bottom: 0.5rem;
margin-top: 2.5rem;
}
h3 { color: var(--accent2); margin-top: 1.5rem; }
.scenario {
background: var(--card);
border: 1px solid var(--border);
border-radius: 8px;
padding: 1.5rem;
margin: 1rem 0;
}
.scenario-header {
display: flex;
align-items: center;
gap: 0.75rem;
margin-bottom: 1rem;
}
.scenario-header .icon { font-size: 1.5rem; }
.scenario-header h3 { margin: 0; }
.chat-bubble {
background: rgba(255, 107, 157, 0.08);
border-radius: 8px;
padding: 0.75rem 1rem;
margin: 0.5rem 0;
font-style: italic;
}
.chat-bubble.user {
background: rgba(74, 222, 128, 0.08);
border-left: 3px solid var(--accent2);
}
.chat-bubble.ai {
background: rgba(255, 107, 157, 0.08);
border-left: 3px solid var(--accent);
}
.chat-bubble .label {
font-size: 0.75rem;
font-weight: bold;
text-transform: uppercase;
color: var(--muted);
margin-bottom: 0.25rem;
font-style: normal;
}
.action-list {
list-style: none;
padding: 0;
margin: 0.5rem 0;
}
.action-list li {
padding: 0.4rem 0;
padding-left: 1.5rem;
position: relative;
}
.action-list li::before {
content: "→";
position: absolute;
left: 0;
color: var(--accent2);
}
code {
background: var(--card);
padding: 0.15rem 0.4rem;
border-radius: 4px;
font-family: "SF Mono", Monaco, monospace;
font-size: 0.9em;
}
pre {
background: var(--card);
border: 1px solid var(--border);
border-radius: 6px;
padding: 1rem;
overflow-x: auto;
font-family: "SF Mono", Monaco, monospace;
font-size: 0.85rem;
line-height: 1.5;
}
.tip {
background: rgba(255, 107, 157, 0.1);
border-left: 3px solid var(--accent);
padding: 1rem 1.25rem;
margin: 1rem 0;
border-radius: 0 6px 6px 0;
}
.warn {
background: rgba(253, 224, 71, 0.1);
border-left: 3px solid var(--warn);
padding: 1rem 1.25rem;
margin: 1rem 0;
border-radius: 0 6px 6px 0;
color: var(--fg);
}
a { color: var(--accent); text-decoration: none; }
a:hover { text-decoration: underline; }
footer {
text-align: center;
color: var(--muted);
padding: 3rem 1rem;
border-top: 1px solid var(--border);
margin-top: 2rem;
}
</style>
</head>
<body>
<header>
<h1>BunnyRunner User Guide</h1>
<p>How to give your AI agent tasks and control your Android device</p>
</header>
<div class="container">
<h2>Getting Started</h2>
<p>Once you've completed the <a href="installation-guide.html">installation</a>, open your browser and navigate to your phone's IP address on port <code>8080</code>. You'll see the BunnyRunner web portal.</p>
<div class="tip">
<strong>Tip:</strong> Bookmark the URL for quick access. It will look something like <code>http://192.168.1.42:8080</code>.
</div>
<h2>How the Agent Controls Your Device</h2>
<p>BunnyRunner uses a <strong>ReAct (Reasoning + Acting) loop</strong> that runs entirely inside the app:</p>
<ol>
<li>The app builds a <strong>system prompt</strong> with your phone's real specs (model, screen size, Android version, manufacturer tips)</li>
<li>The AI <strong>reasons</strong> and outputs an action in text format, e.g. <code>ACTION: tap(500, 800)</code></li>
<li>The app <strong>parses</strong> the action, executes it via AccessibilityService, and <strong>takes a screenshot</strong></li>
<li>The screenshot result is fed back as an <strong>OBSERVATION:</strong> in the next prompt</li>
<li>The AI <strong>repeats</strong> until it outputs <code>DONE: summary</code> or hits the 90-action limit</li>
</ol>
<div class="warn">
<strong>Important:</strong> The AI can only interact with apps that are visible on screen. It cannot access data from apps that are running in the background unless they display it when opened.
</div>
<h2>Use Case Scenarios</h2>
<div class="scenario">
<div class="scenario-header">
<span class="icon">📧</span>
<h3>Open Outlook and Check Emails</h3>
</div>
<div class="chat-bubble user">
<div class="label">You</div>
"Open Outlook and check my emails. Show me any unread messages."
</div>
<div class="chat-bubble ai">
<div class="label">Agent Actions</div>
<ul class="action-list">
<li>Screenshots the current screen</li>
<li>Outputs <code>ACTION: launch_app("com.microsoft.office.outlook")</code></li>
<li>Waits, then screenshots again to see the Outlook UI</li>
<li>Outputs <code>ACTION: tap(540, 420)</code> to open the Inbox if not already visible</li>
<li>Screenshots to read the email list</li>
<li>Outputs <code>DONE: You have 3 unread emails: 1. 'Meeting tomorrow at 3pm', 2. 'Invoice #2042', 3. 'Your Amazon order has shipped'</code></li>
</ul>
</div>
<div class="tip">
<strong>Tip:</strong> If Outlook is behind a lock screen or requires a PIN, the agent will report that it needs your help to unlock the app.
</div>
</div>
<div class="scenario">
<div class="scenario-header">
<span class="icon">📅</span>
<h3>Check Your Calendar for Today</h3>
</div>
<div class="chat-bubble user">
<div class="label">You</div>
"What's on my calendar for today?"
</div>
<div class="chat-bubble ai">
<div class="label">Agent Actions</div>
<ul class="action-list">
<li>Outputs <code>ACTION: launch_app("com.samsung.android.calendar")</code> or <code>ACTION: launch_app("com.google.android.calendar")</code></li>
<li>Navigates to today's view if not already showing</li>
<li>Screenshots to read the events</li>
<li>Outputs <code>DONE: Today you have: 9:00 AM — Standup, 11:30 AM — Design Review, 2:00 PM — 1:1 with Sarah, 4:00 PM — Sprint Planning</code></li>
</ul>
</div>
</div>
<div class="scenario">
<div class="scenario-header">
<span class="icon">💬</span>
<h3>Check Teams Messages and Respond</h3>
</div>
<div class="chat-bubble user">
<div class="label">You</div>
"Check Teams for any new messages and draft a reply to the latest one."
</div>
<div class="chat-bubble ai">
<div class="label">Agent Actions</div>
<ul class="action-list">
<li>Outputs <code>ACTION: launch_app("com.microsoft.teams")</code></li>
<li>Navigates to the Chat tab</li>
<li>Screenshots to read the conversation list</li>
<li>Opens the most recent chat</li>
<li>Reads the latest message</li>
<li>Outputs <code>DONE: Latest message from Alex: 'Can you review the PR by EOD?' I can draft a response: 'Thanks for the heads up, I'll review it and get back to you shortly.' Let me know if you'd like me to type it.</code></li>
</ul>
</div>
<div class="tip">
<strong>Tip:</strong> The agent will ask for your confirmation before sending any message. It types the text into the input field but does not press send unless explicitly told to.
</div>
</div>
<div class="scenario">
<div class="scenario-header">
<span class="icon">🔍</span>
<h3>Open Chrome and Search for Something</h3>
</div>
<div class="chat-bubble user">
<div class="label">You</div>
"Open Chrome and search for 'weather in Paris tomorrow'."
</div>
<div class="chat-bubble ai">
<div class="label">Agent Actions</div>
<ul class="action-list">
<li>Outputs <code>ACTION: launch_app("com.android.chrome")</code></li>
<li>Outputs <code>ACTION: tap(540, 180)</code> to tap the address bar</li>
<li>Outputs <code>ACTION: input_text("weather in Paris tomorrow")</code></li>
<li>Outputs <code>ACTION: press_key("ENTER")</code></li>
<li>Screenshots to show you the search results</li>
<li>Outputs <code>DONE: Chrome is showing search results for 'weather in Paris tomorrow'. The forecast shows partly cloudy with a high of 22°C.</code></li>
</ul>
</div>
</div>
<div class="scenario">
<div class="scenario-header">
<span class="icon">🔔</span>
<h3>Check Notifications</h3>
</div>
<div class="chat-bubble user">
<div class="label">You</div>
"Pull down my notifications and tell me what I missed."
</div>
<div class="chat-bubble ai">
<div class="label">Agent Actions</div>
<ul class="action-list">
<li>Outputs <code>ACTION: press_key("NOTIFICATIONS")</code> to open the notification shade</li>
<li>Screenshots to read the notifications</li>
<li>Outputs <code>DONE: You have 5 notifications: 2 WhatsApp messages, 1 Gmail, 1 Calendar reminder, and a system update available.</code></li>
</ul>
</div>
</div>
<h2>Available Actions</h2>
<p>The agent outputs actions in plain text that the app parses and executes:</p>
<table style="width:100%; border-collapse: collapse; margin-top:1rem;">
<tr style="border-bottom: 1px solid var(--border);">
<th style="text-align:left; padding:0.5rem;">Action</th>
<th style="text-align:left; padding:0.5rem;">Description</th>
<th style="text-align:left; padding:0.5rem;">Example Output</th>
</tr>
<tr><td style="padding:0.5rem;"><code>screenshot</code></td><td style="padding:0.5rem;">Capture current screen</td><td style="padding:0.5rem;"><code>ACTION: screenshot()</code></td></tr>
<tr><td style="padding:0.5rem;"><code>tap</code></td><td style="padding:0.5rem;">Tap at (x, y) coordinates</td><td style="padding:0.5rem;"><code>ACTION: tap(500, 800)</code></td></tr>
<tr><td style="padding:0.5rem;"><code>swipe</code></td><td style="padding:0.5rem;">Swipe from (x1,y1) to (x2,y2)</td><td style="padding:0.5rem;"><code>ACTION: swipe(500, 1000, 500, 200)</code></td></tr>
<tr><td style="padding:0.5rem;"><code>input_text</code></td><td style="padding:0.5rem;">Type text into focused field</td><td style="padding:0.5rem;"><code>ACTION: input_text("Hello")</code></td></tr>
<tr><td style="padding:0.5rem;"><code>launch_app</code></td><td style="padding:0.5rem;">Launch app by package name</td><td style="padding:0.5rem;"><code>ACTION: launch_app("com.microsoft.office.outlook")</code></td></tr>
<tr><td style="padding:0.5rem;"><code>press_key</code></td><td style="padding:0.5rem;">Press hardware key</td><td style="padding:0.5rem;"><code>ACTION: press_key("HOME")</code></td></tr>
<tr><td style="padding:0.5rem;"><code>DONE</code></td><td style="padding:0.5rem;">Signal mission completion</td><td style="padding:0.5rem;"><code>DONE: Task completed successfully.</code></td></tr>
</table>
<div class="tip">
<strong>Note:</strong> The parser automatically strips <think> tags and other reasoning artifacts that some models (like Grok) emit before the actual action. You don't need to do anything special — just use a supported model and the app handles the rest.
</div>
<h2>Safety & Privacy</h2>
<ul>
<li><strong>The agent cannot unlock your phone.</strong> If the screen is locked, it will ask you to unlock it first.</li>
<li><strong>The agent cannot bypass app security.</strong> Apps with biometric or PIN locks will block the agent.</li>
<li><strong>Everything stays on your device.</strong> Screenshots and commands never leave your phone. Only the AI chat requests (with screenshots as base64 images) go to your chosen provider.</li>
<li><strong>You are always in control.</strong> The agent executes one action at a time and reports what it does. You can stop a mission at any time.</li>
</ul>
<h2>Tips for Best Results</h2>
<ul>
<li><strong>Keep your screen on.</strong> Enable the Screen Wake Lock toggle in the dashboard so the agent can work uninterrupted. The app also sets <code>FLAG_KEEP_SCREEN_ON</code> on the main activity.</li>
<li><strong>Use simple, clear instructions.</strong> "Open Gmail and show unread emails" works better than "Check my mail."</li>
<li><strong>Be patient.</strong> The agent takes screenshots between each action, which takes a moment. Complex tasks may use 10-30 actions.</li>
<li><strong>Correct the agent.</strong> If it taps the wrong place, tell it in your next task and it will adjust.</li>
<li><strong>Use vision-capable models.</strong> Models that support image input can "see" screenshots directly, making the agent much more capable.</li>
<li><strong>One task at a time.</strong> The ReAct loop works best with a single, focused mission. Break complex workflows into smaller tasks.</li>
</ul>
</div>
<footer>
<p>BunnyRunner Documentation</p>
</footer>
</body>
</html>