Skip to content

Commit a0cc625

Browse files
DatafyingTechclaude
DatafyingTech
and
claude
committed
Phone access: a local API and an Android floating-dot app
The desktop app can now serve its dictation pipeline to other devices on the user's Tailscale network. localflow/server.py is a small standard-library HTTP server on 127.0.0.1:8770, exposed on the PC's Tailscale name by `tailscale serve` with no firewall change. Bearer token auth, WAV or raw PCM input with resampling, ptt and handsfree modes, and the same one-at-a-time lock as the hotkey path. Off by default; the tray menu gains "Enable phone access" and "Phone setup". The transcribe-and-clean pipeline is extracted into run_pipeline so the hotkey and the API share one implementation. Contract in docs/API.md. android/ is a Kotlin app: a floating dot over every app, hold for one phrase, tap for hands-free with a live waveform and discard/send buttons, the same colours as the desktop dot, text typed at the cursor through an accessibility service with a clipboard fallback. It starts its foreground service as special-use until the microphone permission is granted, so the dot appears on Android 14 before the mic is allowed. Compiled, lint-clean, unit-tested; released separately as android-v* tags built by .github/workflows/android.yml. Guide in docs/ANDROID.md. Version 0.2.0. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 parent a64b2a1 commit a0cc625

46 files changed

Lines changed: 4071 additions & 38 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/ISSUE_TEMPLATE/bug_report.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ body:
8888
attributes:
8989
label: LocalFlow version
9090
description: "`python -m localflow --version`, or the commit you are on."
91-
placeholder: "0.1.1"
91+
placeholder: "0.2.0"
9292
validations:
9393
required: true
9494

‎.github/workflows/android.yml‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
name: android
2+
3+
on:
4+
# No `paths` filter on push: a paths filter also applies to tag pushes, so a tag on a commit
5+
# that did not touch android/ would never build or release. The workflow is cheap enough to
6+
# run on every push to main; pull requests keep the filter.
7+
push:
8+
branches: [main, master]
9+
tags:
10+
- "android-v*"
11+
pull_request:
12+
paths:
13+
- "android/**"
14+
- ".github/workflows/android.yml"
15+
workflow_dispatch:
16+
17+
permissions:
18+
contents: write # needed only for the release step on tags
19+
20+
jobs:
21+
build:
22+
name: assembleDebug + lint + unit tests
23+
runs-on: ubuntu-latest # ships with the Android SDK; the build downloads platform 35 itself
24+
defaults:
25+
run:
26+
working-directory: android
27+
28+
steps:
29+
- uses: actions/checkout@v7
30+
31+
- uses: actions/setup-java@v4
32+
with:
33+
distribution: temurin
34+
java-version: "17"
35+
36+
- uses: gradle/actions/setup-gradle@v4
37+
38+
- name: Make the wrapper executable
39+
run: chmod +x gradlew
40+
41+
- name: Unit tests (WAV builder, text splice)
42+
run: ./gradlew testDebugUnitTest --no-daemon --stacktrace
43+
44+
- name: Build debug APK
45+
run: ./gradlew assembleDebug --no-daemon --stacktrace
46+
47+
- name: Lint
48+
run: ./gradlew lintDebug --no-daemon
49+
50+
- name: Rename APK
51+
run: cp app/build/outputs/apk/debug/app-debug.apk "LocalFlow-${GITHUB_REF_NAME//\//-}.apk"
52+
53+
- uses: actions/upload-artifact@v4
54+
with:
55+
name: app-debug.apk
56+
path: android/app/build/outputs/apk/debug/app-debug.apk
57+
if-no-files-found: error
58+
59+
- uses: actions/upload-artifact@v4
60+
with:
61+
name: lint-report
62+
path: android/app/build/reports/lint-results-debug.html
63+
if-no-files-found: ignore
64+
65+
- name: Release on tag
66+
if: startsWith(github.ref, 'refs/tags/android-v')
67+
uses: softprops/action-gh-release@v2
68+
with:
69+
name: ${{ github.ref_name }}
70+
files: |
71+
android/app/build/outputs/apk/debug/app-debug.apk
72+
android/LocalFlow-${{ github.ref_name }}.apk
73+
generate_release_notes: true

‎CHANGELOG.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,29 @@ All notable changes to LocalFlow are documented here. The format follows
44
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow
55
[Semantic Versioning](https://semver.org/).
66

7+
## [0.2.0] - 2026-09-11
8+
9+
### Added
10+
11+
- **Phone access.** A small HTTP API on the PC (`server.enabled`, off by default) lets another
12+
device send audio and get back the same cleaned text the desktop hotkey produces. It listens
13+
only on `127.0.0.1:8770`; `tailscale serve` exposes it on your Tailscale name with no firewall
14+
change. Bearer-token auth, WAV or raw PCM input with resampling, `mode=ptt` or `handsfree`,
15+
the same one-at-a-time lock as the desktop path. Tray menu gains **Enable phone access** and
16+
**Phone setup** (copies a `URL|TOKEN` line for the phone). Contract in `docs/API.md`.
17+
- **Android app** (`android/`, released separately as `android-v*` tags). A floating dot over
18+
every app: hold for one phrase, tap for hands-free with a live waveform and discard/send
19+
buttons, same colours as the desktop. Text is typed at the cursor through an accessibility
20+
service, with clipboard fallback. Setup guide in `docs/ANDROID.md`. Compiled and unit-tested;
21+
first device runs are by the community.
22+
- `run_pipeline` in `localflow/__main__.py`: the transcribe-and-clean pipeline as a reusable,
23+
side-effect-free function shared by the hotkey path and the API.
24+
25+
### Changed
26+
27+
- The privacy statement now distinguishes "never leaves this machine" (default) from "never
28+
leaves your own devices" (phone access on).
29+
730
## [0.1.1] - 2026-09-11
831

932
### Changed

‎README.md‎

Lines changed: 53 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,9 @@ what you can dictate. LocalFlow is a like-for-like replacement that runs entirel
7373
and has a one-click **Pause (free GPU)** for gaming sessions.
7474
- **A dot, not a window.** A 22-pixel status dot you can drag anywhere. It never steals focus from
7575
whatever you're typing into.
76+
- **Works from your phone.** An Android app puts the same dot on your phone screen. It sends your
77+
voice to your own PC over Tailscale and types the result into whatever app you are in. Your
78+
desktop does the work; nothing goes to anyone else's server.
7679
- **Your words stay yours.** A local history file you own, and nothing else. No telemetry at all.
7780

7881
---
@@ -292,7 +295,9 @@ slow request never costs you the whole speech.
292295
## Privacy and your data
293296

294297
Nothing you say or type is transmitted anywhere. There is no account, no telemetry, no crash
295-
reporting, no update check, and no analytics of any kind. LocalFlow opens no listening socket.
298+
reporting, no update check, and no analytics of any kind. LocalFlow opens no listening socket
299+
unless you turn on phone access, and then only on the local machine for Tailscale to reach
300+
(see [From your phone](#from-your-phone)).
296301

297302
Everything it writes lives in the LocalFlow folder, and that is the complete list:
298303

@@ -372,6 +377,53 @@ synthetic input. Use it for chat, and if a game's rules forbid any synthetic inp
372377

373378
---
374379

380+
## From your phone
381+
382+
The Android app is a floating dot, like the desktop one, that dictates through your PC.
383+
384+
![The Flow Dot in each of its states](docs/images/flow-dot-states.png)
385+
386+
- **Long-press and hold** the dot to dictate one phrase. Release to send.
387+
- **Tap** it to start hands-free. It expands into a pill with a live waveform between an ✕ to
388+
discard and a ✓ to send. Tap ✓ or the dot to finish.
389+
- The colours mean the same as on the desktop: grey idle, red listening, orange hands-free,
390+
blue while your PC thinks, green when the text has landed, a red ring on an error.
391+
- The text is typed at the cursor of whatever field is focused, in any app. If an app blocks
392+
that, it goes to the clipboard and you paste it.
393+
394+
**How it connects.** The phone talks to a small API on the PC over
395+
[Tailscale](https://tailscale.com), a private network between your own devices that works from
396+
anywhere with a signal. The PC transcribes and cleans exactly as it does for the desktop hotkey,
397+
so quality is identical. Round trip from LTE measured at about 40 ms plus the usual processing,
398+
so well under a second per phrase.
399+
400+
**Setup, in order:**
401+
402+
1. Install Tailscale on the PC and on the phone, signed in with the **same** account.
403+
2. On the PC, right-click the dot and turn on **Enable phone access**, then open **Phone setup**
404+
and press **Copy setup line**.
405+
3. Install the Android app from the [Releases page](https://github.com/DatafyingTech/LocalFlow/releases)
406+
(the `.apk` asset of the latest `android-v*` release), open it, and press **Paste setup line**.
407+
4. Grant the four permissions the app asks for, in the order it asks: microphone, display over
408+
other apps, the LocalFlow accessibility service, and notifications.
409+
5. Switch on **Show the dot**.
410+
411+
The full guide, with the exact Settings paths on Samsung and stock Android and a
412+
troubleshooting list, is in [docs/ANDROID.md](docs/ANDROID.md). The API the phone uses is
413+
documented in [docs/API.md](docs/API.md), so you can build your own client.
414+
415+
**What this changes about privacy.** With phone access on, audio travels between your phone and
416+
your PC through Tailscale, which is encrypted end to end and never sees the content. The PC API
417+
listens only on the local machine and is reachable only through that tunnel, with a token the PC
418+
generates. So the promise becomes "never leaves your own devices" rather than "never leaves this
419+
machine". Phone access is off by default. If the PC is asleep or off, the phone cannot dictate;
420+
it says so instead of failing silently.
421+
422+
**iPhone** is not supported. iOS keyboard extensions cannot draw a floating dot or type into
423+
other apps the way Android's accessibility services can.
424+
425+
---
426+
375427
## When something goes wrong
376428

377429
**Start here.** Run this and read what it tells you — it checks every part of the stack:

‎android/.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Gradle / Android build output
2+
.gradle/
3+
build/
4+
app/build/
5+
local.properties
6+
.idea/
7+
*.iml
8+
.kotlin/
9+
captures/

‎android/app/build.gradle.kts‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
plugins {
2+
id("com.android.application")
3+
id("org.jetbrains.kotlin.android")
4+
}
5+
6+
android {
7+
namespace = "tech.datafying.localflow"
8+
compileSdk = 35
9+
10+
defaultConfig {
11+
applicationId = "tech.datafying.localflow"
12+
minSdk = 26
13+
targetSdk = 35
14+
versionCode = 1
15+
versionName = "0.1.0"
16+
}
17+
18+
buildTypes {
19+
debug {
20+
// The debug build is what the GitHub release ships: it is signed with the
21+
// auto-generated debug key, so it installs directly on a phone.
22+
isMinifyEnabled = false
23+
}
24+
release {
25+
isMinifyEnabled = true
26+
isShrinkResources = true
27+
proguardFiles(getDefaultProguardFile("proguard-android-optimize.txt"), "proguard-rules.pro")
28+
}
29+
}
30+
31+
compileOptions {
32+
sourceCompatibility = JavaVersion.VERSION_17
33+
targetCompatibility = JavaVersion.VERSION_17
34+
}
35+
kotlinOptions {
36+
jvmTarget = "17"
37+
}
38+
39+
buildFeatures {
40+
buildConfig = true
41+
}
42+
43+
lint {
44+
// Lint runs in CI for the report; warnings must not turn the build red.
45+
abortOnError = false
46+
checkReleaseBuilds = false
47+
}
48+
}
49+
50+
dependencies {
51+
implementation("androidx.core:core-ktx:1.13.1")
52+
implementation("androidx.appcompat:appcompat:1.7.0")
53+
implementation("com.squareup.okhttp3:okhttp:4.12.0")
54+
testImplementation("junit:junit:4.13.2")
55+
}

‎android/app/proguard-rules.pro‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# OkHttp probes for these platform classes reflectively; they are absent on Android.
2+
-dontwarn okhttp3.internal.platform.**
3+
-dontwarn org.conscrypt.**
4+
-dontwarn org.bouncycastle.**
5+
-dontwarn org.openjsse.**

‎android/app/src/main/AndroidManifest.xml‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<manifest xmlns:android="http://schemas.android.com/apk/res/android">
3+
4+
<uses-permission android:name="android.permission.INTERNET" />
5+
<uses-permission android:name="android.permission.RECORD_AUDIO" />
6+
<uses-permission android:name="android.permission.SYSTEM_ALERT_WINDOW" />
7+
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
8+
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MICROPHONE" />
9+
<!-- Android 14+: lets the dot start before the microphone permission has been granted, so
10+
it can appear and guide the user to the permission instead of silently not showing. -->
11+
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_SPECIAL_USE" />
12+
<uses-permission android:name="android.permission.POST_NOTIFICATIONS" />
13+
<uses-permission android:name="android.permission.VIBRATE" />
14+
<!-- Sideloaded app: asking to be exempt from Doze keeps the dot alive under Samsung's
15+
aggressive battery management. -->
16+
<uses-permission android:name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS" />
17+
18+
<application
19+
android:allowBackup="false"
20+
android:icon="@mipmap/ic_launcher"
21+
android:roundIcon="@mipmap/ic_launcher_round"
22+
android:label="@string/app_name"
23+
android:networkSecurityConfig="@xml/network_security_config"
24+
android:usesCleartextTraffic="true"
25+
android:supportsRtl="true"
26+
android:theme="@style/Theme.LocalFlow">
27+
28+
<activity
29+
android:name=".MainActivity"
30+
android:exported="true"
31+
android:launchMode="singleTop"
32+
android:windowSoftInputMode="adjustResize">
33+
<intent-filter>
34+
<action android:name="android.intent.action.MAIN" />
35+
<category android:name="android.intent.category.LAUNCHER" />
36+
</intent-filter>
37+
</activity>
38+
39+
<service
40+
android:name=".OverlayService"
41+
android:exported="false"
42+
android:foregroundServiceType="microphone|specialUse">
43+
<property
44+
android:name="android.app.PROPERTY_SPECIAL_USE_FGS_SUBTYPE"
45+
android:value="floating dictation control" />
46+
</service>
47+
48+
<service
49+
android:name=".InsertionService"
50+
android:exported="true"
51+
android:label="@string/app_name"
52+
android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
53+
<intent-filter>
54+
<action android:name="android.accessibilityservice.AccessibilityService" />
55+
</intent-filter>
56+
<meta-data
57+
android:name="android.accessibilityservice"
58+
android:resource="@xml/accessibility_service_config" />
59+
</service>
60+
</application>
61+
</manifest>

0 commit comments

Comments
 (0)