Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,9 @@ jobs:
run: |
echo "y" | $ANDROID_SDK_ROOT/cmdline-tools/latest/bin/sdkmanager "ndk;29.0.13113456"

- name: Clone llama.cpp (pinned to b8390)
- name: Clone llama.cpp (pinned to b9574)
run: |
git clone --depth 1 --branch b8390 https://github.com/ggerganov/llama.cpp.git $GITHUB_WORKSPACE/../llama.cpp-master
git clone --depth 1 --branch b9574 https://github.com/ggerganov/llama.cpp.git $GITHUB_WORKSPACE/../llama.cpp-master

- name: Remove prebuilt native libs (compiled from source in CI)
run: rm -rf lib-llama-android/src/main/jniLibs
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,6 @@ CLAUDE.md

# Build artifacts
build_log.txt

# Local Claude Code config (never publish)
.claude/
70 changes: 62 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,6 @@
[![Min SDK](https://img.shields.io/badge/Min%20SDK-API%2030-green.svg)](https://developer.android.com/about/versions/11/highlights)
[![Build](https://github.com/Helldez/HearoPilot-App/actions/workflows/build.yml/badge.svg)](https://github.com/Helldez/HearoPilot-App/actions/workflows/build.yml)

<a href="https://play.google.com/store/apps/details?id=com.hearopilot.app">
<img src="https://play.google.com/intl/en_us/badges/static/images/badges/en_badge_web_generic.png" alt="Get it on Google Play" height="60">
</a>

**On-device audio transcription and AI insights for Android**

HearoPilot combines a local Speech-to-Text engine with a local Large Language Model to deliver
Expand All @@ -19,7 +15,7 @@ server.
## Features

- **Real-time transcription** — streaming STT via Sherpa-ONNX (NeMo Parakeet TDT 0.6B Int8)
- **On-device AI insights** — contextual analysis via local LLM, fully on-device
- **On-device AI insights** — contextual analysis via llama.cpp (Gemma 3 1B Q6_K)
- **100% offline** — privacy-first; no network calls during recording
- **Four recording modes** — Simple Listening, Short Meeting, Long Meeting, Real-Time Translation
- **25 UI languages** — full i18n including localized LLM system prompts
Expand Down Expand Up @@ -133,7 +129,7 @@ AudioRecord (16 kHz mono, PCM 16-bit, AudioSource.MIC)

| Parameter | Value | Rationale |
|---|---|---|
| `AudioSource.MIC` | — | Delivers raw PCM; `VOICE_RECOGNITION` activates HAL noise reduction on some devices that degrades transducer accuracy |
| `AudioSource.MIC` | — | Delivers raw PCM; `VOICE_RECOGNITION` activates HAL noise reduction on some devices (e.g. OnePlus) that degrades transducer accuracy |
| Recording thread priority | `URGENT_AUDIO` | Prevents audio buffer drops under CPU load |
| ADPF hint (VAD/ASR thread) | 50 ms target | Signals scheduler to prefer big cores on big.LITTLE SoCs; prevents ASR parking on efficiency cores |
| VAD window size | 512 samples | Standard Silero-VAD frame size |
Expand Down Expand Up @@ -172,6 +168,9 @@ models to echo the wrapper keywords instead of translating.
### llama.cpp Optimizations (native layer — `ai_chat.cpp`)

`libai-chat.so` is compiled from C++ source on every build (CMake, `externalNativeBuild`).
The prebuilt `.so` was removed in commit `2380ef7`; always run `./gradlew assembleDebug` after
changing any C++ parameter to pick up the new binary.

| Optimization | Value / Setting | Rationale |
|---|---|---|
| Context size | 4096 tokens | Covers worst-case LONG_MEETING (~3968 tokens) while halving KV cache memory vs 8192 |
Expand Down Expand Up @@ -209,7 +208,7 @@ Token budget flows from `SyncSttLlmUseCase` → `LlmRepository` → `LlamaAndroi

### Indicative Targets

- STT latency: < 200 ms per segment (high-end ARM device)
- STT latency: < 200 ms per segment (Snapdragon 888)
- LLM inference: 2–8 s per insight (varies by chip and mode)
- Memory footprint: < 800 MB active (STT + LLM loaded)

Expand Down Expand Up @@ -254,7 +253,7 @@ git lfs install

# 2. Clone (LFS objects are downloaded automatically)
git clone https://github.com/Helldez/HearoPilot-App.git
cd HearoPilot
cd HearoPilot-App

# 3. Configure Firebase
# Copy the example file and fill in your own Firebase project credentials.
Expand Down Expand Up @@ -285,6 +284,32 @@ adb install app/build/outputs/apk/debug/app-debug.apk

---

## Project History

Key milestones from MVP to the current release:

- Initial MVP — on-device STT + LLM, Clean Architecture
- Foreground Service for screen-off recording
- Room database for persistent sessions
- Custom `ModelDownloadManager` with partial-file resume
- VAD configurability (threshold, min silence, max speech)
- GC-free audio buffer + min audio context gate
- i18n: 25 languages with fully localized LLM system prompts
- Material Design 3 + dark mode
- Four recording modes (Simple, Short Meeting, Long Meeting, Real-Time Translation)
- R8 full-mode + ProGuard
- Stateless LLM inference + power optimizations
- LLM system prompt KV cache reuse
- Adaptive LLM unload/reload for Long Meeting (kswapd ANR fix)
- Global transcription search (full-text, highlighted snippets)
- Inline editing of segments, insights, and tasks
- Dual LLM variant support (Q8\_0 / IQ4\_NL) with auto device-tier detection
- Foreground services for model download and LLM batch processing
- Adaptive conservative-thread logic with persisted learning
- `libai-chat.so` compiled from source via CMake (no prebuilt)

---

## Localization

Supported locales (25): `en`, `bg`, `cs`, `da`, `de`, `el`, `es`, `et`, `fi`, `fr`, `hr`, `hu`,
Expand All @@ -296,6 +321,34 @@ consistent JSON parsing.

---

## Troubleshooting

**App crashes on launch**
→ Check that `RECORD_AUDIO` permission is granted
→ Run `./gradlew.bat clean assembleDebug` and verify zero errors

**STT produces no output**
→ Confirm the STT model is downloaded (check Settings)
→ Test on a physical device — emulator audio capture is unreliable
→ `adb logcat | grep SherpaOnnx`

**AI Insights not generated**
→ Confirm the LLM model is downloaded
→ Device needs ~1.5 GB free RAM with the LLM model loaded
→ Check the status chip in the TopBar shows "Ready"
→ `adb logcat | grep "LlamaAndroid\|SyncSttLlm"`

**Download stuck or slow**
→ Downloads resume from where they stopped — just retry
→ Ensure stable Wi-Fi; models are large (STT ~670 MB, LLM ~1 GB)

**Transcription inaccurate**
→ Speak clearly at ~30 cm from the microphone
→ Reduce background noise
→ Do **not** use `VOICE_RECOGNITION` audio source (OnePlus HAL issue)

---

## Roadmap

- [x] Global transcription search (full-text, highlighted snippets)
Expand All @@ -304,6 +357,7 @@ consistent JSON parsing.
- [ ] Export sessions (TXT / JSON / PDF)
- [ ] Speaker diarization
- [ ] Home screen widget for quick record
- [ ] Optional end-to-end-encrypted cloud sync
- [ ] Kotlin Multiplatform (iOS support)

---
Expand Down
4 changes: 2 additions & 2 deletions app/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ android {
applicationId = "com.hearopilot.app"
minSdk = 30
targetSdk = 35
versionCode = 20260327
versionName = "1.25.00"
versionCode = 20260609
versionName = "1.32.01"

testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
vectorDrawables {
Expand Down
Binary file modified app/src/main/ic_launcher-playstore.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions app/src/main/jniLibs/arm64-v8a/libonnxruntime.so
Git LFS file not shown
4 changes: 2 additions & 2 deletions app/src/main/jniLibs/arm64-v8a/libsherpa-onnx-c-api.so
Git LFS file not shown
4 changes: 2 additions & 2 deletions app/src/main/jniLibs/arm64-v8a/libsherpa-onnx-cxx-api.so
Git LFS file not shown
4 changes: 2 additions & 2 deletions app/src/main/jniLibs/arm64-v8a/libsherpa-onnx-jni.so
Git LFS file not shown
13 changes: 8 additions & 5 deletions app/src/main/res/drawable/ic_launcher_background.xml
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
<?xml version="1.0" encoding="utf-8"?>
<shape xmlns:android="http://schemas.android.com/apk/res/android">
<shape xmlns:android="http://schemas.android.com/apk/res/android"
android:shape="rectangle">
<gradient
android:startColor="#5E17EB"
android:endColor="#1E3A8A"
android:angle="135"
android:type="linear"/>
android:type="radial"
android:startColor="#1F1640"
android:endColor="#15102A"
android:centerX="0.5"
android:centerY="0.5"
android:gradientRadius="60dp"/>
</shape>
99 changes: 85 additions & 14 deletions app/src/main/res/drawable/ic_launcher_foreground.xml
Original file line number Diff line number Diff line change
@@ -1,19 +1,90 @@
<vector xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:aapt="http://schemas.android.com/aapt"
android:width="108dp"
android:height="108dp"
android:viewportWidth="3637"
android:viewportHeight="3638">
<group android:scaleX="0.49986255"
android:scaleY="0.5"
android:translateX="909.49994"
android:translateY="909.5">
<path
android:pathData="M896.2,1040 L896.2,1040 1173,1316.7C1528.9,1672.4 2108.9,1672.4 2464.8,1316.7L2741.6,1040 2741.6,1040 2808,1084.9C3060.7,1272.7 3217,1532.3 3217,1819 3217,2105.7 3060.7,2365.3 2808,2553.1L2741.6,2598 2741.6,2592.3 2464.8,2315.6C2108.9,1959.8 1528.9,1959.8 1173,2315.6L896.2,2592.3 896.2,2592.3 896.2,2595.9 833,2553.1C580.3,2365.3 424,2105.7 424,1819 424,1532.3 580.3,1272.7 833,1084.9L896.2,1042.1Z"
android:fillColor="#FFFFFF"
android:fillType="evenOdd"/>
<path
android:pathData="M2758.9,371 L2829.3,491.6C2840.7,511.1 2856.9,527.3 2876.4,538.7L2997,609.2C3003.2,612.8 3007,619.4 3007,626.5 3007,633.6 3003.2,640.3 2997,643.9L2876.4,714.3C2856.9,725.7 2840.7,741.9 2829.3,761.4L2758.9,882C2755.3,888.2 2748.6,892 2741.5,892 2734.4,892 2727.8,888.2 2724.1,882L2653.7,761.4C2642.3,741.9 2626.1,725.7 2606.6,714.3L2486,643.9C2479.8,640.3 2476,633.6 2476,626.5 2476,619.4 2479.8,612.8 2486,609.2L2606.6,538.7C2626.1,527.3 2642.3,511.1 2653.7,491.6L2724.1,371C2727.8,364.8 2734.4,361 2741.5,361 2748.6,361 2755.3,364.8 2758.9,371Z"
android:fillColor="#FFFFFF"
android:fillType="evenOdd"/>
android:viewportWidth="4057"
android:viewportHeight="4057">
<group
android:translateX="913"
android:translateY="913"
android:scaleX="0.55"
android:scaleY="0.55">
<group
android:translateX="-1409"
android:translateY="-1409">
<!-- Bar L3 (innermost left, 1949) -->
<path
android:pathData="M2382,2603 C2382,2525.68 2444.68,2463 2522,2463 L2534,2463 C2611.32,2463 2674,2525.68 2674,2603 L2674,4272 C2674,4349.32 2611.32,4412 2534,4412 L2522,4412 C2444.68,4412 2382,4349.32 2382,4272 Z"
android:fillColor="#FFFFFF"
android:fillType="evenOdd"/>
<!-- Bar L2 (mid left, 1299) -->
<path
android:pathData="M1896,2928 C1896,2850.68 1958.68,2788 2036,2788 L2048,2788 C2125.32,2788 2188,2850.68 2188,2928 L2188,3947 C2188,4024.32 2125.32,4087 2048,4087 L2036,4087 C1958.68,4087 1896,4024.32 1896,3947 Z"
android:fillColor="#FFFFFF"
android:fillAlpha="0.88"
android:fillType="evenOdd"/>
<!-- Bar L1 (outer left, 638) -->
<path
android:pathData="M1409,3188.48 C1409,3110.89 1471.89,3048 1549.48,3048 L1561.51,3048 C1639.09,3048 1701.99,3110.89 1701.99,3188.48 L1701.99,3686.52 C1701.99,3764.11 1639.09,3827 1561.51,3827 L1549.48,3827 C1471.89,3827 1409,3764.11 1409,3686.52 Z"
android:fillColor="#FFFFFF"
android:fillAlpha="0.7"
android:fillType="evenOdd"/>
<!-- AI sparkle, scale 1.2x around (3438, 3438) -->
<group
android:pivotX="3438"
android:pivotY="3438"
android:scaleX="1.2"
android:scaleY="1.2">
<path
android:pathData="M3438,2837.5 C3438,3138 3738,3438 4038.5,3438 C3738,3438 3438,3738 3438,4038.5 C3438,3738 3138,3438 2837.5,3438 C3138,3438 3438,3138 3438,2837.5 Z"
android:fillType="evenOdd">
<aapt:attr name="android:fillColor">
<gradient
android:type="radial"
android:centerX="3438"
android:centerY="3438"
android:gradientRadius="600">
<item android:offset="0" android:color="#FFFFFF"/>
<item android:offset="1" android:color="#EDE9FE"/>
</gradient>
</aapt:attr>
</path>
</group>
<!-- Bar R-inner (mirrored, 1949) -->
<group
android:translateX="4493"
android:translateY="2463"
android:scaleX="-1"
android:scaleY="1">
<path
android:pathData="M0,140.002 C0,62.6812 62.6816,0 140.003,0 L151.997,0 C229.319,0 292,62.6812 292,140.002 L292,1809 C292,1886.32 229.319,1949 151.997,1949 L140.003,1949 C62.6816,1949 0,1886.32 0,1809 Z"
android:fillColor="#FFFFFF"
android:fillType="evenOdd"/>
</group>
<!-- Bar R-mid (mirrored, 1299) -->
<group
android:translateX="4979"
android:translateY="2788"
android:scaleX="-1"
android:scaleY="1">
<path
android:pathData="M0,140.002 C0,62.681 62.6811,0 140.002,0 L151.998,0 C229.319,0 292,62.681 292,140.002 L292,1159 C292,1236.32 229.319,1299 151.998,1299 L140.002,1299 C62.6811,1299 0,1236.32 0,1159 Z"
android:fillColor="#FFFFFF"
android:fillAlpha="0.88"
android:fillType="evenOdd"/>
</group>
<!-- Bar R-outer (mirrored, 779) -->
<group
android:translateX="5466"
android:translateY="3048"
android:scaleX="-1"
android:scaleY="1">
<path
android:pathData="M0,140.476 C0,62.8931 62.8932,0 140.476,0 L152.511,0 C230.094,0 292.987,62.8931 292.987,140.476 L292.987,638.524 C292.987,716.107 230.094,779 152.511,779 L140.476,779 C62.8932,779 0,716.107 0,638.524 Z"
android:fillColor="#FFFFFF"
android:fillAlpha="0.7"
android:fillType="evenOdd"/>
</group>
</group>
</group>
</vector>
69 changes: 69 additions & 0 deletions app/src/main/res/drawable/ic_launcher_monochrome.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
<?xml version="1.0" encoding="utf-8"?>
<vector xmlns:android="http://schemas.android.com/apk/res/android"
android:width="108dp"
android:height="108dp"
android:viewportWidth="4057"
android:viewportHeight="4057">
<group
android:translateX="913"
android:translateY="913"
android:scaleX="0.55"
android:scaleY="0.55">
<group
android:translateX="-1409"
android:translateY="-1409">
<path
android:pathData="M2382,2603 C2382,2525.68 2444.68,2463 2522,2463 L2534,2463 C2611.32,2463 2674,2525.68 2674,2603 L2674,4272 C2674,4349.32 2611.32,4412 2534,4412 L2522,4412 C2444.68,4412 2382,4349.32 2382,4272 Z"
android:fillColor="#FFFFFF"
android:fillType="evenOdd"/>
<path
android:pathData="M1896,2928 C1896,2850.68 1958.68,2788 2036,2788 L2048,2788 C2125.32,2788 2188,2850.68 2188,2928 L2188,3947 C2188,4024.32 2125.32,4087 2048,4087 L2036,4087 C1958.68,4087 1896,4024.32 1896,3947 Z"
android:fillColor="#FFFFFF"
android:fillType="evenOdd"/>
<path
android:pathData="M1409,3188.48 C1409,3110.89 1471.89,3048 1549.48,3048 L1561.51,3048 C1639.09,3048 1701.99,3110.89 1701.99,3188.48 L1701.99,3686.52 C1701.99,3764.11 1639.09,3827 1561.51,3827 L1549.48,3827 C1471.89,3827 1409,3764.11 1409,3686.52 Z"
android:fillColor="#FFFFFF"
android:fillType="evenOdd"/>
<group
android:pivotX="3438"
android:pivotY="3438"
android:scaleX="1.2"
android:scaleY="1.2">
<path
android:pathData="M3438,2837.5 C3438,3138 3738,3438 4038.5,3438 C3738,3438 3438,3738 3438,4038.5 C3438,3738 3138,3438 2837.5,3438 C3138,3438 3438,3138 3438,2837.5 Z"
android:fillColor="#FFFFFF"
android:fillType="evenOdd"/>
</group>
<group
android:translateX="4493"
android:translateY="2463"
android:scaleX="-1"
android:scaleY="1">
<path
android:pathData="M0,140.002 C0,62.6812 62.6816,0 140.003,0 L151.997,0 C229.319,0 292,62.6812 292,140.002 L292,1809 C292,1886.32 229.319,1949 151.997,1949 L140.003,1949 C62.6816,1949 0,1886.32 0,1809 Z"
android:fillColor="#FFFFFF"
android:fillType="evenOdd"/>
</group>
<group
android:translateX="4979"
android:translateY="2788"
android:scaleX="-1"
android:scaleY="1">
<path
android:pathData="M0,140.002 C0,62.681 62.6811,0 140.002,0 L151.998,0 C229.319,0 292,62.681 292,140.002 L292,1159 C292,1236.32 229.319,1299 151.998,1299 L140.002,1299 C62.6811,1299 0,1236.32 0,1159 Z"
android:fillColor="#FFFFFF"
android:fillType="evenOdd"/>
</group>
<group
android:translateX="5466"
android:translateY="3048"
android:scaleX="-1"
android:scaleY="1">
<path
android:pathData="M0,140.476 C0,62.8931 62.8932,0 140.476,0 L152.511,0 C230.094,0 292.987,62.8931 292.987,140.476 L292.987,638.524 C292.987,716.107 230.094,779 152.511,779 L140.476,779 C62.8932,779 0,716.107 0,638.524 Z"
android:fillColor="#FFFFFF"
android:fillType="evenOdd"/>
</group>
</group>
</group>
</vector>
Binary file removed app/src/main/res/drawable/logo_hearo_pilot_3.png
Binary file not shown.
Loading
Loading