Developer: Harsh · License: MIT · 100% on-device, no cloud.
Scroll Instagram Reels, YouTube Shorts, Facebook Reels and similar vertical feeds using hand gestures in the air — no touching the screen. Put your phone on a stand, start detection, and control the feed with your hand in front of the front camera.
Everything runs 100% on-device. No cloud, no APIs, no model training. Hand tracking uses Google MediaPipe Hand Landmarker (a pretrained model); gestures are recognised with custom geometry — there is no ML classification.
Package id:
com.example.auto_scroll· App name: AirScroll · minSdk 26 · targetSdk 36
| Gesture | Action | How it's detected |
|---|---|---|
| Swipe Up — index finger down→up | Scroll to next reel | Index-tip velocity + distance |
| Victory ✌️ — two fingers | Scroll to previous reel | Index + middle extended, ring + pinky folded |
| Thumbs Up | Like — double-tap at screen centre | Thumb extended & topmost, 4 fingers folded |
| Closed Fist → open the hand | Resume (only while paused) | Fist (clench) then open palm |
Scrolling is motion-based (flick your index finger up = next, Victory = previous); thumbs up likes (double-tap centre).
Detection never pauses on its own — no gesture can pause it. You pause it yourself from the overlay bubble or the notification. To resume hands-free, make a fist then open your hand (clench-and-release). A closed fist on its own does nothing; a lone open palm does nothing (so it can't accidentally pause/resume).
Thumbs-up vs fist: a thumbs-up is recognised only when the thumb is extended and is the highest point of the hand. A fist (thumb tucked level with the fingers) is a "Closed Fist" that does nothing by itself — it only arms Resume while paused.
- Confidence gate — frames below
0.8are ignored. - Stability — a pose must persist N consecutive frames (sensitivity-dependent).
- Stable pose — Victory / Thumbs-up must persist a few frames before firing.
- Cooldown — after any action, triggers are blocked for the configured window (0.5–3s).
- Latch — a held pose fires once; the hand must leave the pose to re-arm.
- Smoothing — landmark coordinates are EMA-smoothed to kill jitter.
com.example.auto_scroll
├── AirScrollApp.kt # @HiltAndroidApp
├── camera/ # CameraX + MediaPipe
│ ├── CameraManager.kt # front camera, ImageAnalysis → upright Bitmap
│ └── HandDetectionManager.kt # MediaPipe Hand Landmarker (LIVE_STREAM)
├── gesture/ # Gesture engine (no ML)
│ ├── GestureGeometry.kt # angle/distance helpers
│ ├── GestureClassifier.kt # static pose classification
│ ├── LandmarkSmoother.kt # EMA smoothing
│ └── GestureEngine.kt # stability, hold, cooldown, swipe, pause/resume
├── domain/model/ # GestureType, GestureEvent, AppSettings, HandResult…
├── data/
│ ├── local/ # Room: AppDatabase, AnalyticsDao, entities
│ └── repository/ # Settings (DataStore), Analytics (Room), DetectionState
├── service/
│ ├── AirScrollAccessibilityService.kt # system-wide swipe/tap (dispatchGesture)
│ ├── DetectionService.kt # foreground service: full pipeline + notification
│ └── OverlayService.kt # draggable floating bubble
├── di/ # Manual DI: ServiceLocator + AppViewModelFactory
├── presentation/ # Jetpack Compose UI (Navigation, screens, ViewModels)
│ ├── MainActivity.kt
│ ├── navigation/ # AppNavHost, Routes
│ ├── splash / home / settings / calibration / analytics
│ └── components/ # AirCard, StatusRow, HandLandmarksOverlay…
├── ui/theme/ # Black / Dark gray / Gold Material 3 theme
└── util/ # Constants, PermissionUtils
Tech: Kotlin · Jetpack Compose + Material 3 · CameraX · MediaPipe Hand Landmarker · Accessibility Service · Room · DataStore · Navigation Compose · Coroutines/StateFlow.
DI note: The spec asked for Hilt, but Hilt's Gradle plugin (2.57.x) is incompatible with AGP 9.x (it fails configuration with “Android BaseExtension not found”). Since this project is pinned to AGP 9.2.1 / Gradle 9.4.1, DI is provided by a tiny hand-written container —
di/ServiceLocator(singletons) +di/AppViewModelFactory(ViewModels) — which gives the same shared-singleton wiring. Swapping back to Hilt is a localized change once a Hilt release supports AGP 9.
Download the model and place it at app/src/main/assets/hand_landmarker.task:
https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/latest/hand_landmarker.task
Model card / license: https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker
(See app/src/main/assets/README.txt.) The build will compile without it, but detection
will report an error at runtime until the file is present.
./gradlew :app:assembleDebug # build
./gradlew :app:installDebug # install on a connected device
./gradlew :app:testDebugUnitTest # run gesture unit testsOpen in Android Studio (Giraffe+ / latest) and Run, or use the CLI above.
- Camera — runtime permission.
- Accessibility Service — enable “AirScroll Gesture Control” in system settings (required to perform swipes/taps).
- Display over other apps — for the floating overlay.
- Notifications (Android 13+) — for the foreground-service notification.
- Disable battery optimization (recommended) — so detection survives in the background.
- Place the phone on a stand facing you.
- Tap Start. Optionally tap Show Overlay.
- Open Instagram/YouTube/Facebook and control the feed with gestures: swipe your index finger up = next, Victory ✌️ = previous, thumbs up = like. Pause is manual (overlay/notification); fist → open hand resumes hands-free.
- Test/tune gestures live in Calibration (stop background detection first so the camera is free).
- Enable/disable each gesture individually.
- Sensitivity: Low / Medium / High / Custom.
- Cooldown: 0.5 / 1 / 1.5 / 2 / 3 seconds.
- Like position: normalised tap coordinates, persisted in DataStore.
Total gestures, total scrolls, total likes, session duration, session count, most-used gesture, and a per-gesture breakdown. Clear from the Analytics screen.
- Unit tests (
app/src/test/.../GestureLogicTest.kt): pure-JVM tests build synthetic 21-point hands and verify classification (open palm / thumbs-up / victory) plus the engine safety pipeline (stability, 500ms hold, single-fire latch, pause→resume, low-confidence rejection). Run with./gradlew testDebugUnitTest. - Manual / calibration: the Calibration screen renders the live skeleton, detected gesture, confidence, landmark count and FPS for visual verification on-device.
- Suggested additions: instrumented tests for the accessibility gesture dispatch and
Room DAO (
@RunWith(AndroidJUnit4)), and Compose UI tests for the screens.
No hand / poor lighting (confidence gate + smoother reset), multiple hands (most-confident hand chosen), camera interruption (error surfaced, pipeline stops), screen rotation (frames rotated upright), accessibility disabled (action skipped + warning), overlay permission revoked (overlay self-stops), app minimized (foreground service keeps running), battery optimization (prompt to disable).
Pinned in gradle/libs.versions.toml. Key versions:
| Library | Version |
|---|---|
| Android Gradle Plugin | 9.2.1 |
| Kotlin | 2.2.10 |
| KSP | 2.2.10-2.0.2 |
| Room | 2.7.2 |
| CameraX | 1.4.2 |
| MediaPipe tasks-vision | 0.10.21 |
| DataStore | 1.1.7 |
| Navigation Compose | 2.9.0 |
| Compose BOM | 2026.02.01 |
Build environment note: this project targets a very new AGP (9.x) which uses built-in Kotlin support (only the Compose Kotlin plugin is applied separately). Room is wired through KSP. Because built-in Kotlin disallows the legacy
kotlin.sourceSetsDSL that KSP uses to register generated sources,gradle.propertiessetsandroid.disallowKotlinSourceSets=false. Adjust versions inlibs.versions.tomlto match your installed SDK/Studio if a specific artifact isn't yet published for these exact versions.
- Like works by double-tapping the centre of the screen, which is the standard "like" gesture for Reels / Shorts / Facebook video. If an app likes on a different interaction, it won't be liked.
- Calibration and the background service both use the single front camera; don't run them simultaneously.
- This app automates touch input via the Accessibility Service for personal accessibility convenience; respect the terms of service of the apps you use it with.