diff --git a/README.md b/README.md index 96f601b..c983170 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ The data source handles account sign-in and usage retrieval. Bridge sends only u | Waveshare AMOLED 2.16 | 480×480 AMOLED | Touchscreen, clock screensaver, auto-rotation | | M5StickS3 | 135×240 LCD | Button navigation, compact size | | Waveshare ePaper 3.97 | 800×480 e-paper with four grayscale levels | Temperature and humidity, usage trends (microSD card required) | -| ESP-Mosaico | 480×480 AMOLED | Touchscreen, AI button, auto-rotation | +| ESP-Mosaico | 480×480 AMOLED | Touchscreen, AI button, auto-rotation, camera gestures (experimental) | | Waveshare RLCD 4.2 | 400×300 monochrome reflective LCD | KEY / BOOT navigation, clock, usage trends (microSD card required) | All devices support Codex / Claude usage display, retain the last usage data when disconnected, and support firmware updates over Bluetooth. See the [device guide](docs/devices.md) for a full feature comparison and control shortcuts. diff --git a/README.zh-CN.md b/README.zh-CN.md index c806834..93cbf2e 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -30,7 +30,7 @@ flowchart LR | Waveshare AMOLED 2.16 | 480×480 AMOLED | 触屏、时钟屏保、自动旋转 | | M5StickS3 | 135×240 LCD | 按键翻页、小尺寸 | | Waveshare ePaper 3.97 | 800×480 四灰阶墨水屏 | 温湿度、用量趋势（需 microSD 卡） | -| ESP-Mosaico | 480×480 AMOLED | 触屏、AI 键、自动旋转 | +| ESP-Mosaico | 480×480 AMOLED | 触屏、AI 键、自动旋转、摄像头手势（实验性） | | Waveshare RLCD 4.2 | 400×300 黑白反射式 LCD | KEY / BOOT 翻页、时钟、用量趋势（需 microSD 卡） | 所有设备均支持 Codex / Claude 用量显示、断连保留最后用量与蓝牙固件更新。完整功能对照及快捷操作见[设备说明](docs/devices.md)。 diff --git a/THIRD_PARTY_LICENSES.md b/THIRD_PARTY_LICENSES.md index a6f42ab..b6a5f7a 100644 --- a/THIRD_PARTY_LICENSES.md +++ b/THIRD_PARTY_LICENSES.md @@ -173,6 +173,7 @@ For ESP Component Registry entries, `Registry license` is the API value for the - Source: https://components.espressif.com/components/espressif/usb/versions/1.3.0 - SPDX: `Apache-2.0` - Copyright/notice: https://github.com/espressif/esp-usb/blob/ded385702ecae4d5c15c497149062ce577fa167a/host/usb/LICENSE +- ESP-Mosaico version: `1.5.0`; source: https://components.espressif.com/components/espressif/usb/versions/1.5.0; copyright/notice: https://github.com/espressif/esp-usb/blob/eb4618660a83e80cd57dcda736edbfa884cb88cc/host/usb/LICENSE ## `idf` @@ -527,10 +528,10 @@ SOFTWARE. ## `espressif/dhara` -- Version: `1.0.0` -- Source: https://components.espressif.com/components/espressif/dhara/versions/1.0.0 +- Version: `1.1.0` +- Source: https://components.espressif.com/components/espressif/dhara/versions/1.1.0 - SPDX conclusion from distributed license: `ISC` -- Copyright/notice: https://components.espressif.com/components/espressif/dhara/versions/1.0.0/license +- Copyright/notice: https://components.espressif.com/components/espressif/dhara/versions/1.1.0/license ## `espressif/esp_lcd_touch_cst9220` @@ -566,3 +567,59 @@ SOFTWARE. - Source: https://components.espressif.com/components/tangyumei3535/bmm150_sensorapi/versions/1.0.0 - SPDX conclusion from distributed license: `BSD-3-Clause` - Copyright/notice: https://components.espressif.com/components/tangyumei3535/bmm150_sensorapi/versions/1.0.0/license + +## `espressif/esp-dl` + +- ESP-Mosaico version: `3.3.11` +- Source: https://components.espressif.com/components/espressif/esp-dl/versions/3.3.11 +- SPDX conclusion: `MIT` +- Copyright/notice: https://components.espressif.com/components/espressif/esp-dl/versions/3.3.11/license + +## `espressif/esp_video` + +- ESP-Mosaico version: `2.5.0` +- Source: https://components.espressif.com/components/espressif/esp_video/versions/2.5.0 +- SPDX conclusion: `LicenseRef-Espressif-MIT` +- Copyright/notice: https://components.espressif.com/components/espressif/esp_video/versions/2.5.0/license + +## `espressif/esp_cam_sensor` + +- ESP-Mosaico version: `2.6.0` +- Source: https://components.espressif.com/components/espressif/esp_cam_sensor/versions/2.6.0 +- SPDX conclusion: `Apache-2.0` +- Copyright/notice: https://components.espressif.com/components/espressif/esp_cam_sensor/versions/2.6.0/license + +## `espressif/esp_sccb_intf` + +- ESP-Mosaico version: `0.0.8` +- Source: https://components.espressif.com/components/espressif/esp_sccb_intf/versions/0.0.8 +- SPDX conclusion: `Apache-2.0` +- Copyright/notice: https://components.espressif.com/components/espressif/esp_sccb_intf/versions/0.0.8/license + +## `espressif/dl_fft` + +- ESP-Mosaico version: `0.7.0` +- Source: https://components.espressif.com/components/espressif/dl_fft/versions/0.7.0 +- SPDX conclusion: `MIT` +- Copyright/notice: https://components.espressif.com/components/espressif/dl_fft/versions/0.7.0/license + +## `espressif/usb_host_uvc` + +- ESP-Mosaico version: `2.5.1` +- Source: https://components.espressif.com/components/espressif/usb_host_uvc/versions/2.5.1 +- SPDX conclusion: `Apache-2.0` +- Copyright/notice: https://components.espressif.com/components/espressif/usb_host_uvc/versions/2.5.1/license + +## `espressif/hand_detect` + +- ESP-Mosaico version: `0.2.0` +- Source: https://components.espressif.com/components/espressif/hand_detect/versions/0.2.0 +- SPDX conclusion: `MIT` +- Copyright/notice: https://components.espressif.com/components/espressif/hand_detect/versions/0.2.0/license + +## `espressif/hand_gesture_recognition` + +- ESP-Mosaico version: `0.2.0` +- Source: https://components.espressif.com/components/espressif/hand_gesture_recognition/versions/0.2.0 +- SPDX conclusion: `MIT` +- Copyright/notice: https://components.espressif.com/components/espressif/hand_gesture_recognition/versions/0.2.0/license diff --git a/bridge/src/quotaframe_bridge/sources/firmware_release.py b/bridge/src/quotaframe_bridge/sources/firmware_release.py index bbbe877..a749d0d 100644 --- a/bridge/src/quotaframe_bridge/sources/firmware_release.py +++ b/bridge/src/quotaframe_bridge/sources/firmware_release.py @@ -15,10 +15,11 @@ from quotaframe_bridge.protocol.validation import unique_object, validate_identifier from quotaframe_bridge.release_config import release_repository from quotaframe_bridge.versioning import SemVer, VersionError +from quotaframe_bridge.targets import TARGETS _SHA256 = re.compile(r"^[0-9a-f]{64}$") MAX_MANIFEST_BYTES = 64 * 1024 -MAX_IMAGE_BYTES = 4 * 1024 * 1024 +MAX_IMAGE_BYTES = max(target.ota_partition_bytes for target in TARGETS) class FirmwareReleaseError(ValueError): diff --git a/bridge/src/quotaframe_bridge/targets.py b/bridge/src/quotaframe_bridge/targets.py index 54a27be..822e091 100644 --- a/bridge/src/quotaframe_bridge/targets.py +++ b/bridge/src/quotaframe_bridge/targets.py @@ -102,7 +102,7 @@ def ota_transfer_bytes(self) -> int: full_image="mosaico_usage_panel_full.bin", release_stem="espressif-esp-mosaico", image_chip_id=32, - ota_partition_bytes=4 * 1024 * 1024, + ota_partition_bytes=7 * 1024 * 1024, test_apps=("firmware/targets/esp_mosaico/test_apps/logic",), web_flash=WebFlashDefinition( name="ESP-Mosaico", diff --git a/bridge/tests/test_firmware_release.py b/bridge/tests/test_firmware_release.py index 35b32d6..c9dd724 100644 --- a/bridge/tests/test_firmware_release.py +++ b/bridge/tests/test_firmware_release.py @@ -113,13 +113,24 @@ async def test_rejects_oversized_manifest_image_and_fetch_result(self) -> None: with self.assertRaises(FirmwareReleaseError): await source.download_image(manifest.for_target("m5sticks3")) - too_large = json.loads(self.manifest(size=4 * 1024 * 1024 + 1)) + too_large = json.loads(self.manifest(size=7 * 1024 * 1024 + 1)) source = FirmwareReleaseSource( fetcher=lambda _url, _limit: json.dumps(too_large).encode() ) with self.assertRaises(FirmwareReleaseError): await source.fetch_manifest(self.manifest_url) + async def test_accepts_mosaico_image_within_expanded_ota_slot(self) -> None: + source = FirmwareReleaseSource( + fetcher=lambda _url, _limit: self.manifest( + target="esp_mosaico", firmware_project="mosaico_usage_panel", + size=7 * 1024 * 1024, + ) + ) + manifest = await source.fetch_manifest(self.manifest_url) + image = manifest.for_target("esp_mosaico", "mosaico_usage_panel") + self.assertEqual(image.size, 7 * 1024 * 1024) + async def test_rejects_noncanonical_semver(self) -> None: for version in ("01.2.3", "1.02.3", "1.2.03", "1.2.3-alpha..1", "1.2.3-01"): with self.subTest(version=version): diff --git a/docs/design/amoled-ui.md b/docs/design/amoled-ui.md index 6356396..baa3325 100644 --- a/docs/design/amoled-ui.md +++ b/docs/design/amoled-ui.md @@ -72,18 +72,21 @@ Waveshare：屏幕边距 24px；卡片宽 432px；卡片内边距 20–22px； - **底部导航**：方屏为**纯文字标签 + 下划线指示**，不用色块、不用卡片底。34px 高的三等分标签条，标签 13px 600 大写、字距 0.06em；未选中 `#565D68`，选中 `#C7CCD4` 并在标签下方 7px 处加一条 26×2px（圆角 1px）`#C7CCD4` 短横。导航是次要信息，任何情况下都不能成为屏幕上最亮的元素。**下划线是三个标签共用的同一条**，翻页时以 180ms 缓出横向滑到新标签下方，而不是在旧位置熄灭、新位置点亮；页面本身仍是硬切换。 - **连接状态**：右上角一个 8px 状态色圆点 + 14px `#C7CCD4` 文字。不显示 `SECURE` 之类无操作价值的标签。 - **电池**：连接状态右侧，30×14 描边（1px `#565D68`）+ 2px 正极点，内含 4 段。**分级显示，不显示百分比**：FULL=4 段、HIGH=3 段、MID=2 段、LOW=1 段、无数据=0 段；点亮色 `#C7CCD4`，熄灭段 `#2C313B`；仅 LOW 时点亮段转 `#F5B84C`。**充电**：外框左侧 gap 5px 有 7×12 闪电槽，**始终占位**（不充电时留空，绝不收起，否则插拔 USB 会让 `LINKED` 平移）；充电时填 `#C7CCD4`，段数照常反映电量，不做动画；充电时 LOW 不再变黄（外部供电已接入，此时低电量告警不再提供操作价值），屏保 `LOW` 角标同时撤掉。USB 只是供电，不改变 `LINKED`，界面不出现 `USB` 标签。 +- **摄像头标识**（仅 Mosaico，实验性手势）：`CAMERA` 开关已保存为打开时，连接状态点左侧 10px 显示 26×16 实心相机剪影（顶部 8×4 取景凸起，bg 色镜头环内含 4px 镜头点），与状态点垂直居中。采集中（启动或识别）为 `text` 色；开关打开但采集暂停（设置、配对、OTA）为 `dim`；采集失败时为 `dim` 并叠加 `#C7CCD4` 斜杠（外圈 bg 色描边）。开关关闭时不占位。顶栏不显示手势文字提示，仅保留 OK / 点赞停留进度条。保持灰阶，不使用 good/warn/bad 色。屏保不显示。 +- **相机预览**（仅 Mosaico）：识别就绪且处于普通页面时，点击顶栏摄像头标识切换预览，再次点击关闭；每次启动默认隐藏，不持久化。浮层位于内容区右上角，距右侧 24px、距内容区顶部 8px；按画面比例显示，最长边 160px，不裁切、不拉伸。只显示相机图像及 2px `text` 色手部定位框，无手时隐藏定位框，不叠加检测数量、置信度或状态文字。开关只控制预览显示，手势识别由设置页 CAMERA 控制。时钟、设置、配对、OTA、相机暂停或故障时隐藏浮层；识别恢复且返回普通页面后恢复本次运行的预览选择。 - **时间**：顶栏左侧，22px 700，`text` 色，24 小时制不显示秒。页面归属由底部导航指示，无需重复标题。硬件无温湿度传感器，界面不出现环境信息。 - **屏保态（SCREENSAVER）**：空闲 5 分钟后进入，触摸屏幕或晃动（IMU）唤醒，回到进入前的页面。规则是**整体降一级**——时钟 `muted #8B93A1`（120px，居中 top 158）、日期一行 20px `dim`、顶栏与底部导航全部隐藏、卡片去掉 panel 底与描边（纯 `bg`，AMOLED 省电）。屏幕上不得出现 `text` 级亮度。底部 13px `#2C313B` 提示 `TAP OR SHAKE TO WAKE`，进入 3 秒后淡出。仅电池 LOW 时右上角显示降亮的 `LOW` + 电池图标（`#8A6A2E`），其余状态一概不显示。防烧屏：整块内容每 60 秒随机位移 ±10px。 ## 6. 本地显示设置 -微雪设备长按触屏约 0.8 秒进入全屏设置页。亮度范围 1–100%，调整时预览；自动进入时钟支持 OFF、1、5、10、30 分钟。SAVE 写入 NVS，CANCEL 恢复已保存值；首次启动使用 menuconfig 默认值。OFF 不影响主动切换时钟。设置页打开时保持唤醒，配对和 OTA 优先于设置并撤销未保存预览。 +微雪 AMOLED 2.16 与 ESP-Mosaico 长按触屏约 0.8 秒进入全屏设置页。亮度范围 1–100%，调整时预览；自动进入时钟支持 OFF、1、5、10、30 分钟。SAVE 写入 NVS，CANCEL 恢复已保存值；首次启动使用 menuconfig 默认值。OFF 不影响主动切换时钟。设置页打开时保持唤醒，配对和 OTA 优先于设置并撤销未保存预览。 -设置页采用 24px 边距的行式布局，两项设置以 1px `#1D2128` 分隔线分组，不增加卡片底色与描边。顶栏左侧显示 DISPLAY，右侧复用连接状态与电池；LINKED 保持绿色。设置操作优先保证近距离触控，不要求全部读数使用远距离展示字号。 +设置页采用 24px 边距的行式布局，设置项以 1px `#1D2128` 分隔线分组，不增加卡片底色与描边。顶栏左侧显示 DISPLAY，右侧复用连接状态与电池；LINKED 保持绿色。设置操作优先保证近距离触控，不要求全部读数使用远距离展示字号。 -- **亮度设置行**：复用现有 48px 数值字体；右侧加减按钮为 80×80px，间隔 8px。仅通过按钮调整，不显示不可操作的滑条。到达 1% 或 100% 时禁用对应按钮，边界不循环。 -- **有限选项选择**：AUTO CLOCK 下方标明 AFTER IDLE / MIN；OFF、1、5、10、30 五项为 80×64px，间隔 8px，一次点击选定。只有当前选项反色，不累计填充；OFF 不拼接 MIN。保留非标准 menuconfig 默认时长，未匹配预设时显示当前秒数，直到用户主动选择预设。 +- **亮度设置行**：复用现有 48px 数值字体；右侧加减按钮为 80×80px，Mosaico 为 80×72px，间隔 8px。仅通过按钮调整，不显示不可操作的滑条。到达 1% 或 100% 时禁用对应按钮，边界不循环。 +- **有限选项选择**：AUTO CLOCK 下方标明 AFTER IDLE / MIN；OFF、1、5、10、30 五项为 80×64px，Mosaico 为 80×52px，间隔 8px，一次点击选定。只有当前选项反色，不累计填充；OFF 不拼接 MIN。保留非标准 menuconfig 默认时长，未匹配预设时显示当前秒数，直到用户主动选择预设。 - **操作状态**：16px 圆角，可见边界与点击区域一致。选中项及可用 SAVE 使用 text 底、bg 字；未选项和 CANCEL 使用 panel 底、muted 字。按下时背景和边框产生灰阶反馈；禁用控件使用 panel 底、dim 字，不响应点击。不使用主题阴影或过渡。 +- **开关**（仅 Mosaico `CAMERA`）：独立一行，与上方以分隔线隔开；右侧 80×44px 胶囊开关，与加减按钮右对齐。关闭为 panel 底、muted 圆钮居左；打开为反色 text 底、bg 圆钮居右。与亮度、自动时钟一样只是预览：切换后启用 SAVE，SAVE 一并写入 NVS，CANCEL 恢复已保存值。 - **保存与取消**：底部按钮均为 206×64px，间隔 20px。未修改时禁用 SAVE，修改后启用；CANCEL 始终可用。保存成功后退出，失败时保留页面和预览，在按钮上方显示 SAVE FAILED - TRY AGAIN 并允许重试。不常驻显示持久化实现说明。 实机验证需覆盖最低亮度恢复操作、桌面支架姿态下的按钮命中、OFF、未修改／已修改以及保存失败状态；像素尺寸与桌面预览不能替代触控和 AMOLED 可读性验收。 diff --git a/docs/design/mosaico-gesture-input.md b/docs/design/mosaico-gesture-input.md new file mode 100644 index 0000000..c2ee205 --- /dev/null +++ b/docs/design/mosaico-gesture-input.md @@ -0,0 +1,82 @@ +# Mosaico 实验性手势输入 + +本功能使用左槽 CameraBoard 在设备本地识别手势，默认关闭。用户操作见[使用与安装](../validation/mosaico-gestures.md)，固定依赖与参考仓库见[摄像头依赖说明](../../firmware/targets/esp_mosaico/CAMERA_UPSTREAM.md)。 + +## 交互 + +| 当前页面 | 动作 | 行为 | +| --- | --- | --- | +| 用量页 | 任一方向挥手或左右招手 | 下一页，按总览 → Codex → Claude 循环 | +| 时钟 | 挥手或左右招手 | 只恢复原页面，本轮不翻页 | +| 用量页 | 稳定 OK 或点赞约 500 ms | 进入时钟并保留原页面 | +| 时钟 | OK 或点赞 | 不切换页面 | +| 设置、配对确认、OTA | 任意手势 | 不接受动作，不延后执行 | + +触摸滑动保留双向导航。摄像头唤醒依赖持续采集和识别，时钟屏保不是芯片深度睡眠。 + +设置页的 `CAMERA` 开关与亮度、自动时钟超时一并保存。调整开关只改变设置预览，`SAVE` 成功后才改变采集状态，`CANCEL` 恢复保存值。NVS 使用 `mosaico/settings` 单个整数保存三个字段；记录缺失或无效时关闭相机，亮度和自动时钟超时沿用旧版固件在 `display` 命名空间中保存的值，没有则使用默认值。 + +图像仅在本机处理，不上传或保存。点击摄像头标识切换预览，预览选择不持久化。界面尺寸与状态见[AMOLED 设计规范](amoled-ui.md)。 + +## 处理链路与所有权 + +```text +CameraBoard / esp_video + → 最新 UYVY 帧 + → 等比例 RGB888 转换、镜像与旋转 + → HandDetect → 单手轨迹 / HandGestureCls + → SwipeLeft / SwipeRight / SwipeUp / SwipeDown / Wave / EnterClock + → Mosaico 主循环检查上下文 + → 翻页 / 唤醒 / 进入时钟 +``` + +| 文件 | 职责 | +| --- | --- | +| `firmware/targets/esp_mosaico/main/gesture_camera.cpp` | 左槽引脚、V4L2 流、帧获取与归还、停止及资源释放 | +| `firmware/targets/esp_mosaico/main/gesture_image.cpp` | UYVY 转 RGB888、等比例缩放、镜像和旋转 | +| `firmware/targets/esp_mosaico/main/gesture_tracker.cpp` | 单手轨迹、招手、姿态停留与重复触发抑制 | +| `firmware/targets/esp_mosaico/main/gesture_input.cpp` | 视觉任务、模型生命周期、上下文和有界事件队列 | +| `firmware/targets/esp_mosaico/main/app_main.cpp` | 设置、页面、屏保和输入许可；消费手势并更新 UI | +| `firmware/targets/esp_mosaico/main/display_ui.cpp` | 摄像头标识、停留进度和预览；LVGL 访问受显示锁保护 | +| `firmware/targets/esp_mosaico/main/panel_settings.cpp` | 显示与相机设置的原子持久化 | + +视觉任务串行执行采集、检测、分类和轨迹更新。V4L2 帧转换后立即归还；模型不在主循环或 LVGL 锁内运行。预览回调从视觉任务调用，取得显示锁后复制像素；隐藏时跳过显示锁和绘制。 + +主循环拥有页面和屏保状态。事件队列容量为 1，满时丢弃，不阻塞推理。每个事件带时间戳和上下文代次；超过 300 ms 或代次不匹配时丢弃。触摸、按键、连接变化、页面、旋转和输入许可变化使旧轨迹失效。被接受的手势会保留自身的重复触发抑制状态。 + +关闭相机或进入受保护状态时，视觉任务完成当前帧后停止流并释放模型和缓冲。启动最多尝试三次；连续三次采集失败停用识别。普通故障可通过关闭并重新开启重试；无法安全释放驱动资源时，需重启设备。主循环的触摸、按键和 BLE 路径独立于视觉任务。 + +## 识别参数 + +以下为当前实验性实现的参数，集中定义于 `gesture_tracker.cpp`，不代表识别率或响应时间承诺。 + +- 单手检测置信度至少 0.65；位置归一化到当前屏幕坐标。多手、越界、尺寸异常和不连续观测不构成有效挥手。 +- 挥手至少跨越三帧、持续 150–900 ms；主轴净位移至少 25%，至少为另一轴最大偏移的两倍，且不小于主轴累计路径的 75%。 +- 招手由三段交替水平运动构成，每段至少为画面宽度的 12%。一次连续运动只发出一个动作；再次接受挥动前需要约 400 ms 静止，帧间位移不超过 3% 视为静止。挥手后 1.5 秒内的反向挥动作为收手忽略。 +- OK 和点赞使用分类器的 `ok`、`like` 类别，置信度至少 0.85，稳定保持约 500 ms。相邻帧位移超过 5% 时跳过分类，优先保留轨迹采样频率。 +- 可接受的观测间隔上限为 400 ms。短暂漏检保留候选，长间隔或位置跳变重新开始跟踪。 +- 初始化、外部输入和 OK 触发后，挥手与姿态识别要求连续约 300 ms 无手，并距上次动作至少 500 ms。招手按交替轨迹单独判定。姿态停留不反复触发时钟切换。 + +相机安装校正先水平镜像再顺时针旋转 90°，随后应用屏幕旋转的逆变换。OV3640 的 640×480 图像缩为 320×240，SC101IOT 的 1280×720 图像缩为 320×180，旋转后将实际尺寸传给模型和预览。 + +## 资源与存储 + +检测使用 ESPDet-Pico 224×224，分类使用 MobileNetV2 128×128。模型随代码内嵌在应用中，使用 7 MiB 双 OTA 槽，更新及回滚使用同一份代码和模型。组件版本与模型 SHA-256 见[依赖说明](../../firmware/targets/esp_mosaico/CAMERA_UPSTREAM.md)。 + +720p 双 UYVY 缓冲约占 3.52 MiB。启动预检查要求至少 9 MiB 连续空闲 PSRAM 和 96 KiB 内部内存，为模型及并行 UI 分配留出空间。ESP-DL 内部分配包含断言，预检查不构成运行时内存分配成功的保证。 + +LVGL 使用 CLIB 分配器，避免独立固定大小的字形内存池。日志记录采集转换、检测、分类、反馈耗时以及内存和栈余量；`mean_cycle_ms` 不包含反馈和每轮固定 20 ms 等待，不能直接换算为帧率。 + +## 开发检查 + +在仓库根目录使用目标指定的 SDK： + +```sh +eim --version +eim list +eim run "idf.py -C firmware/targets/esp_mosaico build" v6.1 +eim run "idf.py -C firmware/targets/esp_mosaico/test_apps/logic build" v6.1 +eim run "idf.py -C firmware/test_apps/panel_state build" v6.1 +``` + +Mosaico logic 包含轨迹、动作路由、图像坐标转换和设置持久化用例，共享 panel_state 覆盖屏保。构建测试镜像与运行用例是不同步骤，执行结果应对应具体源码和产物记录。仓库通用检查见[软件与发布验收](../validation/software-acceptance.md)。 diff --git a/docs/devices.md b/docs/devices.md index 5b330d4..3b2f597 100644 --- a/docs/devices.md +++ b/docs/devices.md @@ -8,6 +8,7 @@ | --- | --- | --- | --- | --- | | 屏幕 | 480×480 圆角方形 AMOLED | 135×240 LCD | 800×480 四灰阶墨水屏 | 480×480 AMOLED | | 翻页交互 | 底部触控标签与左右滑动 | 正面按键 | 三向拨轮 | 触控标签、左右滑动与 AI 键 | +| 摄像头手势（实验性） | — | — | — | 需左槽 CameraBoard，默认关闭 | | 用量页面 | 总览 / Codex / Claude | 总览 / Codex / Claude | 主页同屏显示 Codex 与 Claude，另有趋势页 | 总览 / Codex / Claude | | 断连保留最后用量 | ✅ | ✅ | ✅ | ✅ | | 电量与充电图标 | ✅ AXP2101 | ✅ M5PM1 | ✅ AXP2101 | ✅ BQ27220 | @@ -15,7 +16,7 @@ | 屏保用量细条 | ✅ | — | ✅ | ✅ | | 自动旋转 | ✅ 四向 | — 固定 0°/180°（menuconfig） | ✅ 四向 | ✅ 四向 | | 摇动 / 翻转唤醒 | ✅ | — | — | ✅ | -| 屏上设置页 | ✅ 亮度、自动时钟超时 | — 亮度仅 menuconfig | ✅ 刷新间隔、屏幕方向 | ✅ 亮度、自动时钟超时 | +| 屏上设置页 | ✅ 亮度、自动时钟超时 | — 亮度仅 menuconfig | ✅ 刷新间隔、屏幕方向 | ✅ 亮度、自动时钟超时、CAMERA 开关 | | 温湿度显示 | — | — | ✅ 时钟页 | — | | 用量趋势（48 点 / 30 分钟） | — | — | ✅ 需 FAT microSD 卡 | — | | 托盘/菜单栏左键切换时钟（`screen.toggle.v1`） | ✅ | — | ✅ | ✅ | @@ -32,6 +33,7 @@ RLCD 4.2 使用 400×300 黑白反射式 LCD，提供总览、Codex / Claude 详 ## 屏幕快捷操作 - **RLCD 4.2**：KEY 上一页，BOOT 下一页，长按 KEY 进入或退出设置；设置中 KEY 选择项目、BOOT 修改选项。KEY 可关闭当前用量告警。OTA 待确认时 KEY 确认，长按 KEY 拒绝。 -- **480×480 触屏设置（微雪 AMOLED 2.16 与 ESP-Mosaico）**：长按触屏约 0.8 秒进入，两块板共用同一套设置。亮度即时预览；AUTO CLOCK 可选择 OFF、1、5、10、30 分钟。SAVE 保存并在重启后保留，CANCEL 撤销未保存调整。OFF 仅关闭自动进入时钟，托盘/菜单栏左键仍可主动切换；微雪另有 PWR 侧键，ESP-Mosaico 另有 AI 键。 +- **480×480 触屏设置（微雪 AMOLED 2.16 与 ESP-Mosaico）**：长按触屏约 0.8 秒进入，两块板均支持亮度与自动时钟设置。亮度即时预览；AUTO CLOCK 可选择 OFF、1、5、10、30 分钟。SAVE 保存并在重启后保留，CANCEL 撤销未保存调整。OFF 仅关闭自动进入时钟，托盘/菜单栏左键仍可主动切换；微雪另有 PWR 侧键，ESP-Mosaico 另有 AI 键。 +- **ESP-Mosaico 实验性手势**：安装左槽 CameraBoard 后，在设置中打开 CAMERA 并保存。用量页向任意方向挥手或左右招手切到下一页；OK 或点赞稳定保持约半秒进入时钟，挥手或招手唤醒。点击顶栏摄像头标识切换预览。完整说明见[手势指南](validation/mosaico-gestures.md)。 - **Windows 托盘与 macOS 菜单栏**：左键切换支持设备的时钟；鼠标悬停在 Bridge 图标上滚动，上滚查看上一页，下滚查看下一页，页面集合与顺序由各设备决定。操作作用于所有已连接且支持该命令的设备；设备处于时钟时会先回到额度页面。快速连续滚动会限速，OTA 期间不执行翻页。 - 滚轮翻页要求设备通告 `screen.page.v1`；未通告的设备只接收其支持的命令。 diff --git a/docs/roadmap.md b/docs/roadmap.md index e7fd892..1a68ac6 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -18,7 +18,9 @@ | [软件与发布验收](validation/software-acceptance.md) | 测试入口与发布候选验收清单 | | [AMOLED 性能参考](validation/amoled-performance.md) | 内存、刷新耗时与测量条件 | | [ePaper 刷新参考](validation/epaper-refresh.md) | 局刷数据及光学验证边界 | +| [Mosaico 启动流程](validation/mosaico-boot.md) | 显示、连接与电量计初始化顺序 | | [接入架构](design/open-device-access.md) | 状态归属、连接与更新生命周期 | +| [Mosaico 实验性手势](design/mosaico-gesture-input.md) | 操作规则、处理链路与识别参数 | | [接入协议](../protocol/open-device-access.md) | 设备线上字段与校验约束 | | [硬件移植](PORTING.md) | 新目标目录与硬件验收 | | [设计语言](design/amoled-ui.md) | AMOLED 界面规范 | diff --git a/docs/validation/mosaico-boot.md b/docs/validation/mosaico-boot.md new file mode 100644 index 0000000..74c6954 --- /dev/null +++ b/docs/validation/mosaico-boot.md @@ -0,0 +1,24 @@ +# Mosaico 启动流程 + +## 显示与连接 + +屏幕初始化先建立共享 I2C 总线和供电。CO5300 亮度寄存器以零启动，`bsp_display_new()` 设置方向后发出 Display On；首次刷新完成后，界面通过 120 ms 亮度过渡显示画面，避免显示未初始化的帧存。 + +屏幕就绪后创建电量计初始化任务，随后启动 BLE 广播，再初始化 BMI270 和按键。外设初始化期间收到的 BLE 事件进入队列，由主循环处理；Bridge 的实际连接时刻还取决于主机扫描。 + +## 电量计 + +BQ27220 在独立任务中核对配置，必要时写入配置，再封存并确认状态。`bsp_battery_init()` 复用驱动已完成的封存，不重复执行。任务通过原子就绪标志将电量计交给主循环读取；就绪前不显示有效电量，OTA 电源判定保持未就绪。 + +驱动的 `CONFIG_BQ27220_SEAL_SETTLE_MS` 默认值为 2000 ms，Mosaico 配置为 200 ms。等待结束后驱动仍读取状态确认封存；确认失败时初始化失败，不发布有效电量。该值是启动流程的实验性配置，不是芯片时序保证。 + +电池图形创建时按无数据状态着色，主循环获得读数后更新，避免第一帧被误认为满电。电量计配置重写会恢复配置中的默认容量，不能将此过程视为保留电量计学习结果。 + +## 维护入口 + +- [app_main.cpp](../../firmware/targets/esp_mosaico/main/app_main.cpp)：初始化顺序、后台电量计任务和主循环轮询。 +- [display_ui.cpp](../../firmware/targets/esp_mosaico/main/display_ui.cpp)：首次刷新与亮度过渡。 +- [sdkconfig.defaults](../../firmware/targets/esp_mosaico/sdkconfig.defaults)：封存等待及目标配置。 +- [BSP 本地修改](../../firmware/components/esp-mosaico-bsp/UPSTREAM.md)：显示启动序列和电量计集成约束。 + +启动耗时取决于电量计是否重写配置、外设状态和 Bridge 连接时机。比较启动日志时应使用相同硬件、配置和供电条件，并分别记录显示就绪、广播、主循环和数据到达时刻。 diff --git a/docs/validation/mosaico-gestures.md b/docs/validation/mosaico-gestures.md new file mode 100644 index 0000000..dfae811 --- /dev/null +++ b/docs/validation/mosaico-gestures.md @@ -0,0 +1,51 @@ +# Mosaico 实验性手势：使用与安装 + +本地摄像头手势是**实验性功能**，默认关闭。需要左槽 CameraBoard，支持 OV3640 和 SC101IOT，固件自动识别传感器。图像只在设备内处理，不上传、不保存。识别效果受光线、距离和手部是否完整入镜影响。 + +## 开启与操作 + +安装相机板前先断电。开机后长按触屏进入设置，打开 `CAMERA` 并点击 `SAVE`。关闭该开关并保存即可停止采集；`CANCEL` 撤销未保存的调整。 + +| 页面 | 动作 | 效果 | +| --- | --- | --- | +| 用量页 | 向任意方向挥手，或左右招手 | 切到下一页，按总览 → Codex → Claude 循环 | +| 用量页 | 👌 OK 或 👍 点赞稳定保持约半秒 | 进入时钟 | +| 时钟 | 挥手或左右招手 | 恢复进入时钟前的页面 | + +一段连续挥动只执行一次，手停稳后再做下一次动作。触摸、AI 键和 Bridge 操作继续可用；设置、配对确认和 OTA 期间暂停手势输入。 + +## 摄像头标识与预览 + +打开 `CAMERA` 后，顶栏显示摄像头标识：采集中为亮色，暂停时变暗，失败时带斜杠。关闭并保存后标识消失。 + +在普通用量页点击摄像头标识可显示或隐藏预览。预览保留画面比例，显示图像和手部定位框，可用于调整摆放位置。预览默认隐藏，重启后恢复隐藏；关闭预览不影响识别。时钟、设置、配对和 OTA 界面隐藏预览，返回普通页面后恢复本次选择。 + +标识带斜杠时，断电检查左槽相机连接，再开机重试。也可关闭 `CAMERA` 并保存，再打开并保存；仍无法启动时重启设备。屏保期间摄像头继续工作，开启手势会增加耗电。 + +## 构建 + +在仓库根目录运行： + +```sh +eim --version +eim list +eim run "idf.py -C firmware/targets/esp_mosaico build" v6.1 +``` + +构建产物为 `firmware/targets/esp_mosaico/build/mosaico_usage_panel.bin`。生成配置应启用 CLIB 分配器、OV3640 和 SC101IOT；摄像头选项位于 `menuconfig → Camera gestures`。 + +16 MiB Flash 使用两个 7 MiB 应用槽，模型与代码共同更新。NVS 位于 `0x9000`，ota_0 位于 `0x10000`，otadata 位于 `0x810000`，ota_1 位于 `0x820000`。Folder Push 限额为 7 MiB + 512 字节。 + +## 有线烧录 + +按住 BOOT 再开机进入 ROM 下载模式。整段擦写大应用可能断连，应用按 1 MiB 分段烧录，不直接使用 `idf.py flash` 或未拆分的 `@flash_args`。 + +地址从目标应用槽起点逐段递增。使用 ESP-IDF v6.1 环境中的 esptool，参数为 `--chip esp32s31 --no-stub`、460800 波特率、DIO/80 MHz/16 MB，并指定 `--after no-reset`。烧录前核对分段拼接内容与构建应用一致，烧录后确认每段设备哈希校验成功，再通过 POWER 关机重启。 + +仅更新当前应用时，不写入 NVS、分区表或初始 OTA 元数据。完整安装使用构建产物匹配的 bootloader、分区表及初始化数据，应用仍分段写入。整片擦除或写入包含填充区域的合并镜像会影响已保存的配对和设置。 + +## 维护入口 + +应用日志从 UART0 输出：右侧排针 H1 的 15 脚 TX0 / GPIO58，或底板背面 TX 调试焊盘，115200 波特率。Type-C 口使用 USB OTG 下载，摄像头占用的 USB Serial/JTAG 引脚位于左侧排针。日志包含手部坐标、置信度、动作和运行耗时，不包含图像。 + +代码结构、识别参数见[手势设计](../design/mosaico-gesture-input.md)；固定依赖、模型摘要和引脚所有权见[摄像头依赖说明](../../firmware/targets/esp_mosaico/CAMERA_UPSTREAM.md)。 diff --git a/docs/verification.md b/docs/verification.md index 4af5683..dc2737d 100644 --- a/docs/verification.md +++ b/docs/verification.md @@ -9,7 +9,7 @@ | M5StickS3 | 连接、用量显示、页面切换；OTA 接收超时修复后连续 5 次正常升级；电池供电下电源门槛拒绝 | 40% 电量边界、配置变体、故障注入与长期稳定性 | | Waveshare AMOLED 2.16 | 连接、用量显示、页面与时钟切换、正常 OTA；电池供电下电源门槛拒绝 | 40% 边界、RTC 断电保持、唤醒配置、电池与长期屏保 | | Waveshare ePaper 3.97 | 连接、用量显示、页面与时钟切换、本地工具正常 OTA | 多日残影、不同温度/方向、SD 读写与断电恢复 | -| ESP-Mosaico | 连接、用量显示、页面与时钟切换、本地工具正常 OTA；拔除外部电源后拒绝更新 | 电量边界、插电识别延迟、续航与故障注入 | +| ESP-Mosaico | 连接、用量显示、页面与时钟切换、本地工具正常 OTA；拔除外部电源后拒绝更新；实验性摄像头手势 | 电量边界、插电识别延迟、续航与故障注入 | 以上基础功能和正常 OTA 覆盖来自 2026-09 的开发设备测试，未固定同一套发布候选资产。脚本 OTA 证明设备升级路径，不涵盖安装版 Bridge 的下载、桌面确认和独占连接调度。 diff --git a/firmware/components/esp-mosaico-bsp/UPSTREAM.md b/firmware/components/esp-mosaico-bsp/UPSTREAM.md index eea9067..196ad2b 100644 --- a/firmware/components/esp-mosaico-bsp/UPSTREAM.md +++ b/firmware/components/esp-mosaico-bsp/UPSTREAM.md @@ -28,5 +28,17 @@ and two-pixel vertical boundaries, clipped to the display dimensions. The CO5300 RASET command requires an even start row and an even row count (datasheet page 161); LVGL renders the expanded rows so the pixel buffer matches the transfer window. +The CO5300 initialization sequence sets brightness register 0x51 to zero instead of +0xFF and omits the Display On command with its 600 ms delay. `bsp_display_new()` +issues Display On after the orientation is set. The panel therefore stays dark until +the application raises the brightness, which it must do after the first frame has been +written; upstream shows undefined frame memory at full brightness during that delay. + +`bsp_battery_init()` does not seal the BQ27220 again after `bq27220_create()`. The +driver seals and verifies on every path that returns a handle, and each seal waits +`CONFIG_BQ27220_SEAL_SETTLE_MS` unconditionally, so the upstream call only added +that wait. The driver default is 2000 ms; this target selects 200 ms in +`sdkconfig.defaults` and retains the sealed-state check after the wait. + When re-syncing, preserve these integration changes as well as the manifest constraint and review the repository diff for other board-specific changes. diff --git a/firmware/components/esp-mosaico-bsp/onboard/battery.c b/firmware/components/esp-mosaico-bsp/onboard/battery.c index d6c0fe7..027fb41 100644 --- a/firmware/components/esp-mosaico-bsp/onboard/battery.c +++ b/firmware/components/esp-mosaico-bsp/onboard/battery.c @@ -86,14 +86,8 @@ static esp_err_t battery_attach(void) .cedv = &s_default_cedv, }; s_gauge = bq27220_create(&config); + /* bq27220_create() returns a handle only after sealing the gauge and confirming it. */ ESP_RETURN_ON_FALSE(s_gauge, ESP_FAIL, TAG, "attach BQ27220 0x%02X failed", BSP_BATTERY_I2C_ADDR); - const esp_err_t seal_ret = bq27220_seal(s_gauge); - if (seal_ret != ESP_OK) { - ESP_LOGE(TAG, "seal BQ27220 after initialization failed: %s", esp_err_to_name(seal_ret)); - bq27220_delete(s_gauge); - s_gauge = NULL; - return seal_ret; - } const uint16_t voltage_mv = bq27220_get_voltage(s_gauge); ESP_LOGI(TAG, "BQ27220 online: address=0x%02X SDA=%d SCL=%d %u mV", BSP_BATTERY_I2C_ADDR, diff --git a/firmware/components/esp-mosaico-bsp/onboard/display.c b/firmware/components/esp-mosaico-bsp/onboard/display.c index ffda2a4..2f544d4 100644 --- a/firmware/components/esp-mosaico-bsp/onboard/display.c +++ b/firmware/components/esp-mosaico-bsp/onboard/display.c @@ -130,11 +130,13 @@ static const co5300_lcd_init_cmd_t s_vendor_init[] = { {0x35, (uint8_t[]){0x00}, 1, 0}, #endif {0x53, (uint8_t[]){0x20}, 1, 0}, - {0x51, (uint8_t[]){0xFF}, 1, 0}, + /* Frame memory is undefined until the first flush, so the panel starts at zero + * brightness; the application raises it once a frame has been written. Display On + * is issued by bsp_display_new() after the orientation is set. */ + {0x51, (uint8_t[]){0x00}, 1, 0}, {0x63, (uint8_t[]){0xFF}, 1, 0}, {0x2A, (uint8_t[]){0x00, 0x00, 0x01, 0xDF}, 4, 0}, {0x2B, (uint8_t[]){0x00, 0x00, 0x01, 0xDF}, 4, 0}, - {0x29, NULL, 0, 600}, }; /* diff --git a/firmware/components/esp_desktop_buddy_folder_push/Kconfig b/firmware/components/esp_desktop_buddy_folder_push/Kconfig index 51a2a2b..24a8980 100644 --- a/firmware/components/esp_desktop_buddy_folder_push/Kconfig +++ b/firmware/components/esp_desktop_buddy_folder_push/Kconfig @@ -23,7 +23,7 @@ config ESP_DESKTOP_BUDDY_FOLDER_PUSH_MAX_DECODED_CHUNK config ESP_DESKTOP_BUDDY_FOLDER_PUSH_MAX_TRANSFER_BYTES int "Max transfer size bytes" - range 0 4194816 + range 0 7340544 default 4194816 help Maximum total decoded payload bytes accepted by a single `char_begin` diff --git a/firmware/components/esp_desktop_buddy_folder_push/UPSTREAM.md b/firmware/components/esp_desktop_buddy_folder_push/UPSTREAM.md index 00e9541..26eb94e 100644 --- a/firmware/components/esp_desktop_buddy_folder_push/UPSTREAM.md +++ b/firmware/components/esp_desktop_buddy_folder_push/UPSTREAM.md @@ -8,3 +8,7 @@ This component is a vendored source snapshot from: - License: Apache-2.0 (see the SPDX headers in the source files) The upstream component manifest uses a sibling `path` dependency on `../esp_desktop_buddy`. ESP-IDF Component Manager does not include that sibling when it downloads a Git subdirectory component, so the pinned source is kept locally while `esp_desktop_buddy` itself remains a managed Git dependency. + +The transfer-size Kconfig range permits up to 7 MiB plus 512 manifest bytes for +the Mosaico gesture firmware. The default remains 4 MiB plus 512 bytes; each +target selects its limit, and the OTA backend independently checks slot capacity. diff --git a/firmware/components/usage_panel_state/include/usage_panel_state/screensaver_state.hpp b/firmware/components/usage_panel_state/include/usage_panel_state/screensaver_state.hpp index c683598..89750de 100644 --- a/firmware/components/usage_panel_state/include/usage_panel_state/screensaver_state.hpp +++ b/firmware/components/usage_panel_state/include/usage_panel_state/screensaver_state.hpp @@ -35,6 +35,8 @@ class ScreensaverController { uint64_t idle_delay_ms = kDefaultIdleDelayMs); void reset(uint64_t now_ms, Page page); + /** Enters the clock once; repeated requests preserve the original restore page. */ + void enter_clock(uint64_t now_ms, Page current_page); /** Zero disables automatic entry; explicit toggles remain available. */ void set_idle_delay(uint64_t delay_ms) { idle_delay_ms_ = delay_ms; } void note_touch_down(uint64_t now_ms); diff --git a/firmware/components/usage_panel_state/screensaver_state.cpp b/firmware/components/usage_panel_state/screensaver_state.cpp index 9eb120e..7009b47 100644 --- a/firmware/components/usage_panel_state/screensaver_state.cpp +++ b/firmware/components/usage_panel_state/screensaver_state.cpp @@ -56,6 +56,11 @@ void ScreensaverController::enter(uint64_t now_ms, Page current_page) ignore_wake_until_ms_ = 0; } +void ScreensaverController::enter_clock(uint64_t now_ms, Page current_page) +{ + if (!active_) enter(now_ms, current_page); +} + void ScreensaverController::exit(uint64_t now_ms) { active_ = false; diff --git a/firmware/targets/esp_mosaico/CAMERA_UPSTREAM.md b/firmware/targets/esp_mosaico/CAMERA_UPSTREAM.md new file mode 100644 index 0000000..766fac5 --- /dev/null +++ b/firmware/targets/esp_mosaico/CAMERA_UPSTREAM.md @@ -0,0 +1,98 @@ +# Experimental camera gestures: dependencies + +## Repository reference map + +Start here when continuing camera or gesture development. Repository default +branches are browsing references; reproduce this integration using the pinned +commits below and [dependencies.lock](dependencies.lock), not the latest branch. + +| Repository | What to reference | Integration baseline | +| --- | --- | --- | +| [esp-mosaico-bsp](https://github.com/esp-mosaico/esp-mosaico-bsp) | Board power, shared I2C, expansion slots and display ownership | Commit `392860b1d1a123c3377947074b2af1f600e86c5d`; preserve the local changes in [BSP UPSTREAM.md](../../components/esp-mosaico-bsp/UPSTREAM.md) | +| [esp-mosaico-claw upstream](https://github.com/esp-mosaico/esp-mosaico-claw) / [project fork](https://github.com/eMUQI/esp-mosaico-claw) | Left-slot OV3640 wiring and USB Serial/JTAG pad arbitration | [Committed subboard source](https://github.com/eMUQI/esp-mosaico-claw/blob/903fe11ec3ca4a8881afef104e53a8da040896ac/components/subboard_support/subboard.c); local uncommitted camera changes were not imported | +| [esp-video-components](https://github.com/espressif/esp-video-components) | `esp_video` DVP/V4L2 capture, buffer lifetime and `esp_cam_sensor` OV3640 modes | Registry packages `esp_video` 2.5.0 and `esp_cam_sensor` 2.6.0 | +| [ESP-DL](https://github.com/espressif/esp-dl) | [HandDetect](https://github.com/espressif/esp-dl/tree/master/models/hand_detect), [HandGestureRecognizer](https://github.com/espressif/esp-dl/tree/master/models/hand_gesture_recognition), preprocessing and model selection | Registry packages `esp-dl` 3.3.11, `hand_detect` 0.2.0 and `hand_gesture_recognition` 0.2.0; model digests below | +| [ESP-VISION](https://github.com/espressif/esp-vision) | Mosaico board port and Camera → Image → AI → Display pipeline patterns | Architecture reference only; not a dependency and no source snapshot imported | +| [ESP-WHO](https://github.com/espressif/esp-who) | Capture/inference task separation and visual application lifecycle | Architecture reference only; not the starting point for this Mosaico target | + +## Continuing this implementation + +- Read the [interaction design](../../../docs/design/mosaico-gesture-input.md) for page cycling, wake-only behavior, OK / thumbs-up holds and protected states. +- Start in [gesture_camera.cpp](main/gesture_camera.cpp) for capture/pin ownership, [gesture_image.cpp](main/gesture_image.cpp) for image orientation, [gesture_tracker.cpp](main/gesture_tracker.cpp) for temporal recognition, and [gesture_input.cpp](main/gesture_input.cpp) for the worker lifecycle. [app_main.cpp](main/app_main.cpp) owns action routing and UI state. +- Follow the [usage and installation guide](../../../docs/validation/mosaico-gestures.md). Camera gestures are experimental and disabled by default. +- Preserve the 7 MiB dual-slot layout, default-off capture setting, local-only image processing and existing BSP display fixes. Recheck the current tree and lockfile before changing dependencies; do not edit generated `managed_components` as the source of truth. + +## Locked component versions + +The target uses registry components pinned by `dependencies.lock`: + +| Component | Version | Role | +| --- | --- | --- | +| espressif/esp_video | 2.5.0 | DVP/V4L2 capture and buffer lifetime | +| espressif/esp_cam_sensor | 2.6.0 | OV3640 and SC101IOT sensor configuration | +| espressif/esp-dl | 3.3.11 | S31 inference and preprocessing | +| espressif/hand_detect | 0.2.0 | ESPDet-Pico 224×224 hand detection | +| espressif/hand_gesture_recognition | 0.2.0 | MobileNetV2 128×128 classification; `ok` and `like` commands | + +The published model components explicitly select their `models/p4` resources +for `esp32s31`. Both models are embedded in the application so A/B update and +rollback keep model weights and code together. Only RGB888 conversions are +enabled in ESP-DL; camera UYVY conversion and geometric mapping live in +`main/gesture_image.cpp`. No managed component is modified. + +Left-slot wiring and USB Serial/JTAG pad arbitration in +`main/gesture_camera.cpp` are adapted from `components/subboard_support/subboard.c` +in the esp-mosaico-claw source snapshot +`903fe11ec3ca4a8881afef104e53a8da040896ac` (Apache-2.0, Espressif Systems). +The pin assignments were read from the committed snapshot, not its working-tree +edits. This integration does not import the module manager, board manager, Lua, +or the esp-claw video wrapper. Preserve the copyright notice in the adapted file. + +The camera uses its onboard oscillator and the existing BSP subboard I2C bus. +GPIO14 is D4 during capture; GPIO33 is D2 and cannot concurrently serve USB +Serial/JTAG. GPIO34 is the active-low illuminator and remains high. UART logging +is retained. Start/stop and frame ownership belong to the vision worker; the +application task exchanges context and bounded events only. + +Start-up timing follows `boards/ESP32_S31_MOSAICO/camera.c` in +[esp-vision](https://github.com/espressif/esp-vision) and `mosaico_module_camera` +in [esp-mosaico-bsp](https://github.com/esp-mosaico/esp-mosaico-bsp): 20 ms of +stabilization after the slot is claimed, a 500 ms frame timeout, up to three +start attempts 100 ms apart, and one discarded frame per buffer after stream +start. Frames flagged with `V4L2_BUF_FLAG_ERROR` are requeued unconverted. +SCCB runs at 100 kHz on the BSP subboard I2C1 bus, matching +`mosaico_module_camera`; that bus has only internal pull-ups. CameraBoards are +fitted with either an OV3640 (SCCB 0x3c) or an SC101IOT (0x68); both are built +with auto-detection, as in `mosaico_module_camera`. OV3640 uses 640x480 UYVY; +SC101IOT uses its full 1280x720 UYVY frame at the driver's nominal 15 fps. +The SC101IOT VGA register table crops the center of the 1280x720 array, so +720p retains more of the scene for close-range gestures. Format negotiation +follows `mosaico_module_camera`: read the configured sensor size with +`VIDIOC_G_FMT`, request UYVY with `VIDIOC_S_FMT`, and verify the returned size. +No sensor register table or board timing is modified. + +The complete frame is fitted within 320x240 while preserving its aspect ratio: +SC101IOT produces 320x180 RGB888 before rotation, and OV3640 produces 320x240. +The model and debug preview receive the actual rotated dimensions. The two +720p UYVY capture buffers require about 3.52 MiB; the startup precheck requires +at least a 9 MiB contiguous free PSRAM block for capture, model arenas and UI load. +The configured sensor frame rate differs from the end-to-end recognition rate, +which also includes conversion, inference and scheduling. + +`CONFIG_MOSAICO_CAMERA_ROTATION` applies the mounting correction after the +horizontal mirror; the inverse of the actual applied display rotation follows +it. A context change discards any unfinished gesture. Both reference pipelines +rotate the raw frame 90 degrees counter-clockwise (`PPA_SRM_ROTATION_ANGLE_90`) +without mirroring to make it model-upright. Mirroring first keeps swipe +directions in the user's view, so the equivalent default is 90 degrees +clockwise with mirroring enabled. + +Package copyright and license sources are recorded in +[THIRD_PARTY_LICENSES.md](../../../THIRD_PARTY_LICENSES.md). + +## Embedded model digests + +SHA-256 of the selected published S31/P4 model files: + +- `espdet_pico_224_224_hand.espdl` (576944 bytes): `a8d1857bab33bfd23bc72e1543a156ef33645b47b9ac130776aacb6a215439ea` +- `mobilenetv2_0_5_128_128_gesture.espdl` (935136 bytes): `fc4c4e0b4ba52e651dd802d5e0dc738f9eac899b107a9ce4638635e746be3c5d` diff --git a/firmware/targets/esp_mosaico/README.md b/firmware/targets/esp_mosaico/README.md index a123257..1858f20 100644 --- a/firmware/targets/esp_mosaico/README.md +++ b/firmware/targets/esp_mosaico/README.md @@ -25,7 +25,7 @@ eim run "idf.py -C firmware/targets/esp_mosaico --preview set-target esp32s31" v eim run "idf.py -C firmware/targets/esp_mosaico build" v6.1 ``` -构建产物为 `build/mosaico_usage_panel.bin`，两个 app 分区均为 4 MiB，Folder Push 总传输限额为 4 MiB + 512 字节，额外空间用于 manifest。 +构建产物为 `build/mosaico_usage_panel.bin`，两个 app 分区均为 7 MiB，Folder Push 总传输限额为 7 MiB + 512 字节，额外空间用于 manifest。 ## 当前实现 @@ -36,6 +36,19 @@ eim run "idf.py -C firmware/targets/esp_mosaico build" v6.1 - `usage_core` / `usage_protocol` / `usage_ble` / `usage_ota` 与其他目标共用，BLE 广播前缀 `QF-Mosaico-`，状态身份 `ESP-Mosaico`，型号标识 `esp_mosaico`； - 触控标签与左右滑动切页之外，板载 AI 键单击可唤醒并切换到下一页。 +### 实验性本地手势 + +左槽 CameraBoard（OV3640 或 SC101IOT）支持本地手势，默认关闭。在设置中打开 `CAMERA` 并点击 `SAVE` 后启用，关闭并保存即可停止采集。图像只在设备内处理，不上传、不保存；识别效果受光线、距离和手部是否完整入镜影响。 + +- 用量页：向任意方向挥手或左右招手切到下一页，按总览 → Codex → Claude 循环。 +- 进入时钟：👌 OK 或 👍 点赞稳定保持约半秒。 +- 唤醒：在时钟页面挥手或左右招手，恢复原页面。 +- 预览：点击顶栏摄像头标识显示或隐藏画面与手部定位框；默认隐藏，重启后恢复隐藏。关闭预览不影响识别。 + +一段连续挥动只执行一次，手停稳后再做下一次动作。设置、配对和 OTA 期间暂停手势输入，隐藏预览。触摸和 AI 键继续可用；时钟中摄像头仍工作，开启手势会增加耗电。 + +亮度、自动时钟超时和相机开关作为一条 NVS 记录保存；记录缺失或无效时关闭相机，亮度和自动时钟超时沿用旧版固件保存的值，没有则使用默认值。模型随代码内嵌应用，在当前双槽分区中共同更新。使用、安装和故障处理见[手势指南](../../../docs/validation/mosaico-gestures.md)，实现结构见[手势设计](../../../docs/design/mosaico-gesture-input.md)，固定版本、参考仓库和模型摘要见 [CAMERA_UPSTREAM.md](CAMERA_UPSTREAM.md)。 + ### 显示刷新 默认保留 CO5300 TE 输出，使用局部刷新和单个 PSRAM 绘图缓冲。BSP 在每轮 @@ -46,7 +59,7 @@ TE 信号缺失时最多等待 25 ms 后继续刷新，并在超时、恢复状 启动日志应包含 `partial refresh: PSRAM single buffer`。硬件验收应比较导航条、 用量条动画和整页切换的帧间隔，并检查四个旋转方向、休眠唤醒后的撕裂与触控。 40 MHz 四线 QSPI 传输整屏 RGB565 的理论下限约为 23 ms；整页更新仍可能跨越 -屏幕扫描周期，按帧 TE 同步不等于所有更新均无撕裂。此刷新路径尚待实机验收。 +屏幕扫描周期，按帧 TE 同步不等于所有更新均无撕裂。 ### 与 AMOLED 目标的板级差异 @@ -66,7 +79,7 @@ TE 信号缺失时最多等待 25 ms 后继续刷新，并在超时、恢复状 eim run "idf.py -C firmware/targets/esp_mosaico/test_apps/logic build" v6.1 ``` -只覆盖本目标独有的 `local_clock` 和 `orientation`。共享面板行为由 `firmware/test_apps/panel_state` 覆盖。没有实机时该命令只证明测试镜像编译通过。 +覆盖本目标独有的 `local_clock`、`orientation`、手势轨迹/动作路由、图像坐标转换和设置持久化。共享面板行为由 `firmware/test_apps/panel_state` 覆盖。没有实机时该命令只证明测试镜像编译通过。 ## 后续步骤 diff --git a/firmware/targets/esp_mosaico/dependencies.lock b/firmware/targets/esp_mosaico/dependencies.lock index 80c305b..ef2fab9 100644 --- a/firmware/targets/esp_mosaico/dependencies.lock +++ b/firmware/targets/esp_mosaico/dependencies.lock @@ -74,12 +74,76 @@ dependencies: type: service version: 0.5.3 espressif/dhara: - component_hash: 88968cff2e88853a29366ad556e9c27596d18e15028b6e9f3cd627665f914440 + component_hash: 46b686fd0599aed21fee222c249744f05eed28d8e62446bcc83b6b50bcc390e0 dependencies: [] source: registry_url: https://components.espressif.com type: service - version: 1.0.0 + version: 1.1.0 + espressif/dl_fft: + component_hash: 3e8f7d34e04bfa6333f289a6d126745bebe0cee9311b81a6a5d106047f63ef65 + dependencies: + - name: idf + require: private + version: '>=5.0' + source: + registry_url: https://components.espressif.com + type: service + version: 0.7.0 + espressif/esp-dl: + component_hash: 233ee8d37dfedeb637fc74713b87bd861ac530cfbcc4c63c57c2e51064cdda31 + dependencies: + - name: espressif/dl_fft + registry_url: https://components.espressif.com + require: private + version: '>=0.6.0' + - name: espressif/esp_new_jpeg + registry_url: https://components.espressif.com + require: private + version: ^1 + - name: idf + require: private + version: '*' + source: + registry_url: https://components.espressif.com + type: service + targets: + - esp32 + - esp32c2 + - esp32c3 + - esp32c5 + - esp32c6 + - esp32p4 + - esp32s2 + - esp32s3 + - esp32s31 + version: 3.3.11 + espressif/esp_cam_sensor: + component_hash: 0dd6634a71cc8756f0e46c961476304a23a82db4f4c4deb8aecfa30161d2185c + dependencies: + - name: espressif/cmake_utilities + registry_url: https://components.espressif.com + require: private + version: 0.* + - name: espressif/esp_sccb_intf + registry_url: https://components.espressif.com + require: private + version: '>=0.0.5' + - name: idf + require: private + version: '>=5.3' + source: + registry_url: https://components.espressif.com + type: service + targets: + - esp32p4 + - esp32s3 + - esp32c3 + - esp32c5 + - esp32c6 + - esp32c61 + - esp32s31 + version: 2.6.0 espressif/esp_codec_dev: component_hash: 4779f31a3ba3f9b38aee1afe6ccb9ba8f6108dc3a6e77d39d7d714e9a215371b dependencies: @@ -259,6 +323,62 @@ dependencies: - esp32c61 - esp32h4 version: 1.0.2 + espressif/esp_sccb_intf: + component_hash: c071b189e49f40940722aea01a5489f873385cc39cd5b14012341135b85d1a9d + dependencies: + - name: idf + require: private + version: '>=5.3' + source: + registry_url: https://components.espressif.com + type: service + version: 0.0.8 + espressif/esp_video: + component_hash: 5694b16c12b8bb26b56ad3f999ad10119f1314d64b2d1c518bc7dfed902df9e1 + dependencies: + - name: espressif/cmake_utilities + registry_url: https://components.espressif.com + require: private + version: 0.* + - name: espressif/esp_cam_sensor + registry_url: https://components.espressif.com + require: private + version: 2.6.* + - matches: + - if: $CONFIG{ESP_VIDEO_USE_CUSTOMIZED_ESP_H264_VERSION} == False + name: espressif/esp_h264 + registry_url: https://components.espressif.com + require: private + rules: + - if: target in [esp32p4] + version: 1.4.* + - name: espressif/esp_ipa + registry_url: https://components.espressif.com + require: private + rules: + - if: target in [esp32p4] + version: 2.4.* + - name: idf + require: private + version: '>=5.4' + - name: espressif/usb_host_uvc + registry_url: https://components.espressif.com + require: private + rules: + - if: target in [esp32p4, esp32s3, esp32s31] + version: 2.5.* + source: + registry_url: https://components.espressif.com/ + type: service + targets: + - esp32p4 + - esp32s3 + - esp32c3 + - esp32c5 + - esp32c6 + - esp32c61 + - esp32s31 + version: 2.5.0 espressif/freetype: component_hash: aa9f7657a69b2b1c1f49044017fc424081d4e0fe6e97834cb43883548d44e3ca dependencies: @@ -269,6 +389,28 @@ dependencies: registry_url: https://components.espressif.com type: service version: 2.14.3~1 + espressif/hand_detect: + component_hash: 82b537405e093703864bd3ec7af31c3bfbc3d76cff4cf0d9e618ab7889af482c + dependencies: + - name: espressif/esp-dl + registry_url: https://components.espressif.com + require: private + version: ~3.3.0 + source: + registry_url: https://components.espressif.com/ + type: service + version: 0.2.0 + espressif/hand_gesture_recognition: + component_hash: 99389204096e649fe10e5a28e4104e814975f8a27a046b6b3c52dfc5ede1acc4 + dependencies: + - name: espressif/hand_detect + registry_url: https://components.espressif.com + require: private + version: ~0.2.0 + source: + registry_url: https://components.espressif.com/ + type: service + version: 0.2.0 espressif/i2c_bus: component_hash: 483112c1619c54c38d0b734fb25f1678733267858a77b7af2f48cc7db6dbbd84 dependencies: @@ -325,6 +467,47 @@ dependencies: registry_url: https://components.espressif.com/ type: service version: 1.4.4 + espressif/usb: + component_hash: 6c2cff118141dbbca181d0d27d71801fdddfc78f7ac03ae43f6e5a8cbd89cb99 + dependencies: + - name: idf + require: private + version: '>=5.5.3' + source: + registry_url: https://components.espressif.com + type: service + targets: + - esp32s2 + - esp32s3 + - esp32s31 + - esp32p4 + - esp32h4 + - linux + version: 1.5.0 + espressif/usb_host_uvc: + component_hash: 3dde9c89980212de1e394a29d3fe782a2988a08fd2a16af34eee6f642aab6ebe + dependencies: + - name: idf + require: private + version: '>=5.0' + - name: espressif/usb + registry_url: https://components.espressif.com + require: public + rules: + - if: idf_version >=6.0 + - if: target not in ["linux"] + version: ^1.0.0 + source: + registry_url: https://components.espressif.com + type: service + targets: + - esp32s2 + - esp32s3 + - esp32p4 + - esp32h4 + - esp32s31 + - linux + version: 2.5.1 espressif/zlib: component_hash: 9083cada65f1c9d298ef92266232d071c6f2f335164e2799554b3eae73aedc55 dependencies: @@ -387,11 +570,14 @@ direct_dependencies: - espressif/esp_lcd_co5300 - espressif/esp_lcd_touch_cst9220 - espressif/esp_lvgl_adapter +- espressif/esp_video +- espressif/hand_detect +- espressif/hand_gesture_recognition - espressif/spi_nand_flash - espressif2022/bmi270 - idf - lvgl/lvgl - tangyumei3535/bmm150_sensorapi -manifest_hash: 381e620403dd6ec5c85708e211ed42ac6af8ae3c1200c0c5392c28805b22019b +manifest_hash: 53e3bd813b8217d5baee702eb587a7358830bd13a6fd85c8f6f41b0109f559f2 target: esp32s31 version: 3.0.0 diff --git a/firmware/targets/esp_mosaico/main/CMakeLists.txt b/firmware/targets/esp_mosaico/main/CMakeLists.txt index 7e959f5..ae8aa2b 100644 --- a/firmware/targets/esp_mosaico/main/CMakeLists.txt +++ b/firmware/targets/esp_mosaico/main/CMakeLists.txt @@ -5,6 +5,9 @@ idf_component_register( "local_clock.cpp" "orientation.cpp" "power_monitor.cpp" + "panel_settings.cpp" + "gesture_input.cpp" + "gesture_tracker.cpp" INCLUDE_DIRS "." REQUIRES nvs_flash @@ -17,3 +20,7 @@ idf_component_register( ) target_compile_features(${COMPONENT_LIB} PUBLIC cxx_std_17) + +if(CONFIG_MOSAICO_GESTURE_INPUT) + target_sources(${COMPONENT_LIB} PRIVATE "gesture_camera.cpp" "gesture_image.cpp") +endif() diff --git a/firmware/targets/esp_mosaico/main/Kconfig.projbuild b/firmware/targets/esp_mosaico/main/Kconfig.projbuild index f09a69f..af2b254 100644 --- a/firmware/targets/esp_mosaico/main/Kconfig.projbuild +++ b/firmware/targets/esp_mosaico/main/Kconfig.projbuild @@ -88,6 +88,46 @@ config MOSAICO_USAGE_PANEL_TILT_WAKE_DEGREES endmenu +menu "Camera gestures" + +config MOSAICO_GESTURE_INPUT + bool "Build local CameraBoard gesture input" + default y + help + Include CameraBoard capture (OV3640 or SC101IOT) and ESP-DL models. Runtime capture remains off + until enabled and saved in the display settings. The camera occupies + the left slot and GPIO33, so USB Serial/JTAG cannot be used concurrently. + +config MOSAICO_CAMERA_ROTATION + int "Camera mounting correction (0, 90, 180 or 270 degrees clockwise)" + range 0 270 + default 90 + depends on MOSAICO_GESTURE_INPUT + help + Rotates the mirrored raw image into panel rotation-0 coordinates. The + current display rotation is then inverted. The reference pipelines + (mosaico_module_camera, esp-vision) rotate the raw frame 90 degrees + counter-clockwise without mirroring to make it model-upright; with + mirroring applied first, the same upright view needs 90 clockwise. + +config MOSAICO_CAMERA_MIRROR + bool "Mirror the camera horizontally before rotation" + default y + depends on MOSAICO_GESTURE_INPUT + help + Mirror the front-facing camera into the user's screen coordinates. + +config MOSAICO_CAMERA_PREVIEW + bool "Enable camera preview toggled by the status icon" + default y + depends on MOSAICO_GESTURE_INPUT + help + The status-bar camera icon toggles a preview of the model input + and hand location box. The preview starts hidden, its visibility is + not persisted, and toggling it does not change gesture recognition. + +endmenu + menu "Bluetooth" config MOSAICO_USAGE_PANEL_BLE_ADVERTISING_PREFIX diff --git a/firmware/targets/esp_mosaico/main/app_main.cpp b/firmware/targets/esp_mosaico/main/app_main.cpp index 64f8cd4..44648c5 100644 --- a/firmware/targets/esp_mosaico/main/app_main.cpp +++ b/firmware/targets/esp_mosaico/main/app_main.cpp @@ -4,6 +4,8 @@ #include "bsp/esp_mosaico.h" #include "display_ui.hpp" #include "local_clock.hpp" +#include "gesture_input.hpp" +#include "panel_settings.hpp" #include "usage_panel_state/panel_presentation.hpp" #include "power_monitor.hpp" #include "usage_panel_state/render_schedule.hpp" @@ -80,9 +82,24 @@ OtaSession ota(ota_backend); AtomicPowerSource ota_power_source; BleService ble; DisplayUi ui; +GestureInput gestures; LinkUpdate ble_link{}; std::atomic status_page{Page::Overview}; std::atomic ai_button_pressed{false}; +PowerMonitor power_monitor; +std::atomic power_monitor_ready{false}; + +// Gauge configuration and settling run asynchronously to avoid delaying BLE and UI startup. +// The main loop must observe the readiness flag before reading the gauge. +void power_monitor_task(void* /*context*/) +{ + if (power_monitor.begin()) { + power_monitor_ready.store(true, std::memory_order_release); + } else { + ESP_LOGW(TAG, "BQ27220 telemetry unavailable; battery will be hidden"); + } + vTaskDelete(nullptr); +} const char* get_ui_location(void* context) { @@ -156,10 +173,22 @@ extern "C" void app_main(void) ScreenRotation initial_rotation = kFixedRotation; OrientationTracker orientation_tracker; - // The IMU shares the board I2C bus, which bsp_display_start() brings up - // together with the VCC_3V3 rail, so the panel has to come first. + // Display initialization establishes the shared I2C bus and the IMU's VCC_3V3 supply. ESP_ERROR_CHECK(ui.begin(initial_rotation) ? ESP_OK : ESP_FAIL); + // Start gauge initialization on the established bus without blocking other peripheral setup. + if (xTaskCreate(power_monitor_task, "power_init", 4096, nullptr, 1, nullptr) != pdPASS) { + ESP_LOGW(TAG, "BQ27220 start task unavailable; battery will be hidden"); + } + + // Advertising starts before the remaining peripherals so the bridge can reconnect + // while they initialize; link events wait in the queue for the main loop. + ESP_ERROR_CHECK(events.begin() ? ESP_OK : ESP_ERR_NO_MEM); + // Install the power gate before BLE can receive OTA manifests. + // The power verdict remains unavailable until the main-loop poll obtains a valid reading. + ota.set_power_source(&ota_power_source); + ESP_ERROR_CHECK(ble.start(events, ble_config)); + const bool orientation_sensor_ready = (kAutoRotationEnabled || kWakeDetectionEnabled) && init_orientation_sensor(); @@ -184,19 +213,27 @@ extern "C" void app_main(void) } } - PowerMonitor power_monitor; - const bool power_monitor_ready = power_monitor.begin(); - if (!power_monitor_ready) { - ESP_LOGW(TAG, "BQ27220 telemetry unavailable; battery will be hidden"); - } const bool ai_button_ready = init_ai_button(); - DisplaySettings saved_settings = load_display_settings({ + const PanelSettings persisted_settings = load_panel_settings({ CONFIG_MOSAICO_USAGE_PANEL_DISPLAY_BRIGHTNESS_PERCENT, CONFIG_MOSAICO_USAGE_PANEL_SCREENSAVER_TIMEOUT_SECONDS}); + DisplaySettings saved_settings = persisted_settings.display; DisplaySettings settings = saved_settings; bool settings_open = false; - ui.apply_settings(settings, false); + // camera_enabled is persisted and drives capture; camera_setting is the + // settings-page preview, committed by SAVE together with the display values. + bool camera_enabled = persisted_settings.camera_enabled; + bool camera_setting = camera_enabled; +#if CONFIG_MOSAICO_CAMERA_PREVIEW + gestures.set_preview_sink( + [](void* context, const uint8_t* rgb, unsigned width, unsigned height, const HandObservation& hand) { + static_cast(context)->show_camera_preview(rgb, width, height, hand); + }, + &ui); +#endif + if (!gestures.begin()) ESP_LOGW(TAG, "gesture worker unavailable"); + ui.apply_settings(settings, camera_setting, false); ScreenRotation desired_rotation = initial_rotation; // ui.begin() brought the panel up at kFixedRotation. Anything the startup @@ -205,12 +242,6 @@ extern "C" void app_main(void) ScreenRotation applied_rotation = kFixedRotation; uint64_t next_orientation_poll_ms = 0; - ESP_ERROR_CHECK(events.begin() ? ESP_OK : ESP_ERR_NO_MEM); - // Wired before BLE so no peer can reach the manifest path ungated; the - // verdict itself is published by the main-loop power poll below. - ota.set_power_source(&ota_power_source); - ESP_ERROR_CHECK(ble.start(events, ble_config)); - ESP_LOGI(TAG, "ESP-Mosaico usage panel started"); Page page = Page::Overview; @@ -259,6 +290,7 @@ extern "C" void app_main(void) dirty = true; } if (ui.take_touch_down()) { + gestures.invalidate(); const bool was_asleep = screensaver.view(now_ms).active; screensaver.note_touch_down(now_ms); if (was_asleep) { @@ -269,6 +301,7 @@ extern "C" void app_main(void) } if (ai_button_ready && ai_button_pressed.exchange(false, std::memory_order_relaxed)) { + gestures.invalidate(); const bool was_asleep = screensaver.view(now_ms).active; screensaver.note_touch_down(now_ms); if (was_asleep) { @@ -289,17 +322,21 @@ extern "C" void app_main(void) const bool settings_blocked = ble_link.has_passkey || ota_view.phase != OtaPhase::Idle; if (settings_open && settings_blocked) { settings = saved_settings; + camera_setting = camera_enabled; settings_open = false; - ui.apply_settings(settings, false); + ui.apply_settings(settings, camera_setting, false); screensaver.reset(now_ms, page); dirty = true; } - if (action != SettingsAction::None && !settings_blocked) { + const bool camera_toggled = ui.take_camera_toggle(); + if ((action != SettingsAction::None || camera_toggled) && !settings_blocked) { bool save_error = false; const DisplaySettings before = settings; + const bool camera_before = camera_setting; const bool was_open = settings_open; if (action == SettingsAction::Open) settings_open = true; else if (settings_open) { + if (camera_toggled) camera_setting = !camera_setting; if (action == SettingsAction::Dimmer) settings.brightness = settings.brightness <= 10 ? 1 : settings.brightness - 10; else if (action == SettingsAction::Brighter) @@ -307,19 +344,25 @@ extern "C" void app_main(void) else if (action >= SettingsAction::TimeoutOff && action <= SettingsAction::Timeout30) select_clock_timeout(action, settings.clock_timeout_seconds); else if (action == SettingsAction::Save) { - const esp_err_t error = save_display_settings(settings); + const esp_err_t error = save_panel_settings({settings, camera_setting}); save_error = error != ESP_OK; - if (!save_error) { saved_settings = settings; settings_open = false; } - else ESP_LOGW(TAG, "display settings save failed: %s", esp_err_to_name(error)); + if (!save_error) { + saved_settings = settings; + camera_enabled = camera_setting; + settings_open = false; + } else ESP_LOGW(TAG, "settings save failed: %s", esp_err_to_name(error)); } else if (action == SettingsAction::Cancel) { settings = saved_settings; + camera_setting = camera_enabled; settings_open = false; } } - if (!ui.apply_settings(settings, settings_open, save_error, + if (!ui.apply_settings(settings, camera_setting, settings_open, save_error, settings.brightness != saved_settings.brightness || - settings.clock_timeout_seconds != saved_settings.clock_timeout_seconds)) { + settings.clock_timeout_seconds != saved_settings.clock_timeout_seconds || + camera_setting != camera_enabled)) { settings = before; + camera_setting = camera_before; settings_open = was_open; ESP_LOGW(TAG, "display settings preview failed"); } @@ -329,6 +372,7 @@ extern "C" void app_main(void) } const Page requested = ui.requested_page(); if (requested != page) { + gestures.invalidate(); page = requested; status_page.store(page, std::memory_order_relaxed); dirty = true; @@ -339,8 +383,10 @@ extern "C" void app_main(void) if (event.type == AppEventType::Usage) { model.apply(event.usage, esp_timer_get_time() / 1000); } else if (event.type == AppEventType::Link) { + gestures.invalidate(); ble_link = event.link; } else if (event.type == AppEventType::ScreenPage) { + gestures.invalidate(); if (ota_view.phase == OtaPhase::Idle && !ble_link.has_passkey) { if (screensaver.view(now_ms).active) page = screensaver.restore_page(); page = page_after_swipe(page, event.previous_page @@ -348,14 +394,17 @@ extern "C" void app_main(void) ui.request_page(page); screensaver.reset(now_ms, page); settings = saved_settings; + camera_setting = camera_enabled; settings_open = false; - ui.apply_settings(settings, false); + ui.apply_settings(settings, camera_setting, false); status_page.store(page, std::memory_order_relaxed); } } else if (event.type == AppEventType::ScreenToggle) { + gestures.invalidate(); settings = saved_settings; + camera_setting = camera_enabled; settings_open = false; - ui.apply_settings(settings, false); + ui.apply_settings(settings, camera_setting, false); const bool was_asleep = screensaver.view(now_ms).active; screensaver.note_remote_toggle(now_ms, page); if (was_asleep) { @@ -387,7 +436,7 @@ extern "C" void app_main(void) if (now_ms >= next_power_poll_ms) { next_power_poll_ms = now_ms + POWER_POLL_PERIOD_MS; raw_sample = {}; - if (power_monitor_ready) power_monitor.read(raw_sample); + if (power_monitor_ready.load(std::memory_order_acquire)) power_monitor.read(raw_sample); battery_view = power_filter.update(raw_sample, now_ms); const OtaPowerReading reading = derive_power_reading(battery_view, raw_sample); @@ -438,6 +487,7 @@ extern "C" void app_main(void) woke = tilt_detector.update(accel_x, accel_y, accel_z) || woke; #endif if (woke) { + gestures.invalidate(); screensaver.note_shake(now_ms); dirty = true; } @@ -455,6 +505,32 @@ extern "C" void app_main(void) screensaver.update( now_ms, page, ble_link.has_passkey, ota_view.phase != OtaPhase::Idle || show_ota_error || settings_open); + const bool gesture_allowed = !settings_open && !ble_link.has_passkey && + ota_view.phase == OtaPhase::Idle && !show_ota_error && desired_rotation == applied_rotation; + gestures.configure(camera_enabled, gesture_allowed, + screensaver.view(now_ms).active, page, applied_rotation); + GestureEvent gesture{}; + if (gestures.take(gesture, esp_timer_get_time() / 1000)) { + const GestureEffect effect = route_gesture(gesture.action, gesture_allowed, + screensaver.view(now_ms).active); + if (effect == GestureEffect::Wake) { + screensaver.note_touch_down(now_ms); + if (!screensaver.view(now_ms).active) page = screensaver.restore_page(); + } else if (effect == GestureEffect::Clock) { + screensaver.enter_clock(now_ms, page); + } else if (effect == GestureEffect::Next) { + page = page_after_swipe(page, SwipeDirection::Next); + screensaver.reset(now_ms, page); + } + if (effect != GestureEffect::None) { + ui.request_page(page); + status_page.store(page, std::memory_order_relaxed); + gestures.configure(camera_enabled, gesture_allowed, + screensaver.view(now_ms).active, page, applied_rotation, true); + ESP_LOGI(TAG, "gesture action=%u effect=%u", unsigned(gesture.action), unsigned(effect)); + dirty = true; + } + } const ScreensaverView screensaver_view = screensaver.view(now_ms); const PanelPresentation presentation{ .clock = format_clock_view( @@ -488,6 +564,21 @@ extern "C" void app_main(void) has_last_render_key = true; dirty = false; } + const auto gesture_status = gestures.status(); + const auto gesture_progress = gestures.progress(); + const bool gesture_feedback_visible = camera_enabled && gesture_allowed && !screensaver_view.active; + static bool shown_camera_enabled = false; + static GestureStatus shown_gesture_status = GestureStatus::Off; + static uint8_t shown_gesture_progress = 0; + static bool shown_gesture_feedback = false; + if ((shown_camera_enabled != camera_enabled || shown_gesture_status != gesture_status || + shown_gesture_progress != gesture_progress || shown_gesture_feedback != gesture_feedback_visible) && + ui.show_camera_state(camera_enabled, gesture_status, gesture_progress, gesture_feedback_visible)) { + shown_camera_enabled = camera_enabled; + shown_gesture_status = gesture_status; + shown_gesture_progress = gesture_progress; + shown_gesture_feedback = gesture_feedback_visible; + } last_checked_second = second; ble.verify_boot(); diff --git a/firmware/targets/esp_mosaico/main/display_ui.cpp b/firmware/targets/esp_mosaico/main/display_ui.cpp index c08b15f..8e778e1 100644 --- a/firmware/targets/esp_mosaico/main/display_ui.cpp +++ b/firmware/targets/esp_mosaico/main/display_ui.cpp @@ -4,6 +4,7 @@ #include #include "bsp/esp_mosaico.h" +#include "esp_heap_caps.h" #include "esp_log.h" #include "usage_fonts/usage_fonts.h" #include "esp_timer.h" @@ -151,13 +152,7 @@ lv_display_t* start_display( ESP_LOGE(TAG, "touch initialization failed"); return nullptr; } - const esp_err_t error = bsp_display_brightness_set( - CONFIG_MOSAICO_USAGE_PANEL_DISPLAY_BRIGHTNESS_PERCENT); - if (error != ESP_OK) { - ESP_LOGE(TAG, "display brightness setting failed: %s", - esp_err_to_name(error)); - return nullptr; - } + // The panel comes up at zero brightness; the first completed refresh starts the fade-in. return display; } @@ -247,6 +242,19 @@ void style_panel(lv_obj_t* panel, uint32_t background = COLOR_PANEL, lv_obj_remove_flag(panel, LV_OBJ_FLAG_SCROLLABLE); } +/** Creates an unstyled, non-interactive primitive for icons drawn from shapes. */ +lv_obj_t* make_shape(lv_obj_t* parent, int x, int y, int width, int height, int radius) +{ + lv_obj_t* shape = lv_obj_create(parent); + lv_obj_remove_style_all(shape); + lv_obj_set_pos(shape, x, y); + lv_obj_set_size(shape, width, height); + lv_obj_set_style_radius(shape, radius, 0); + lv_obj_remove_flag(shape, LV_OBJ_FLAG_CLICKABLE); + lv_obj_remove_flag(shape, LV_OBJ_FLAG_SCROLLABLE); + return shape; +} + lv_obj_t* make_label(lv_obj_t* parent, const char* text, int x, int y, const lv_font_t* font, uint32_t color = COLOR_TEXT) { @@ -357,6 +365,8 @@ bool DisplayUi::begin(ScreenRotation initial_rotation) build_provider_page(screen, Provider::Codex); build_provider_page(screen, Provider::Claude); build_navigation(screen); + // Below the screensaver and overlays, which raise themselves when shown. + build_camera_preview(screen); build_screensaver(screen); build_settings(screen); build_pairing_overlay(screen); @@ -364,6 +374,8 @@ bool DisplayUi::begin(ScreenRotation initial_rotation) show_page(Page::Overview, /*animate=*/false); lv_display_add_event_cb( display_, transition_refresh_ready, LV_EVENT_REFR_READY, this); + // Frame memory is undefined until the first refresh, which then fades the panel in. + transition_refresh_pending_ = true; return true; } @@ -453,6 +465,9 @@ void DisplayUi::build_header(lv_obj_t* screen) battery_segments_[i] = lv_obj_create(body); lv_obj_set_pos(battery_segments_[i], 2 + i * 6, 2); lv_obj_set_size(battery_segments_[i], 5, 8); + // The first frame precedes the first battery reading and shows it as unavailable. + lv_obj_set_style_bg_color(battery_segments_[i], lv_color_hex(COLOR_BATTERY_OFF), 0); + lv_obj_set_style_bg_opa(battery_segments_[i], LV_OPA_COVER, 0); lv_obj_set_style_border_width(battery_segments_[i], 0, 0); lv_obj_set_style_radius(battery_segments_[i], 1, 0); lv_obj_set_style_pad_all(battery_segments_[i], 0, 0); @@ -468,6 +483,38 @@ void DisplayUi::build_header(lv_obj_t* screen) lv_obj_set_style_pad_all(terminal, 0, 0); lv_obj_remove_flag(terminal, LV_OBJ_FLAG_SCROLLABLE); +#if CONFIG_MOSAICO_GESTURE_INPUT + // Solid camera silhouette left of the connection status, vertically centred + // on its dot. The header fonts carry no camera glyph, so it is drawn from + // primitives: body and viewfinder in the state colour, a background-coloured + // lens ring and a state-coloured lens centre. + static const lv_point_precise_t kSlash[] = {{3, 2}, {23, 26}}; + camera_indicator_ = make_shape(screen, 275, 13, 26, 28, 0); +#if CONFIG_MOSAICO_CAMERA_PREVIEW + lv_obj_add_flag(camera_indicator_, LV_OBJ_FLAG_CLICKABLE); + lv_obj_set_ext_click_area(camera_indicator_, 8); + lv_obj_add_event_cb(camera_indicator_, camera_preview_toggle_event, LV_EVENT_CLICKED, this); +#endif + camera_indicator_parts_[0] = make_shape(camera_indicator_, 6, 3, 8, 4, 1); + camera_indicator_parts_[1] = make_shape(camera_indicator_, 0, 6, 26, 16, 4); + camera_indicator_lens_ = make_shape(camera_indicator_, 8, 9, 10, 10, 5); + camera_indicator_parts_[2] = make_shape(camera_indicator_, 11, 12, 4, 4, 2); + lv_obj_set_style_bg_color(camera_indicator_lens_, lv_color_hex(COLOR_BG), 0); + for (lv_obj_t* part : {camera_indicator_parts_[0], camera_indicator_parts_[1], + camera_indicator_lens_, camera_indicator_parts_[2]}) + lv_obj_set_style_bg_opa(part, LV_OPA_COVER, 0); + for (size_t i = 0; i < 2; ++i) { + lv_obj_t* slash = lv_line_create(camera_indicator_); + camera_indicator_slash_[i] = slash; + lv_line_set_points(slash, kSlash, 2); + lv_obj_set_style_line_width(slash, i == 0 ? 6 : 2, 0); + lv_obj_set_style_line_color(slash, lv_color_hex(i == 0 ? COLOR_BG : COLOR_RESET), 0); + lv_obj_set_style_line_rounded(slash, true, 0); + lv_obj_remove_flag(slash, LV_OBJ_FLAG_CLICKABLE); + } + lv_obj_add_flag(camera_indicator_, LV_OBJ_FLAG_HIDDEN); +#endif + lv_obj_t* divider = lv_obj_create(screen); lv_obj_set_pos(divider, MARGIN, HEADER_H); lv_obj_set_size(divider, CARD_W, 1); @@ -769,20 +816,20 @@ void DisplayUi::build_settings(lv_obj_t* screen) lv_obj_set_style_bg_color(settings_panel_, lv_color_hex(COLOR_BG), 0); lv_obj_set_style_bg_opa(settings_panel_, LV_OPA_COVER, 0); lv_obj_remove_flag(settings_panel_, LV_OBJ_FLAG_SCROLLABLE); - auto* title = make_label(settings_panel_, "BRIGHTNESS", MARGIN, 24, + auto* title = make_label(settings_panel_, "BRIGHTNESS", MARGIN, 16, &lv_font_montserrat_16, COLOR_MUTED); lv_obj_set_style_text_letter_space(title, 1, 0); - settings_brightness_ = make_label(settings_panel_, "", MARGIN, 58, &lv_font_montserrat_48); + settings_brightness_ = make_label(settings_panel_, "", MARGIN, 42, &lv_font_montserrat_48); lv_obj_t* divider = lv_obj_create(settings_panel_); lv_obj_remove_style_all(divider); - lv_obj_set_pos(divider, MARGIN, 158); + lv_obj_set_pos(divider, MARGIN, 128); lv_obj_set_size(divider, CARD_W, 1); lv_obj_set_style_bg_color(divider, lv_color_hex(COLOR_LINE), 0); lv_obj_set_style_bg_opa(divider, LV_OPA_COVER, 0); - title = make_label(settings_panel_, "AUTO CLOCK", MARGIN, 180, + title = make_label(settings_panel_, "AUTO CLOCK", MARGIN, 145, &lv_font_montserrat_16, COLOR_MUTED); lv_obj_set_style_text_letter_space(title, 1, 0); - settings_timeout_ = make_label(settings_panel_, "AFTER IDLE / MIN", MARGIN, 207, + settings_timeout_ = make_label(settings_panel_, "AFTER IDLE / MIN", MARGIN, 170, &lv_font_montserrat_14, COLOR_MUTED); settings_hint_ = make_label(settings_panel_, "", MARGIN, 322, &montserrat_semibold_13, COLOR_TEXT); @@ -791,13 +838,13 @@ void DisplayUi::build_settings(lv_obj_t* screen) SettingsAction::TimeoutOff, SettingsAction::Timeout1, SettingsAction::Timeout5, SettingsAction::Timeout10, SettingsAction::Timeout30, SettingsAction::Save, SettingsAction::Cancel}; const int x[] = {288, 376, 24, 112, 200, 288, 376, 250, MARGIN}; - const int y[] = {58, 58, 244, 244, 244, 244, 244, 349, 349}; + const int y[] = {38, 38, 200, 200, 200, 200, 200, 349, 349}; for (size_t i = 0; i < 9; ++i) { lv_obj_t* button = lv_button_create(settings_panel_); settings_buttons_[i] = button; lv_obj_remove_style_all(button); lv_obj_set_pos(button, x[i], y[i]); - lv_obj_set_size(button, i < 7 ? 80 : 206, i < 2 ? 80 : 64); + lv_obj_set_size(button, i < 7 ? 80 : 206, i < 2 ? 72 : (i < 7 ? 52 : 64)); style_panel(button); lv_obj_set_style_text_color(button, lv_color_hex(COLOR_MUTED), 0); lv_obj_set_style_text_font(button, i < 2 ? &lv_font_montserrat_30 : &lv_font_montserrat_20, 0); @@ -816,10 +863,50 @@ void DisplayUi::build_settings(lv_obj_t* screen) lv_label_set_text(label, labels[i]); lv_obj_center(label); } +#if CONFIG_MOSAICO_GESTURE_INPUT + divider = lv_obj_create(settings_panel_); + lv_obj_remove_style_all(divider); + lv_obj_set_pos(divider, MARGIN, 264); + lv_obj_set_size(divider, CARD_W, 1); + lv_obj_set_style_bg_color(divider, lv_color_hex(COLOR_LINE), 0); + lv_obj_set_style_bg_opa(divider, LV_OPA_COVER, 0); + title = make_label(settings_panel_, "CAMERA", MARGIN, 276, &lv_font_montserrat_16, COLOR_MUTED); + lv_obj_set_style_text_letter_space(title, 1, 0); + make_label(settings_panel_, "HAND GESTURES / ON DEVICE", MARGIN, 298, + &lv_font_montserrat_14, COLOR_MUTED); + // Not checkable: app_main owns the previewed state and draws it through + // apply_settings, so a failed preview can roll back like the other controls. + camera_switch_track_ = lv_button_create(settings_panel_); + lv_obj_remove_style_all(camera_switch_track_); + lv_obj_set_pos(camera_switch_track_, 376, 272); + lv_obj_set_size(camera_switch_track_, 80, 44); + style_panel(camera_switch_track_, COLOR_PANEL, 22); + lv_obj_set_style_bg_color(camera_switch_track_, lv_color_hex(COLOR_TRACK), LV_STATE_PRESSED); + lv_obj_set_style_border_color(camera_switch_track_, lv_color_hex(COLOR_MUTED), LV_STATE_PRESSED); + lv_obj_set_style_bg_color(camera_switch_track_, lv_color_hex(COLOR_TEXT), LV_STATE_CHECKED); + lv_obj_set_style_border_color(camera_switch_track_, lv_color_hex(COLOR_TEXT), LV_STATE_CHECKED); + lv_obj_set_style_bg_color(camera_switch_track_, lv_color_hex(COLOR_RESET), LV_STATE_CHECKED | LV_STATE_PRESSED); + lv_obj_add_event_cb(camera_switch_track_, camera_toggle_event, LV_EVENT_CLICKED, this); + camera_switch_knob_ = make_shape(camera_switch_track_, 0, 0, 32, 32, 16); + lv_obj_set_style_bg_opa(camera_switch_knob_, LV_OPA_COVER, 0); + lv_obj_set_style_bg_color(camera_switch_knob_, lv_color_hex(COLOR_MUTED), 0); + lv_obj_align(camera_switch_knob_, LV_ALIGN_LEFT_MID, 5, 0); + gesture_progress_ = lv_bar_create(screen); + lv_obj_remove_style_all(gesture_progress_); + lv_obj_set_pos(gesture_progress_, MARGIN, HEADER_H - 4); + lv_obj_set_size(gesture_progress_, CARD_W, 3); + lv_bar_set_range(gesture_progress_, 0, 100); + lv_obj_set_style_bg_color(gesture_progress_, lv_color_hex(COLOR_TRACK), LV_PART_MAIN); + lv_obj_set_style_bg_opa(gesture_progress_, LV_OPA_COVER, LV_PART_MAIN); + lv_obj_set_style_bg_color(gesture_progress_, lv_color_hex(COLOR_RESET), LV_PART_INDICATOR); + lv_obj_set_style_bg_opa(gesture_progress_, LV_OPA_COVER, LV_PART_INDICATOR); + lv_obj_add_flag(gesture_progress_, LV_OBJ_FLAG_HIDDEN); +#endif lv_obj_add_flag(settings_panel_, LV_OBJ_FLAG_HIDDEN); } -bool DisplayUi::apply_settings(const DisplaySettings& settings, bool visible, bool save_error, bool modified) +bool DisplayUi::apply_settings(const DisplaySettings& settings, bool camera, bool visible, + bool save_error, bool modified) { DisplayLock lock; if (!lock) return false; @@ -845,6 +932,11 @@ bool DisplayUi::apply_settings(const DisplaySettings& settings, bool visible, bo else lv_label_set_text_fmt(settings_timeout_, "AFTER IDLE / MIN - CURRENT %lu SEC", static_cast(settings.clock_timeout_seconds)); lv_label_set_text(settings_hint_, save_error ? "SAVE FAILED - TRY AGAIN" : ""); + if (camera_switch_track_) { + lv_obj_set_state(camera_switch_track_, LV_STATE_CHECKED, camera); + lv_obj_set_style_bg_color(camera_switch_knob_, lv_color_hex(camera ? COLOR_BG : COLOR_MUTED), 0); + lv_obj_align(camera_switch_knob_, camera ? LV_ALIGN_RIGHT_MID : LV_ALIGN_LEFT_MID, camera ? -5 : 5, 0); + } if (visible && !was_visible) { lv_obj_remove_flag(settings_panel_, LV_OBJ_FLAG_HIDDEN); lv_obj_move_foreground(settings_panel_); @@ -852,6 +944,112 @@ bool DisplayUi::apply_settings(const DisplaySettings& settings, bool visible, bo return true; } +void DisplayUi::camera_toggle_event(lv_event_t* event) +{ + auto* ui = static_cast(lv_event_get_user_data(event)); + if (ui->settings_visible_.load(std::memory_order_relaxed)) + ui->camera_toggle_.store(true, std::memory_order_relaxed); +} + +void DisplayUi::camera_preview_toggle_event(lv_event_t* event) +{ + auto* ui = static_cast(lv_event_get_user_data(event)); + if (!ui->camera_preview_ || + !ui->camera_preview_available_.load(std::memory_order_relaxed)) return; + const bool enabled = !ui->camera_preview_enabled_.load(std::memory_order_relaxed); + ui->camera_preview_enabled_.store(enabled, std::memory_order_relaxed); + lv_obj_set_flag(ui->camera_preview_, LV_OBJ_FLAG_HIDDEN, !enabled); +} + +bool DisplayUi::show_camera_state(bool enabled, GestureStatus status, uint8_t progress, bool visible) +{ + if (!camera_indicator_) return true; + DisplayLock lock; + if (!lock) return false; + // Text-bright while the sensor is starting or capturing; dim while the switch + // is on but capture is paused (settings, pairing, OTA); dim and struck + // through when capture failed and needs a switch-off/on retry or a reboot. + const bool capturing = status == GestureStatus::Starting || status == GestureStatus::Ready; + const bool failed = status == GestureStatus::CameraError || status == GestureStatus::MemoryError || + status == GestureStatus::Fault; + const lv_color_t color = lv_color_hex(capturing ? COLOR_TEXT : COLOR_DIM); + lv_obj_set_flag(camera_indicator_, LV_OBJ_FLAG_HIDDEN, !enabled); + for (lv_obj_t* part : camera_indicator_parts_) lv_obj_set_style_bg_color(part, color, 0); + for (lv_obj_t* slash : camera_indicator_slash_) lv_obj_set_flag(slash, LV_OBJ_FLAG_HIDDEN, !failed); + lv_obj_set_flag(gesture_progress_, LV_OBJ_FLAG_HIDDEN, !visible || !progress); + const bool available = enabled && visible && status == GestureStatus::Ready; + camera_preview_available_.store(available, std::memory_order_relaxed); + if (camera_preview_) + lv_obj_set_flag(camera_preview_, LV_OBJ_FLAG_HIDDEN, + !available || !camera_preview_enabled_.load(std::memory_order_relaxed)); + if (visible && progress) { + lv_bar_set_value(gesture_progress_, progress, LV_ANIM_OFF); + lv_obj_move_foreground(gesture_progress_); + } + return true; +} + +void DisplayUi::build_camera_preview(lv_obj_t* screen) +{ +#if CONFIG_MOSAICO_CAMERA_PREVIEW + // Half-resolution RGB preview; either rotated dimension is at most 160 pixels. + constexpr uint32_t kSide = 160; + const uint32_t stride = lv_draw_buf_width_to_stride(kSide, LV_COLOR_FORMAT_RGB565); + camera_preview_pixels_ = static_cast( + heap_caps_aligned_calloc(LV_DRAW_BUF_ALIGN, stride, kSide, MALLOC_CAP_SPIRAM)); + if (!camera_preview_pixels_) return; + camera_preview_ = make_shape(screen, SCREEN_W - MARGIN - kSide, CONTENT_Y + 8, kSide, kSide, 0); + camera_preview_canvas_ = lv_canvas_create(camera_preview_); + lv_canvas_set_buffer(camera_preview_canvas_, camera_preview_pixels_, kSide, 120, LV_COLOR_FORMAT_RGB565); + lv_obj_set_pos(camera_preview_canvas_, 0, 0); + camera_preview_box_ = make_shape(camera_preview_, 0, 0, 1, 1, 0); + lv_obj_set_style_border_width(camera_preview_box_, 2, 0); + lv_obj_set_style_border_color(camera_preview_box_, lv_color_hex(COLOR_TEXT), 0); + lv_obj_add_flag(camera_preview_box_, LV_OBJ_FLAG_HIDDEN); + lv_obj_add_flag(camera_preview_, LV_OBJ_FLAG_HIDDEN); +#else + (void)screen; +#endif +} + +void DisplayUi::show_camera_preview(const uint8_t* rgb, unsigned width, unsigned height, const HandObservation& hand) +{ + if (!camera_preview_enabled_.load(std::memory_order_relaxed) || + !camera_preview_available_.load(std::memory_order_relaxed) || + !camera_preview_canvas_ || !rgb || width > 320 || height > 320) return; + const uint32_t w = width / 2; + const uint32_t h = height / 2; + DisplayLock lock; + if (!lock) return; + // Visibility can change while this frame waits for the display lock. + if (lv_obj_has_flag(camera_preview_, LV_OBJ_FLAG_HIDDEN)) return; + const lv_image_dsc_t* image = lv_canvas_get_image(camera_preview_canvas_); + if (image->header.w != w || image->header.h != h) { + lv_canvas_set_buffer(camera_preview_canvas_, camera_preview_pixels_, w, h, LV_COLOR_FORMAT_RGB565); + lv_obj_set_size(camera_preview_, w, h); + lv_obj_set_x(camera_preview_, SCREEN_W - MARGIN - w); + } + const uint32_t stride = lv_canvas_get_image(camera_preview_canvas_)->header.stride; + for (uint32_t y = 0; y < h; ++y) { + auto* row = reinterpret_cast(camera_preview_pixels_ + y * stride); + for (uint32_t x = 0; x < w; ++x) { + const uint8_t* p = rgb + (size_t(y) * 2 * width + x * 2) * 3; + row[x] = uint16_t(((p[0] & 0xF8) << 8) | ((p[1] & 0xFC) << 3) | (p[2] >> 3)); + } + } + // The image cache is disabled (CONFIG_LV_CACHE_DEF_SIZE=0), so invalidation redraws the new pixels. + lv_obj_invalidate(camera_preview_canvas_); + if (hand.count) { + const int bw = int(hand.width * w); + const int bh = int(hand.height * h); + lv_obj_set_pos(camera_preview_box_, int(hand.x * w) - bw / 2, int(hand.y * h) - bh / 2); + lv_obj_set_size(camera_preview_box_, bw, bh); + lv_obj_remove_flag(camera_preview_box_, LV_OBJ_FLAG_HIDDEN); + } else { + lv_obj_add_flag(camera_preview_box_, LV_OBJ_FLAG_HIDDEN); + } +} + void DisplayUi::settings_event(lv_event_t* event) { auto* binding = static_cast(lv_event_get_user_data(event)); diff --git a/firmware/targets/esp_mosaico/main/display_ui.hpp b/firmware/targets/esp_mosaico/main/display_ui.hpp index bc0c7db..834430f 100644 --- a/firmware/targets/esp_mosaico/main/display_ui.hpp +++ b/firmware/targets/esp_mosaico/main/display_ui.hpp @@ -6,6 +6,7 @@ #include "esp_lcd_panel_io.h" #include "lvgl.h" #include "orientation.hpp" +#include "gesture_input.hpp" #include "usage_panel_state/display_settings.hpp" #include "usage_panel_state/page_state.hpp" #include "usage_panel_state/panel_presentation.hpp" @@ -29,7 +30,21 @@ class DisplayUi { bool begin(ScreenRotation initial_rotation); void request_page(Page page) { requested_page_.store(page, std::memory_order_relaxed); } - bool apply_settings(const DisplaySettings& settings, bool visible, bool save_error = false, bool modified = false); + /** Shows the settings preview; camera is the previewed switch state, committed by SAVE. */ + bool apply_settings(const DisplaySettings& settings, bool camera, bool visible, + bool save_error = false, bool modified = false); + /** Consumes one press of the settings camera switch. The switch never changes itself. */ + bool take_camera_toggle() { return camera_toggle_.exchange(false, std::memory_order_relaxed); } + /** + * Updates the header camera icon and the OK-hold progress bar. + * Returns false if the LVGL lock was unavailable, so the caller can retry. + */ + bool show_camera_state(bool enabled, GestureStatus status, uint8_t progress, bool feedback_visible); + /** + * Draws the model input at half size with the detected box (debug preview). + * Called from the vision worker; drawing requires an enabled, available preview. + */ + void show_camera_preview(const uint8_t* rgb, unsigned width, unsigned height, const HandObservation& hand); SettingsAction take_settings_action() { return settings_action_.exchange(SettingsAction::None, std::memory_order_relaxed); @@ -105,7 +120,10 @@ class DisplayUi { }; void build_settings(lv_obj_t* screen); + void build_camera_preview(lv_obj_t* screen); static void settings_event(lv_event_t* event); + static void camera_toggle_event(lv_event_t* event); + static void camera_preview_toggle_event(lv_event_t* event); static void long_press_event(lv_event_t* event); void build_header(lv_obj_t* screen); void build_overview(lv_obj_t* screen); @@ -140,6 +158,20 @@ class DisplayUi { lv_obj_t* settings_buttons_[9]{}; lv_obj_t* settings_timeout_ = nullptr; lv_obj_t* settings_hint_ = nullptr; + lv_obj_t* camera_switch_track_ = nullptr; + lv_obj_t* camera_switch_knob_ = nullptr; + lv_obj_t* camera_indicator_ = nullptr; + lv_obj_t* camera_indicator_parts_[3]{}; // Viewfinder bump, body and lens centre. + lv_obj_t* camera_indicator_lens_ = nullptr; // Background-coloured lens ring. + lv_obj_t* camera_indicator_slash_[2]{}; // Background halo and stroke. + lv_obj_t* gesture_progress_ = nullptr; + std::atomic camera_preview_enabled_{false}; + std::atomic camera_preview_available_{false}; // Allows hidden frames to skip the display lock. + lv_obj_t* camera_preview_ = nullptr; + lv_obj_t* camera_preview_canvas_ = nullptr; + lv_obj_t* camera_preview_box_ = nullptr; + uint8_t* camera_preview_pixels_ = nullptr; // RGB565 canvas buffer in PSRAM. + std::atomic camera_toggle_{false}; uint8_t brightness_percent_ = 50; // Serialized by the LVGL lock. std::atomic requested_page_{Page::Overview}; std::atomic ota_active_{false}; diff --git a/firmware/targets/esp_mosaico/main/gesture_camera.cpp b/firmware/targets/esp_mosaico/main/gesture_camera.cpp new file mode 100644 index 0000000..31e3881 --- /dev/null +++ b/firmware/targets/esp_mosaico/main/gesture_camera.cpp @@ -0,0 +1,246 @@ +// SPDX-FileCopyrightText: 2026 Espressif Systems (Shanghai) CO LTD +// SPDX-License-Identifier: Apache-2.0 +#include "gesture_camera.hpp" +#include "gesture_image.hpp" + +#include +#include +#include +#include +#include + +#include "bsp/esp_mosaico.h" +#include "esp_log.h" +#include "esp_video_device.h" +#include "esp_video_init.h" +#include "esp_video_ioctl.h" +#include "freertos/FreeRTOS.h" +#include "freertos/task.h" +#include "hal/usb_serial_jtag_ll.h" +#include "linux/videodev2.h" + +namespace usage_panel::mosaico { +namespace { +constexpr char TAG[] = "gesture_camera"; +constexpr gpio_num_t kPowerDown = GPIO_NUM_48; +constexpr gpio_num_t kFlash = GPIO_NUM_34; +// Start-up timing follows esp-vision boards/ESP32_S31_MOSAICO/camera.c. +constexpr uint32_t kPowerStabilizationMs = 20; +constexpr suseconds_t kFrameTimeoutUs = 500000; +constexpr unsigned kPrimeAttempts = 3; +// esp-mosaico-bsp mosaico_module_camera and its module manager access the +// subboard I2C1 bus, which has only internal pull-ups, at 100 kHz. +constexpr uint32_t kSccbFrequencyHz = 100000; +constexpr uint16_t kModuleEepromAddress = 0x50; +constexpr uint16_t kOv3640Address = 0x3c; +constexpr uint16_t kSc101iotAddress = 0x68; +constexpr int kProbeTimeoutMs = 50; + +bool checked_ioctl(int fd, unsigned long command, void* arg) +{ + if (ioctl(fd, command, arg) == 0) return true; + ESP_LOGW(TAG, "video ioctl 0x%lx failed: errno=%d", command, errno); + return false; +} +} + +bool GestureCamera::fail(const char* step, int code) +{ + failure_ = step; + failure_code_ = code; + return false; +} + +bool GestureCamera::start() +{ + failure_ = nullptr; + failure_code_ = 0; + if (pins_claimed_ || fd_ >= 0) return fail("start while resources are held"); + if (const esp_err_t error = bsp_subboard_init(); error != ESP_OK) return fail("subboard init", error); + // D2 shares GPIO33 with Serial/JTAG. The UART console remains available. + restore_usb_pad_ = usb_serial_jtag_ll_phy_is_pad_enabled(); + restore_usb_clock_ = usb_serial_jtag_ll_module_is_enabled(); + restore_usb_interrupts_ = restore_usb_clock_ ? usb_serial_jtag_ll_get_intr_ena_status() : 0; + if (restore_usb_clock_) usb_serial_jtag_ll_disable_intr_mask(USB_SERIAL_JTAG_LL_INTR_MASK); + usb_serial_jtag_ll_phy_enable_pad(false); + usb_serial_jtag_ll_enable_bus_clock(false); + pins_claimed_ = true; + gpio_reset_pin(GPIO_NUM_14); + gpio_set_direction(GPIO_NUM_14, GPIO_MODE_INPUT); + // The illuminator is active-low and must never be driven by gesture capture. + gpio_set_level(kFlash, 1); + gpio_set_direction(kFlash, GPIO_MODE_OUTPUT); + // The CameraBoard oscillator and sensor need time after the slot is claimed + // before the first SCCB access in esp_video_init. + vTaskDelay(pdMS_TO_TICKS(kPowerStabilizationMs)); + // Diagnostic only: the sensor may still be held in power-down before + // esp_video_init drives PWDN, so a missing sensor ACK here is not an error. + i2c_master_bus_handle_t sccb = bsp_subboard_get_i2c_bus(); + const auto ack = [&](uint16_t address) { + return i2c_master_probe(sccb, address, kProbeTimeoutMs) == ESP_OK ? "ack" : "no ack"; + }; + const bool eeprom_ack = i2c_master_probe(sccb, kModuleEepromAddress, kProbeTimeoutMs) == ESP_OK; + ESP_LOGI(TAG, "subboard I2C probe: module EEPROM 0x50 %s, OV3640 0x3c %s, SC101IOT 0x68 %s", + eeprom_ack ? "ack" : "no ack", ack(kOv3640Address), ack(kSc101iotAddress)); + + esp_video_init_dvp_config_t dvp{}; + dvp.sccb_config.init_sccb = false; + dvp.sccb_config.i2c_handle = sccb; + dvp.sccb_config.freq = kSccbFrequencyHz; + dvp.reset_pin = GPIO_NUM_53; + dvp.pwdn_pin = kPowerDown; + dvp.dvp_pin.data_width = CAM_CTLR_DATA_WIDTH_8; + const gpio_num_t data[] = {GPIO_NUM_16, GPIO_NUM_15, GPIO_NUM_33, GPIO_NUM_4, + GPIO_NUM_14, GPIO_NUM_12, GPIO_NUM_18, GPIO_NUM_13}; + for (unsigned i = 0; i < 8; ++i) dvp.dvp_pin.data_io[i] = data[i]; + dvp.dvp_pin.vsync_io = GPIO_NUM_55; + dvp.dvp_pin.de_io = GPIO_NUM_19; + dvp.dvp_pin.pclk_io = GPIO_NUM_17; + dvp.dvp_pin.xclk_io = GPIO_NUM_NC; + dvp.xclk_freq = 0; // CameraBoard supplies its own 24 MHz oscillator. + esp_video_init_config_t config{}; + config.dvp = &dvp; + const esp_err_t error = esp_video_init_with_flags(&config, ESP_VIDEO_INIT_FLAGS_DVP); + if (error != ESP_OK) { + ESP_LOGW(TAG, "left-slot camera unavailable: %s", esp_err_to_name(error)); + return fail("esp_video_init (sensor not detected?)", error); + } + video_initialized_ = true; + fd_ = open(ESP_VIDEO_DVP_DEVICE_NAME, O_RDWR); + // esp_video_init succeeds without registering the device when sensor + // detection fails; code reports whether the module EEPROM answered. + if (fd_ < 0 && errno == ENOENT) + return fail("no sensor detected (OV3640 0x3c / SC101IOT 0x68)", eeprom_ack ? 1 : 0); + if (fd_ < 0) return fail("open video device", errno); + v4l2_format format{}; + format.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; + // Preserve the detected sensor's configured field of view, as in mosaico_module_camera. + if (!checked_ioctl(fd_, VIDIOC_G_FMT, &format)) return fail("get format", errno); + const unsigned requested_width = format.fmt.pix.width; + const unsigned requested_height = format.fmt.pix.height; + if (!((requested_width == 1280 && requested_height == 720) || + (requested_width == 640 && requested_height == 480))) return fail("unsupported frame size"); + format.fmt.pix.pixelformat = V4L2_PIX_FMT_UYVY; + if (!checked_ioctl(fd_, VIDIOC_S_FMT, &format)) return fail("set format", errno); + if (format.fmt.pix.pixelformat != V4L2_PIX_FMT_UYVY || + format.fmt.pix.width != requested_width || format.fmt.pix.height != requested_height) return fail("format mismatch"); + width_ = format.fmt.pix.width; + height_ = format.fmt.pix.height; + stride_ = format.fmt.pix.bytesperline; + if (!stride_) stride_ = width_ * 2; + timeval timeout{0, kFrameTimeoutUs}; + if (!checked_ioctl(fd_, VIDIOC_S_DQBUF_TIMEOUT, &timeout)) return fail("set dequeue timeout", errno); + v4l2_requestbuffers request{}; + request.count = 2; + request.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; + request.memory = V4L2_MEMORY_MMAP; + if (!checked_ioctl(fd_, VIDIOC_REQBUFS, &request) || request.count != 2) + return fail("request buffers", errno); + for (unsigned i = 0; i < 2; ++i) { + v4l2_buffer buffer{}; + buffer.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; + buffer.memory = V4L2_MEMORY_MMAP; + buffer.index = i; + if (!checked_ioctl(fd_, VIDIOC_QUERYBUF, &buffer)) return fail("query buffer", errno); + buffers_[i] = mmap(nullptr, buffer.length, PROT_READ | PROT_WRITE, + MAP_SHARED, fd_, buffer.m.offset); + if (buffers_[i] == MAP_FAILED) { buffers_[i] = nullptr; return fail("map buffer", errno); } + lengths_[i] = buffer.length; + if (!checked_ioctl(fd_, VIDIOC_QBUF, &buffer)) return fail("queue buffer", errno); + } + int type = V4L2_BUF_TYPE_VIDEO_CAPTURE; + streaming_ = checked_ioctl(fd_, VIDIOC_STREAMON, &type); + if (!streaming_) return fail("stream on", errno); + // The first frames arrive only after sensor start-up. Discard one per + // buffer, allowing a few frame timeouts before reporting a dead stream. + for (unsigned primed = 0, attempts = 0; primed < 2;) { + v4l2_buffer buffer{}; + buffer.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; + buffer.memory = V4L2_MEMORY_MMAP; + if (ioctl(fd_, VIDIOC_DQBUF, &buffer) != 0) { + if (++attempts >= kPrimeAttempts) return fail("prime stream (no frames)", errno); + continue; + } + if (!checked_ioctl(fd_, VIDIOC_QBUF, &buffer)) return fail("prime stream requeue", errno); + ++primed; + } + ESP_LOGI(TAG, "camera UYVY %ux%u stride=%u buffers=2", width_, height_, stride_); + return true; +} + +bool GestureCamera::read_rgb(uint8_t* output, unsigned rotation, bool mirror) +{ + v4l2_buffer buffer{}; + buffer.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; + buffer.memory = V4L2_MEMORY_MMAP; + if (!streaming_) return fail("read while stopped"); + if (!checked_ioctl(fd_, VIDIOC_DQBUF, &buffer)) return fail("dequeue frame (no frames?)", errno); + // At most two capture buffers exist. Prefer the newer completed frame if + // inference left both queued; never accumulate camera latency across cycles. + timeval nonblocking{}; + bool timeout_ok = checked_ioctl(fd_, VIDIOC_S_DQBUF_TIMEOUT, &nonblocking); + if (timeout_ok) { + v4l2_buffer newer{}; + newer.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; + newer.memory = V4L2_MEMORY_MMAP; + if (ioctl(fd_, VIDIOC_DQBUF, &newer) == 0) { + const bool returned = checked_ioctl(fd_, VIDIOC_QBUF, &buffer); + buffer = newer; + timeout_ok = returned; + } + timeval blocking{0, kFrameTimeoutUs}; + timeout_ok = checked_ioctl(fd_, VIDIOC_S_DQBUF_TIMEOUT, &blocking) && timeout_ok; + } + bool converted = false; + if (buffer.index < 2 && buffers_[buffer.index] && !(buffer.flags & V4L2_BUF_FLAG_ERROR)) { + const size_t bytes = buffer.bytesused ? buffer.bytesused : lengths_[buffer.index]; + if (bytes <= lengths_[buffer.index]) + converted = prepare_gesture_image(static_cast(buffers_[buffer.index]), + bytes, width_, height_, stride_, output, rotation, mirror); + } + const bool returned = checked_ioctl(fd_, VIDIOC_QBUF, &buffer); + if (!converted) return fail("convert frame", int(buffer.bytesused)); + if (!returned || !timeout_ok) return fail("requeue frame", errno); + return true; +} + +bool GestureCamera::stop() +{ + if (streaming_) { + int type = V4L2_BUF_TYPE_VIDEO_CAPTURE; + if (!checked_ioctl(fd_, VIDIOC_STREAMOFF, &type)) return fail("stream off", errno); + streaming_ = false; + } + if (fd_ >= 0) { + for (unsigned i = 0; i < 2; ++i) { + if (buffers_[i]) { + if (munmap(buffers_[i], lengths_[i]) != 0) return fail("unmap buffer", errno); + buffers_[i] = nullptr; + } + } + if (close(fd_) != 0) return fail("close video device", errno); + fd_ = -1; + } + if (video_initialized_) { + if (const esp_err_t error = esp_video_deinit_with_flags(ESP_VIDEO_INIT_FLAGS_DVP); error != ESP_OK) + return fail("esp_video_deinit", error); + video_initialized_ = false; + } + if (pins_claimed_) { + gpio_set_level(kPowerDown, 1); + gpio_set_direction(kPowerDown, GPIO_MODE_OUTPUT); + gpio_set_level(kFlash, 1); + // DVP is stopped before restoring the shared pads and EEPROM address line. + gpio_reset_pin(GPIO_NUM_33); + if (restore_usb_clock_) usb_serial_jtag_ll_enable_bus_clock(true); + usb_serial_jtag_ll_phy_enable_pad(restore_usb_pad_); + if (restore_usb_clock_ && restore_usb_interrupts_) + usb_serial_jtag_ll_ena_intr_mask(restore_usb_interrupts_); + if (const esp_err_t error = bsp_subboard_apply_address_select(BSP_SUBBOARD_SLOT_LEFT); error != ESP_OK) + return fail("restore address select", error); + pins_claimed_ = false; + } + return true; +} + +} // namespace usage_panel::mosaico diff --git a/firmware/targets/esp_mosaico/main/gesture_camera.hpp b/firmware/targets/esp_mosaico/main/gesture_camera.hpp new file mode 100644 index 0000000..a16ea83 --- /dev/null +++ b/firmware/targets/esp_mosaico/main/gesture_camera.hpp @@ -0,0 +1,43 @@ +#pragma once + +#include "gesture_image.hpp" + +#include +#include + +namespace usage_panel::mosaico { + +/** Owns the left-slot CameraBoard (OV3640 or SC101IOT) and its V4L2 buffers on the vision worker only. */ +class GestureCamera { +public: + bool start(); + /** Copies/converts one frame and requeues it before returning to the caller. */ + bool read_rgb(uint8_t* output, unsigned rotation, bool mirror); + /** Packed RGB dimensions for the active capture format and rotation. */ + GestureImageSize image_size(unsigned rotation) const { return gesture_image_size(width_, height_, rotation); } + /** Returns false if resources cannot be safely released; do not restart afterward. */ + bool stop(); + /** Names the step of the most recent failed call, or nullptr; code is errno or esp_err_t. */ + const char* failure() const { return failure_; } + int failure_code() const { return failure_code_; } + +private: + bool fail(const char* step, int code = 0); + + const char* failure_ = nullptr; + int failure_code_ = 0; + int fd_ = -1; + bool pins_claimed_ = false; + bool video_initialized_ = false; + bool streaming_ = false; + bool restore_usb_pad_ = false; + bool restore_usb_clock_ = false; + uint32_t restore_usb_interrupts_ = 0; + void* buffers_[2]{}; + size_t lengths_[2]{}; + unsigned width_ = 0; + unsigned height_ = 0; + unsigned stride_ = 0; +}; + +} // namespace usage_panel::mosaico diff --git a/firmware/targets/esp_mosaico/main/gesture_image.cpp b/firmware/targets/esp_mosaico/main/gesture_image.cpp new file mode 100644 index 0000000..45bbc3a --- /dev/null +++ b/firmware/targets/esp_mosaico/main/gesture_image.cpp @@ -0,0 +1,58 @@ +#include "gesture_image.hpp" + +#include + +namespace usage_panel::mosaico { +namespace { +uint8_t channel(int value) +{ + return static_cast(std::clamp(value / 256, 0, 255)); +} +} + +GestureImageSize gesture_image_size(unsigned width, unsigned height, unsigned turns) +{ + if (width < 2 || width > 4096 || (width & 1) || height == 0 || height > 4096 || turns > 3) + return {}; + GestureImageSize size{kGestureImageWidth, kGestureImageHeight}; + if (width * kGestureImageHeight >= height * kGestureImageWidth) + size.height = std::max(1U, height * kGestureImageWidth / width); + else + size.width = std::max(1U, width * kGestureImageHeight / height); + if (turns & 1) std::swap(size.width, size.height); + return size; +} + +bool prepare_gesture_image(const uint8_t* src, size_t size, unsigned width, + unsigned height, unsigned stride, uint8_t* dst, + unsigned turns, bool mirror) +{ + if (!src || !dst || width < 2 || width > 4096 || height == 0 || height > 4096 || + (width & 1) || stride < width * 2 || size < size_t(stride) * height || turns > 3) + return false; + const auto size_out = gesture_image_size(width, height, 0); + const unsigned out_width = turns & 1 ? size_out.height : size_out.width; + for (unsigned y = 0; y < size_out.height; ++y) { + for (unsigned x = 0; x < size_out.width; ++x) { + const unsigned sx = x * width / size_out.width; + const unsigned sy = y * height / size_out.height; + const uint8_t* p = src + size_t(sy) * stride + (sx & ~1U) * 2; + const int luma = std::max(0, int(p[(sx & 1) ? 3 : 1]) - 16); + const int u = int(p[0]) - 128; + const int v = int(p[2]) - 128; + const unsigned mx = mirror ? size_out.width - 1 - x : x; + unsigned ox = mx, oy = y; + if (turns == 1) { ox = size_out.height - 1 - y; oy = mx; } + if (turns == 2) { ox = size_out.width - 1 - mx; oy = size_out.height - 1 - y; } + if (turns == 3) { ox = y; oy = size_out.width - 1 - mx; } + uint8_t* rgb = dst + (oy * out_width + ox) * 3; + // UYVY is converted with limited-range BT.601 YCbCr coefficients. + rgb[0] = channel(298 * luma + 409 * v + 128); + rgb[1] = channel(298 * luma - 100 * u - 208 * v + 128); + rgb[2] = channel(298 * luma + 516 * u + 128); + } + } + return true; +} + +} // namespace usage_panel::mosaico diff --git a/firmware/targets/esp_mosaico/main/gesture_image.hpp b/firmware/targets/esp_mosaico/main/gesture_image.hpp new file mode 100644 index 0000000..3c31a9f --- /dev/null +++ b/firmware/targets/esp_mosaico/main/gesture_image.hpp @@ -0,0 +1,46 @@ +#pragma once + +#include +#include + +namespace usage_panel::mosaico { + +constexpr unsigned kGestureImageWidth = 320; +constexpr unsigned kGestureImageHeight = 240; +constexpr size_t kGestureImageBytes = kGestureImageWidth * kGestureImageHeight * 3; + +struct GestureImageSize { + unsigned width; + unsigned height; +}; + +/** + * Fits the complete frame within 320x240 before rotation, preserving its aspect ratio + * to integer-pixel precision. Odd quarter-turn counts swap the output dimensions. + * @param width Source width in pixels; must be even and within 2..4096. + * @param height Source height in pixels; must be within 1..4096. + * @param quarter_turns Clockwise rotation in 90-degree steps; valid range is 0..3. + * @return Output dimensions, or zero dimensions if any argument is invalid. + */ +GestureImageSize gesture_image_size(unsigned width, unsigned height, unsigned quarter_turns); + +/** + * Converts the complete UYVY frame to RGB888 using gesture_image_size() dimensions, + * then applies horizontal mirroring and clockwise rotation. Output has no row padding. + * Both buffers remain caller-owned and must not overlap; neither pointer is retained. + * @param src Non-null source buffer containing packed UYVY rows with optional row padding. + * @param size Source buffer size in bytes; must be at least stride * height. + * @param width Source width in pixels; must be even and within 2..4096. + * @param height Source height in pixels; must be within 1..4096. + * @param stride Source row stride in bytes; must be at least width * 2. + * @param dst Non-null output buffer with at least kGestureImageBytes capacity; capacity is not checked. + * @param quarter_turns Clockwise rotation in 90-degree steps; valid range is 0..3. + * @param mirror Whether to mirror horizontally before rotation. + * @return True after conversion; false for null pointers, invalid dimensions or rotation, + * insufficient source size, or an invalid stride. Validation failure leaves dst unchanged. + */ +bool prepare_gesture_image(const uint8_t* src, size_t size, unsigned width, + unsigned height, unsigned stride, uint8_t* dst, + unsigned quarter_turns, bool mirror); + +} // namespace usage_panel::mosaico diff --git a/firmware/targets/esp_mosaico/main/gesture_input.cpp b/firmware/targets/esp_mosaico/main/gesture_input.cpp new file mode 100644 index 0000000..9b84308 --- /dev/null +++ b/firmware/targets/esp_mosaico/main/gesture_input.cpp @@ -0,0 +1,312 @@ +#include "gesture_input.hpp" +#include "sdkconfig.h" + +#include "esp_log.h" +#include "esp_timer.h" +#include "freertos/task.h" + +#if CONFIG_MOSAICO_GESTURE_INPUT +#include +#include +#include +#include +#include "esp_heap_caps.h" +#include "gesture_camera.hpp" +#include "gesture_image.hpp" +#include "hand_detect.hpp" +#include "hand_gesture_recognition.hpp" +#endif + +namespace usage_panel::mosaico { +#if CONFIG_MOSAICO_GESTURE_INPUT +static_assert(CONFIG_MOSAICO_CAMERA_ROTATION % 90 == 0, "Camera rotation must be a multiple of 90 degrees"); +#endif +namespace { +// context_ layout: bits 0..2 hold the flags below, bits 3..4 the display rotation, +// bits 5..6 the page, bit 7 marks an accepted gesture and bits 8..31 count generations. +constexpr uint32_t kEnabled = 1; +constexpr uint32_t kAllowed = 2; +constexpr uint32_t kAsleep = 4; +constexpr uint32_t kContextMask = 0x7f; +constexpr uint32_t kAcceptedGesture = 0x80; +constexpr uint32_t kGeneration = 0x100; +constexpr char TAG[] = "gesture_input"; +// Transient start-up failures are retried as in esp-vision's camera init. +constexpr unsigned kCameraStartAttempts = 3; +constexpr uint32_t kCameraRetryDelayMs = 100; +} + +bool GestureInput::begin() +{ +#if CONFIG_MOSAICO_GESTURE_INPUT + queue_ = xQueueCreate(1, sizeof(GestureEvent)); + if (!queue_) { status_ = GestureStatus::MemoryError; return false; } + if (xTaskCreate(task_entry, "hand_input", 8192, this, 3, nullptr) != pdPASS) { + vQueueDelete(queue_); + queue_ = nullptr; + status_ = GestureStatus::MemoryError; + return false; + } +#endif + return true; +} + +void GestureInput::configure(bool enabled, bool allowed, bool asleep, Page page, ScreenRotation rotation, + bool accepted_gesture) +{ + const uint32_t flags = (enabled ? kEnabled : 0) | (allowed ? kAllowed : 0) | + (asleep ? kAsleep : 0) | (uint32_t(rotation) << 3) | (uint32_t(page) << 5); + const uint32_t old = context_.load(); + if ((old & kContextMask) != flags || accepted_gesture) { + context_ = ((old + kGeneration) & ~(kContextMask | kAcceptedGesture)) | flags | + (accepted_gesture ? kAcceptedGesture : 0); + progress_ = 0; + } +} + +void GestureInput::invalidate() +{ + context_ = (context_.load() + kGeneration) & ~kAcceptedGesture; + progress_ = 0; +} + +bool GestureInput::take(GestureEvent& event, uint64_t now_ms) +{ + if (!queue_ || xQueueReceive(queue_, &event, 0) != pdTRUE) return false; + const uint32_t context = context_.load(); + return event.context == context && (context & (kEnabled | kAllowed)) == (kEnabled | kAllowed) && + now_ms >= event.time_ms && now_ms - event.time_ms <= 300; +} + +void GestureInput::task_entry(void* context) +{ + static_cast(context)->run(); + vTaskDelete(nullptr); +} + +void GestureInput::run() +{ +#if CONFIG_MOSAICO_GESTURE_INPUT + GestureCamera camera; + GestureTracker tracker; + std::unique_ptr detector; + std::unique_ptr recognizer; + uint8_t* rgb = nullptr; + uint32_t previous_context = UINT32_MAX; + bool active = false; + bool fault = false; + bool fatal_cleanup = false; + unsigned capture_failures = 0; + uint64_t next_report = 0; + HandObservation previous_observation{}; + unsigned measured_frames = 0; + uint64_t measured_us = 0; + uint64_t capture_us = 0, detect_us = 0, classify_us = 0, feedback_us = 0; + unsigned warmup = 0; + // Retain failure details for periodic diagnostics while recognition is disabled. + // Disabling capture clears recoverable faults; cleanup faults persist until reboot. + const char* failure = nullptr; + int failure_code = 0; + uint64_t next_failure_log = 0; + const auto record = [&](const char* step, int code) { + failure = step ? step : "unknown"; + failure_code = code; + next_failure_log = 0; + }; + + for (;;) { + const uint32_t context = context_.load(); + const bool enabled = context & kEnabled; + const bool allowed = context & kAllowed; + if (context != previous_context) { + tracker.reset((context & kAcceptedGesture) != 0); + progress_ = 0; + previous_context = context; + } + if (!enabled || !allowed || fault) { + if (active || rgb) { + if (!camera.stop()) { + ESP_LOGE(TAG, "camera cleanup failed; recognition disabled until reboot"); + record(camera.failure(), camera.failure_code()); + fatal_cleanup = true; + fault = true; + } + recognizer.reset(); + detector.reset(); + heap_caps_free(rgb); + rgb = nullptr; + active = false; + } + progress_ = 0; + const uint64_t now_ms = esp_timer_get_time() / 1000; + if (fault && failure && now_ms >= next_failure_log) { + ESP_LOGW(TAG, "capture unavailable: %s (code %d)", failure, failure_code); + next_failure_log = now_ms + 5000; + } + if (!enabled && !fatal_cleanup) { fault = false; failure = nullptr; status_ = GestureStatus::Off; } + else if (fatal_cleanup) status_ = GestureStatus::Fault; + else if (!fault) status_ = GestureStatus::Paused; + vTaskDelay(pdMS_TO_TICKS(100)); + continue; + } + if (!active) { + status_ = GestureStatus::Starting; + // ESP-DL allocates model arenas during construction. Keep a conservative + // margin for both arenas, two 1280x720 UYVY buffers and the concurrently running UI. + if (heap_caps_get_largest_free_block(MALLOC_CAP_SPIRAM) < 9 * 1024 * 1024 || + heap_caps_get_free_size(MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT) < 96 * 1024) { + // Code is the largest free PSRAM block in KiB. + record("memory precheck", + int(heap_caps_get_largest_free_block(MALLOC_CAP_SPIRAM) / 1024)); + status_ = GestureStatus::MemoryError; + fault = true; + continue; + } + rgb = static_cast(heap_caps_malloc(kGestureImageBytes, MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT)); + if (!rgb) { + record("frame buffer allocation", int(kGestureImageBytes)); + status_ = GestureStatus::MemoryError; + fault = true; + continue; + } + bool started = false; + for (unsigned attempt = 1; attempt <= kCameraStartAttempts; ++attempt) { + if (camera.start()) { started = true; break; } + record(camera.failure(), camera.failure_code()); + ESP_LOGW(TAG, "camera start %u/%u failed: %s (code %d)", attempt, + kCameraStartAttempts, failure, failure_code); + if (!camera.stop()) { + record(camera.failure(), camera.failure_code()); + fatal_cleanup = true; + break; + } + if (attempt < kCameraStartAttempts) vTaskDelay(pdMS_TO_TICKS(kCameraRetryDelayMs)); + } + if (!started) { + status_ = fatal_cleanup ? GestureStatus::Fault : GestureStatus::CameraError; + fault = true; + continue; + } + detector.reset(new (std::nothrow) HandDetect(HandDetect::ESPDET_PICO_224_224_HAND, false)); + // ESP-DL 3.3.11 requires the classifier model to be loaded before destruction. + recognizer.reset(new (std::nothrow) HandGestureCls(HandGestureCls::MOBILENETV2_0_5_S8_V1, false)); + if (!detector || !recognizer) { + record("model allocation", 0); + status_ = GestureStatus::MemoryError; + fault = true; + continue; + } + detector->set_score_thr(0.5F); + tracker.reset(); + active = true; + capture_failures = 0; + warmup = 3; + status_ = GestureStatus::Ready; + ESP_LOGI(TAG, "local gestures ready; input starts after the view is clear of hands"); + } + const uint64_t started_us = esp_timer_get_time(); + // Display rotation maps logical pixels to physical coordinates; camera + // observations require the inverse transform after the mounting correction. + const unsigned display_rotation = (context >> 3) & 3; + const unsigned rotation = (CONFIG_MOSAICO_CAMERA_ROTATION / 90 + 4 - display_rotation) % 4; +#ifdef CONFIG_MOSAICO_CAMERA_MIRROR + const bool mirror = true; +#else + const bool mirror = false; +#endif + if (!camera.read_rgb(rgb, rotation, mirror)) { + tracker.reset(); + progress_ = 0; + if (++capture_failures >= 3) { + record(camera.failure(), camera.failure_code()); + fault = true; + status_ = GestureStatus::CameraError; + } + vTaskDelay(pdMS_TO_TICKS(20)); + continue; + } + capture_failures = 0; + if (warmup) { --warmup; continue; } + const auto image_size = camera.image_size(rotation); + dl::image::img_t image{rgb, + static_cast(image_size.width), + static_cast(image_size.height), + dl::image::DL_IMAGE_PIX_TYPE_RGB888}; + const uint64_t captured_us = esp_timer_get_time(); + const auto& hands = detector->run(image); + const uint64_t detected_us = esp_timer_get_time(); + HandObservation observation{}; + observation.count = hands.size(); + if (hands.size() == 1) { + const auto& hand = hands.front(); + if (hand.box.size() == 4) { + observation.x = (hand.box[0] + hand.box[2]) * 0.5F / image.width; + observation.y = (hand.box[1] + hand.box[3]) * 0.5F / image.height; + observation.width = float(hand.box[2] - hand.box[0]) / image.width; + observation.height = float(hand.box[3] - hand.box[1]) / image.height; + observation.score = hand.score; + if (!(context & kAsleep) && hand.score >= 0.65F && + hand_is_steady(previous_observation, observation) && + hand.box[0] >= 0 && hand.box[1] >= 0 && hand.box[2] <= image.width && + hand.box[3] <= image.height && hand.box[2] > hand.box[0] && hand.box[3] > hand.box[1]) { + const auto classes = recognizer->run_crop(image, hand.box); + for (const auto& result : classes) { + if (result.cat_name && (std::strcmp(result.cat_name, "ok") == 0 || + std::strcmp(result.cat_name, "like") == 0)) + observation.ok_score = std::max(observation.ok_score, result.score); + } + } + } + } + previous_observation = observation; + const uint64_t finished_us = esp_timer_get_time(); + // A context change during the frame discards it as input; it is still previewed. + const bool stale = context != context_.load(); + const GestureAction action = stale ? GestureAction::None + : tracker.update(observation, finished_us / 1000); + if (!stale) progress_ = tracker.hold_progress(); + if (action != GestureAction::None) { + ESP_LOGI(TAG, "tracker action=%u", unsigned(action)); + const GestureEvent event{action, context, finished_us / 1000}; + xQueueSend(queue_, &event, 0); + } +#if CONFIG_MOSAICO_CAMERA_PREVIEW + // The sink waits for the display lock; events are queued first because they expire. + if (preview_sink_) preview_sink_(preview_context_, rgb, image.width, image.height, observation); +#endif + // Calibration aid: every observation with a hand, in per mille with the time given to + // the tracker, so that a captured log can be replayed through GestureTracker. + if (observation.count) { + ESP_LOGI(TAG, "hand t=%u n=%u s=%d c=%d,%d wh=%d,%d ok=%d", + unsigned(finished_us / 1000), observation.count, int(observation.score * 1000), + int(observation.x * 1000), int(observation.y * 1000), int(observation.width * 1000), + int(observation.height * 1000), int(observation.ok_score * 1000)); + } + const uint64_t feedback_done_us = esp_timer_get_time(); + capture_us += captured_us - started_us; + detect_us += detected_us - captured_us; + classify_us += finished_us - detected_us; + feedback_us += feedback_done_us - finished_us; + measured_us += finished_us - started_us; + ++measured_frames; + if (finished_us / 1000 >= next_report) { + ESP_LOGI(TAG, "frames=%u mean_cycle_ms=%llu internal_free=%u psram_free=%u stack_free=%u", + measured_frames, static_cast(measured_us / measured_frames / 1000), + unsigned(heap_caps_get_free_size(MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT)), + unsigned(heap_caps_get_free_size(MALLOC_CAP_SPIRAM)), unsigned(uxTaskGetStackHighWaterMark(nullptr))); + ESP_LOGI(TAG, "mean_stage_us capture_convert=%llu detect=%llu classify=%llu feedback=%llu", + static_cast(capture_us / measured_frames), + static_cast(detect_us / measured_frames), + static_cast(classify_us / measured_frames), + static_cast(feedback_us / measured_frames)); + capture_us = detect_us = classify_us = feedback_us = 0; + measured_us = 0; + measured_frames = 0; + next_report = finished_us / 1000 + 10000; + } + vTaskDelay(pdMS_TO_TICKS(20)); + } +#endif +} + +} // namespace usage_panel::mosaico diff --git a/firmware/targets/esp_mosaico/main/gesture_input.hpp b/firmware/targets/esp_mosaico/main/gesture_input.hpp new file mode 100644 index 0000000..624bc72 --- /dev/null +++ b/firmware/targets/esp_mosaico/main/gesture_input.hpp @@ -0,0 +1,54 @@ +#pragma once + +#include +#include +#include "freertos/FreeRTOS.h" +#include "freertos/queue.h" +#include "gesture_tracker.hpp" +#include "orientation.hpp" + +namespace usage_panel::mosaico { + +enum class GestureStatus : uint8_t { Off, Starting, Ready, Paused, CameraError, MemoryError, Fault }; + +struct GestureEvent { + GestureAction action = GestureAction::None; + uint32_t context = 0; + uint64_t time_ms = 0; +}; + +/** Worker-owned camera/model pipeline. The application owns context and event consumption. */ +class GestureInput { +public: + /** Receives each model input image and its observation on the vision worker. */ + using PreviewSink = void (*)(void* context, const uint8_t* rgb, unsigned width, unsigned height, + const HandObservation& hand); + + /** Must be called before begin(); only used with CONFIG_MOSAICO_CAMERA_PREVIEW. */ + void set_preview_sink(PreviewSink sink, void* context) + { + preview_sink_ = sink; + preview_context_ = context; + } + bool begin(); + /** Main task only. accepted_gesture marks a context change caused by a consumed gesture. */ + void configure(bool enabled, bool allowed, bool asleep, Page page, ScreenRotation rotation, + bool accepted_gesture = false); + /** Cancels pending recognition after another input is accepted. Main task only. */ + void invalidate(); + bool take(GestureEvent& event, uint64_t now_ms); + GestureStatus status() const { return status_.load(); } + uint8_t progress() const { return progress_.load(); } + +private: + static void task_entry(void* context); + void run(); + QueueHandle_t queue_ = nullptr; + std::atomic context_{0}; + std::atomic status_{GestureStatus::Off}; + std::atomic progress_{0}; + PreviewSink preview_sink_ = nullptr; + void* preview_context_ = nullptr; +}; + +} // namespace usage_panel::mosaico diff --git a/firmware/targets/esp_mosaico/main/gesture_tracker.cpp b/firmware/targets/esp_mosaico/main/gesture_tracker.cpp new file mode 100644 index 0000000..b336e9b --- /dev/null +++ b/firmware/targets/esp_mosaico/main/gesture_tracker.cpp @@ -0,0 +1,206 @@ +#include "gesture_tracker.hpp" + +#include +#include + +namespace usage_panel::mosaico { +namespace { +// Short detection gaps preserve the current swipe or pose hold. Longer gaps restart +// tracking at the next usable sample. The limit allows for classification and preview latency. +constexpr uint64_t kMaxSampleGapMs = 400; +// After a swipe or wave, motion must cease for this long before the next swipe or wave, so the +// follow-through, an immediate return and the rest of a longer wave cannot repeat it. +constexpr uint64_t kSettleMs = 400; +// An opposite swipe within this interval is treated as the return stroke. +constexpr uint64_t kReturnStrokeMs = 1500; +constexpr uint64_t kOkHoldMs = 500; +// A completed hold requires at least three samples because each gap is shorter than the hold. +static_assert(kMaxSampleGapMs < kOkHoldMs); +// Frame-to-frame movement up to this distance is treated as detector jitter. +constexpr float kRestStep = 0.03F; +// Larger frame-to-frame movement skips pose classification to preserve motion sampling cadence. +constexpr float kSteadyStep = 0.05F; +// Each of the three alternating horizontal wave legs must cover this fraction of the image width. +constexpr float kWaveLeg = 0.12F; + +GestureAction opposite(GestureAction action) +{ + switch (action) { + case GestureAction::SwipeLeft: return GestureAction::SwipeRight; + case GestureAction::SwipeRight: return GestureAction::SwipeLeft; + case GestureAction::SwipeUp: return GestureAction::SwipeDown; + case GestureAction::SwipeDown: return GestureAction::SwipeUp; + default: return GestureAction::None; + } +} +} + +GestureEffect route_gesture(GestureAction action, bool allowed, bool asleep) +{ + if (!allowed || action == GestureAction::None) return GestureEffect::None; + if (action == GestureAction::EnterClock) return asleep ? GestureEffect::None : GestureEffect::Clock; + if (asleep) return GestureEffect::Wake; + // Swipes and waves share forward-only navigation. + return GestureEffect::Next; +} + +bool hand_is_steady(const HandObservation& previous, const HandObservation& current) +{ + return previous.count == 1 && current.count == 1 && + std::fabs(current.x - previous.x) <= kSteadyStep && + std::fabs(current.y - previous.y) <= kSteadyStep; +} + +void GestureTracker::reset(bool accepted_gesture) +{ + const uint64_t last_action = last_action_; + const uint64_t last_motion = last_motion_; + const GestureAction last_swipe = last_swipe_; + const bool waiting_release = waiting_release_; + const bool settling = settling_; + *this = GestureTracker{}; + if (accepted_gesture) { + last_action_ = last_action; + last_motion_ = last_motion; + last_swipe_ = last_swipe; + waiting_release_ = waiting_release; + settling_ = settling; + } +} + +void GestureTracker::begin_stroke(const HandObservation& h, uint64_t now) +{ + started_ = now; + samples_ = 1; + start_x_ = h.x; + start_y_ = h.y; + path_x_ = path_y_ = max_dx_ = max_dy_ = 0; +} + +GestureAction GestureTracker::update(const HandObservation& h, uint64_t now) +{ + if (h.count > 1) { + // Multiple hands invalidate ownership and require release before another swipe or pose. + absent_ = false; + tracking_ = holding_ok_ = false; + waiting_release_ = true; + progress_ = 0; + return GestureAction::None; + } + const bool usable = h.count == 1 && std::isfinite(h.x) && std::isfinite(h.y) && + std::isfinite(h.width) && std::isfinite(h.height) && std::isfinite(h.score) && + h.score >= 0.65F && h.x >= 0 && h.x <= 1 && h.y >= 0 && h.y <= 1 && + h.width >= 0.06F && h.height >= 0.06F && h.width <= 0.9F && h.height <= 0.95F; + if (!usable) { + // Missed and low-confidence frames are dropouts, not new input. + if (h.count == 0) { + if (!absent_) { absent_ = true; absent_since_ = now; } + if (now - absent_since_ >= 300 && now - last_action_ >= 500) + waiting_release_ = false; + } else { + // A weakly detected hand is still present and cannot count as a release. Motion + // blur lowers the score, so it does not count as rest either. + absent_ = false; + last_motion_ = now; + } + if (tracking_ && now - last_sample_ > kMaxSampleGapMs) tracking_ = holding_ok_ = false; + if (!holding_ok_) progress_ = 0; + return GestureAction::None; + } + absent_ = false; + progress_ = 0; + + const bool continuous = tracking_ && now >= last_sample_ && now - last_sample_ <= kMaxSampleGapMs && + std::fabs(h.x - previous_.x) <= std::max(0.18F, (h.width + previous_.width) * 0.75F) && + std::fabs(h.y - previous_.y) <= std::max(0.18F, (h.height + previous_.height) * 0.75F); + const float step_x = continuous ? h.x - previous_.x : 0; + const float step_y = continuous ? h.y - previous_.y : 0; + tracking_ = true; + previous_ = h; + last_sample_ = now; + if (std::fabs(step_x) > kRestStep || std::fabs(step_y) > kRestStep) last_motion_ = now; + const bool rested = now - last_motion_ >= kSettleMs; + + if (!continuous) holding_ok_ = false; + if (!continuous || rested) { + leg_x_ = 0; + legs_ = 0; + } + if (settling_ && rested) settling_ = false; + if (!continuous || rested) { + begin_stroke(h, now); + } else { + ++samples_; + path_x_ += std::fabs(step_x); + path_y_ += std::fabs(step_y); + max_dx_ = std::max(max_dx_, std::fabs(h.x - start_x_)); + max_dy_ = std::max(max_dy_, std::fabs(h.y - start_y_)); + // A hand still at its origin has not started a stroke. The origin follows it, so the + // stroke window opens when the hand starts to move rather than when it appeared. + if (max_dx_ <= kRestStep && max_dy_ <= kRestStep) begin_stroke(h, now); + } + if (std::fabs(step_x) > kRestStep) { + if (leg_x_ * step_x < 0) { + if (std::fabs(leg_x_) >= kWaveLeg) ++legs_; + leg_x_ = 0; + } + leg_x_ += step_x; + } + const float dx = h.x - start_x_; + const float dy = h.y - start_y_; + + GestureAction action = GestureAction::None; + if (!settling_) { + // Hand withdrawal after a pose or touch can resemble a swipe, so swipes require release. + // Waves use alternating horizontal motion as their confirmation criterion. + if (!waiting_release_ && samples_ >= 3 && now - started_ >= 150 && now - started_ <= 900) { + if (std::fabs(dx) >= 0.25F && std::fabs(dx) >= 2 * max_dy_ && + max_dy_ <= 0.15F && std::fabs(dx) >= path_x_ * 0.75F) { + action = dx < 0 ? GestureAction::SwipeLeft : GestureAction::SwipeRight; + } else if (std::fabs(dy) >= 0.25F && std::fabs(dy) >= 2 * max_dx_ && + max_dx_ <= 0.15F && std::fabs(dy) >= path_y_ * 0.75F) { + action = dy < 0 ? GestureAction::SwipeUp : GestureAction::SwipeDown; + } + } + if (action != GestureAction::None && action == opposite(last_swipe_) && + now - last_action_ < kReturnStrokeMs) { + begin_stroke(h, now); + holding_ok_ = false; + return GestureAction::None; + } + if (action == GestureAction::None && legs_ >= 2 && std::fabs(leg_x_) >= kWaveLeg && + path_x_ >= 2 * path_y_) { + action = GestureAction::Wave; + } + } + + if (!waiting_release_ && std::isfinite(h.ok_score) && h.ok_score >= 0.85F) { + if (!holding_ok_ || std::fabs(h.x - ok_x_) > 0.08F || std::fabs(h.y - ok_y_) > 0.08F) { + holding_ok_ = true; + ok_since_ = now; + ok_x_ = h.x; + ok_y_ = h.y; + } + progress_ = static_cast(std::min(100, (now - ok_since_) * 100 / kOkHoldMs)); + if (now - ok_since_ >= kOkHoldMs) action = GestureAction::EnterClock; + } else { + holding_ok_ = false; + } + if (action != GestureAction::None) { + // Clock poses require release before another swipe or pose to suppress withdrawal-triggered wakeups. + waiting_release_ = action == GestureAction::EnterClock; + last_swipe_ = waiting_release_ ? GestureAction::None : action; + last_action_ = last_motion_ = now; + settling_ = !waiting_release_; + holding_ok_ = false; + progress_ = 0; + leg_x_ = 0; + legs_ = 0; + } else if (now - started_ > 900) { + // Restart expired candidates so subsequent motion can form a new stroke. + begin_stroke(h, now); + } + return action; +} + +} // namespace usage_panel::mosaico diff --git a/firmware/targets/esp_mosaico/main/gesture_tracker.hpp b/firmware/targets/esp_mosaico/main/gesture_tracker.hpp new file mode 100644 index 0000000..d4984a2 --- /dev/null +++ b/firmware/targets/esp_mosaico/main/gesture_tracker.hpp @@ -0,0 +1,78 @@ +#pragma once + +#include + +namespace usage_panel::mosaico { + +enum class GestureAction : uint8_t { None, SwipeLeft, SwipeRight, EnterClock, SwipeUp, SwipeDown, Wave }; +enum class GestureEffect : uint8_t { None, Wake, Next, Clock }; + +/** Applies application input policy before any page or screensaver mutation. */ +GestureEffect route_gesture(GestureAction action, bool allowed, bool asleep); + +/** A detection in normalized, upright screen coordinates; count includes all hands. */ +struct HandObservation { + unsigned count = 0; + float x = 0; + float y = 0; + float width = 0; + float height = 0; + float score = 0; + float ok_score = 0; // Confidence of a clock pose: OK or thumbs-up. +}; + +/** + * Returns whether consecutive single-hand observations satisfy the pose-classification + * motion threshold. Skipping classification during motion preserves trajectory sampling cadence. + */ +bool hand_is_steady(const HandObservation& previous, const HandObservation& current); + +/** + * Time-based single-hand recognition. A straight stroke is a swipe; a horizontal + * back-and-forth motion too small for a swipe is a Wave. One burst of motion issues one + * command: the next requires the hand to rest or leave the view, and a stroke opposite to + * the previous swipe is ignored briefly as the hand returning. + * After EnterClock, several visible hands or reset(), swipes and poses require sustained + * absence of hands first; a wave does not. + */ +class GestureTracker { +public: + GestureAction update(const HandObservation& hand, uint64_t now_ms); + /** + * Invalidates prior observations; swipes and poses then require sustained absence of hands. + * An accepted gesture's context change may preserve its settling and release state. + */ + void reset(bool accepted_gesture = false); + uint8_t hold_progress() const { return progress_; } + +private: + void begin_stroke(const HandObservation& hand, uint64_t now_ms); + + bool waiting_release_ = true; + bool absent_ = false; + bool tracking_ = false; // previous_ and last_sample_ describe the same hand. + bool holding_ok_ = false; + bool settling_ = false; // A swipe or wave was issued and its motion has not ceased. + uint64_t absent_since_ = 0; + uint64_t last_action_ = 0; + uint64_t last_motion_ = 0; + GestureAction last_swipe_ = GestureAction::None; // None after EnterClock. + uint64_t started_ = 0; + uint64_t last_sample_ = 0; + uint64_t ok_since_ = 0; + unsigned samples_ = 0; + uint8_t progress_ = 0; + HandObservation previous_{}; + float start_x_ = 0; + float start_y_ = 0; + float path_x_ = 0; + float path_y_ = 0; + float max_dx_ = 0; + float max_dy_ = 0; + float leg_x_ = 0; // Signed horizontal travel since the last reversal. + unsigned legs_ = 0; // Completed horizontal legs long enough to belong to a wave. + float ok_x_ = 0; + float ok_y_ = 0; +}; + +} // namespace usage_panel::mosaico diff --git a/firmware/targets/esp_mosaico/main/idf_component.yml b/firmware/targets/esp_mosaico/main/idf_component.yml index de5196b..2453d44 100644 --- a/firmware/targets/esp_mosaico/main/idf_component.yml +++ b/firmware/targets/esp_mosaico/main/idf_component.yml @@ -1,6 +1,11 @@ dependencies: - # Display support is required by this target. Declare its dependencies here - # because BSP config rules cannot be evaluated before sdkconfig exists. - # Keeping both dependencies in the first pass preserves the locked graph. - espressif/esp_lvgl_adapter: "0.6.2" + # The BSP pulls LVGL only when its display support is enabled, and that rule + # cannot be evaluated on a clean tree, where no sdkconfig exists yet. Naming + # LVGL here resolves it in the Component Manager's first pass so components + # that require it by name are satisfiable from the first configure. lvgl/lvgl: "^9.5.0,<9.6.0" + # The display adapter is required before BSP sdkconfig rules can be evaluated. + espressif/esp_lvgl_adapter: "0.6.2" + espressif/esp_video: "2.5.0" + espressif/hand_detect: "0.2.0" + espressif/hand_gesture_recognition: "0.2.0" diff --git a/firmware/targets/esp_mosaico/main/panel_settings.cpp b/firmware/targets/esp_mosaico/main/panel_settings.cpp new file mode 100644 index 0000000..91e7c23 --- /dev/null +++ b/firmware/targets/esp_mosaico/main/panel_settings.cpp @@ -0,0 +1,51 @@ +#include "panel_settings.hpp" +#include "nvs.h" + +namespace usage_panel::mosaico { +namespace { +constexpr char kNamespace[] = "mosaico"; +constexpr char kSettingsKey[] = "settings"; +// One NVS integer keeps display and capture preferences in the same storage update. +// Bits 0..6 hold brightness, bit 7 enables capture, and bits 8..31 hold timeout seconds. +constexpr uint32_t kBrightnessMask = 0x7f; +constexpr uint32_t kCameraMask = 0x80; +constexpr unsigned kTimeoutShift = 8; + +bool valid(const DisplaySettings& settings) +{ + return settings.brightness >= 1 && settings.brightness <= 100 && + settings.clock_timeout_seconds <= 86400; +} +} + +PanelSettings load_panel_settings(DisplaySettings defaults) +{ + // Firmware without camera gestures stored brightness and timeout as separate + // keys in the shared "display" namespace; they apply until the next save. + nvs_handle_t handle; + if (nvs_open(kNamespace, NVS_READONLY, &handle) != ESP_OK) return {load_display_settings(defaults)}; + uint32_t packed = 0; + const esp_err_t error = nvs_get_u32(handle, kSettingsKey, &packed); + nvs_close(handle); + const DisplaySettings display{static_cast(packed & kBrightnessMask), + packed >> kTimeoutShift}; + if (error != ESP_OK || !valid(display)) return {load_display_settings(defaults)}; + return {display, (packed & kCameraMask) != 0}; +} + +esp_err_t save_panel_settings(const PanelSettings& settings) +{ + if (!valid(settings.display)) return ESP_ERR_INVALID_ARG; + const uint32_t packed = settings.display.brightness | + (settings.camera_enabled ? kCameraMask : 0) | + (settings.display.clock_timeout_seconds << kTimeoutShift); + nvs_handle_t handle; + esp_err_t error = nvs_open(kNamespace, NVS_READWRITE, &handle); + if (error != ESP_OK) return error; + error = nvs_set_u32(handle, kSettingsKey, packed); + if (error == ESP_OK) error = nvs_commit(handle); + nvs_close(handle); + return error; +} + +} // namespace usage_panel::mosaico diff --git a/firmware/targets/esp_mosaico/main/panel_settings.hpp b/firmware/targets/esp_mosaico/main/panel_settings.hpp new file mode 100644 index 0000000..52dc734 --- /dev/null +++ b/firmware/targets/esp_mosaico/main/panel_settings.hpp @@ -0,0 +1,23 @@ +#pragma once + +#include "usage_panel_state/display_settings.hpp" + +namespace usage_panel::mosaico { + +struct PanelSettings { + DisplaySettings display; + bool camera_enabled = false; +}; + +/** + * Loads one settings record. A missing or invalid record disables capture and falls back to + * the separate display keys written by earlier firmware, then to defaults for each field. + */ +PanelSettings load_panel_settings(DisplaySettings defaults); +/** + * Saves all fields in one NVS value. Brightness must be 1..100 and timeout 0..86400 seconds. + * Returns ESP_ERR_INVALID_ARG for invalid display values, otherwise the NVS operation status. + */ +esp_err_t save_panel_settings(const PanelSettings& settings); + +} // namespace usage_panel::mosaico diff --git a/firmware/targets/esp_mosaico/partitions.csv b/firmware/targets/esp_mosaico/partitions.csv index a3eb613..2e32cad 100644 --- a/firmware/targets/esp_mosaico/partitions.csv +++ b/firmware/targets/esp_mosaico/partitions.csv @@ -1,6 +1,6 @@ # Name, Type, SubType, Offset, Size nvs, data, nvs, 0x9000, 0x6000 phy_init, data, phy, 0xf000, 0x1000 -ota_0, app, ota_0, 0x10000, 0x400000 -ota_1, app, ota_1, 0x410000, 0x400000 +ota_0, app, ota_0, 0x10000, 0x700000 otadata, data, ota, 0x810000, 0x2000 +ota_1, app, ota_1, 0x820000, 0x700000 diff --git a/firmware/targets/esp_mosaico/sdkconfig.defaults b/firmware/targets/esp_mosaico/sdkconfig.defaults index b380051..0482530 100644 --- a/firmware/targets/esp_mosaico/sdkconfig.defaults +++ b/firmware/targets/esp_mosaico/sdkconfig.defaults @@ -12,12 +12,18 @@ CONFIG_SPIRAM_MODE_OCT=y CONFIG_SPIRAM_SPEED_250M=y CONFIG_SPIRAM_XIP_FROM_PSRAM=y +# The driver default of 2000 ms delays the first battery reading; the driver still +# confirms the sealed state after this wait and fails initialization otherwise. +CONFIG_BQ27220_SEAL_SETTLE_MS=200 + CONFIG_BSP_DISPLAY_ENABLE=y CONFIG_BSP_CO5300_ENABLE_TE=y CONFIG_LV_COLOR_DEPTH_16=y CONFIG_LV_OS_FREERTOS=y CONFIG_LV_DEF_REFR_PERIOD=15 +# Clock glyph buffers and UI objects share the ESP-IDF heap instead of a fixed LVGL pool. +CONFIG_LV_USE_CLIB_MALLOC=y CONFIG_LV_USE_CLIB_STRING=y CONFIG_LV_USE_CLIB_SPRINTF=y CONFIG_LV_OBJ_STYLE_CACHE=y @@ -63,5 +69,40 @@ CONFIG_ESP_DESKTOP_BUDDY_RX_INBOX_DEPTH=32 CONFIG_ESP_DESKTOP_BUDDY_TX_QUEUE_DEPTH=4 CONFIG_ESP_DESKTOP_BUDDY_FOLDER_PUSH_MAX_RAW_CHUNK=4096 CONFIG_ESP_DESKTOP_BUDDY_FOLDER_PUSH_MAX_DECODED_CHUNK=2880 -CONFIG_ESP_DESKTOP_BUDDY_FOLDER_PUSH_MAX_TRANSFER_BYTES=4194816 +CONFIG_ESP_DESKTOP_BUDDY_FOLDER_PUSH_MAX_TRANSFER_BYTES=7340544 CONFIG_ESP_DESKTOP_BUDDY_FOLDER_PUSH_FLAT_PATHS_ONLY=y + +# CameraBoard D2 occupies the USB Serial/JTAG pad; retain UART diagnostics. +CONFIG_ESP_CONSOLE_SECONDARY_NONE=y +CONFIG_ESP_VIDEO_ENABLE_DVP_VIDEO_DEVICE=y +CONFIG_CAMERA_OV3640=y +CONFIG_CAMERA_OV3640_DVP_YUV422_UYVY_640X480_30FPS=y +CONFIG_CAMERA_OV3640_DVP_DEFAULT_FMT_YUV422_UYVY_640X480_30FPS=y +# CameraBoards ship with either sensor, as in esp-mosaico-bsp mosaico_module_camera; +# esp_video probes both and keeps the one that answers. +CONFIG_CAMERA_SC101IOT=y +CONFIG_CAMERA_SC101IOT_AUTO_DETECT_DVP_INTERFACE_SENSOR=y +# SC101IOT VGA crops the sensor center; 720p retains the full field of view. +CONFIG_CAMERA_SC101IOT_DVP_YUV422_1280X720_15FPS=y +CONFIG_CAMERA_SC101IOT_DVP_DEFAULT_FMT_YUV422_1280X720_15FPS=y +# CONFIG_CAMERA_SC101IOT_DVP_YUV422_640X480_15FPS is not set +# CONFIG_CAMERA_SC101IOT_DVP_DEFAULT_FMT_YUV422_640X480_15FPS is not set +CONFIG_MOSAICO_GESTURE_INPUT=y +CONFIG_MOSAICO_CAMERA_ROTATION=90 +CONFIG_MOSAICO_CAMERA_MIRROR=y + +# The vision pipeline supplies RGB888; other ESP-DL converters are unused. +# CONFIG_PIX_CVT_RGB565_TO_RGB565_SUPPORT is not set +# CONFIG_PIX_CVT_RGB565_TO_RGB888_SUPPORT is not set +# CONFIG_PIX_CVT_RGB565_TO_GRAY_SUPPORT is not set +# CONFIG_PIX_CVT_RGB565_TO_HSV_SUPPORT is not set +# CONFIG_PIX_CVT_RGB888_TO_RGB565_SUPPORT is not set +# CONFIG_PIX_CVT_RGB888_TO_GRAY_SUPPORT is not set +# CONFIG_PIX_CVT_RGB888_TO_HSV_SUPPORT is not set +# CONFIG_PIX_CVT_GRAY_TO_GRAY_SUPPORT is not set +# CONFIG_PIX_CVT_HSV_TO_HSV_MASK_SUPPORT is not set +# CONFIG_PIX_CVT_YUV_TO_RGB888_SUPPORT is not set +# CONFIG_PIX_CVT_YUV_TO_RGB565_SUPPORT is not set +# CONFIG_PIX_CVT_YUV_TO_GRAY_SUPPORT is not set +# CONFIG_PIX_CVT_YUV_TO_HSV_SUPPORT is not set +# CONFIG_PIX_CVT_YUV_TO_YUV_SUPPORT is not set diff --git a/firmware/targets/esp_mosaico/test_apps/logic/main/CMakeLists.txt b/firmware/targets/esp_mosaico/test_apps/logic/main/CMakeLists.txt index 0a70c1a..8539caa 100644 --- a/firmware/targets/esp_mosaico/test_apps/logic/main/CMakeLists.txt +++ b/firmware/targets/esp_mosaico/test_apps/logic/main/CMakeLists.txt @@ -3,10 +3,15 @@ idf_component_register( "test_main.cpp" "test_local_clock.cpp" "test_orientation.cpp" + "test_gestures.cpp" + "test_panel_settings.cpp" + "../../../main/panel_settings.cpp" "../../../main/local_clock.cpp" "../../../main/orientation.cpp" + "../../../main/gesture_tracker.cpp" + "../../../main/gesture_image.cpp" INCLUDE_DIRS "../../../main" - REQUIRES unity usage_protocol usage_panel_state + REQUIRES unity usage_protocol usage_panel_state nvs_flash WHOLE_ARCHIVE ) target_compile_features(${COMPONENT_LIB} PUBLIC cxx_std_17) diff --git a/firmware/targets/esp_mosaico/test_apps/logic/main/test_gestures.cpp b/firmware/targets/esp_mosaico/test_apps/logic/main/test_gestures.cpp new file mode 100644 index 0000000..be5efe3 --- /dev/null +++ b/firmware/targets/esp_mosaico/test_apps/logic/main/test_gestures.cpp @@ -0,0 +1,303 @@ +#include "gesture_tracker.hpp" +#include "gesture_image.hpp" +#include "unity.h" + +#include +#include + +using namespace usage_panel::mosaico; + +namespace { +std::array image_output; + +HandObservation hand(float x, float y = 0.5F, float ok = 0) +{ + return {1, x, y, 0.25F, 0.35F, 0.9F, ok}; +} +void arm(GestureTracker& tracker) +{ + tracker.update({}, 1000); + tracker.update({}, 1400); +} +} + +TEST_CASE("one burst of motion is one swipe and the return stroke is ignored", "[gesture]") +{ + for (bool vertical : {false, true}) { + for (int direction : {-1, 1}) { + // Positions are given for the positive direction and mirrored for the negative one. + const auto point = [vertical, direction](float position) { + const float mapped = 0.5F + direction * (position - 0.5F); + return vertical ? hand(0.5F, mapped) : hand(mapped); + }; + const auto expected = vertical + ? (direction < 0 ? GestureAction::SwipeUp : GestureAction::SwipeDown) + : (direction < 0 ? GestureAction::SwipeLeft : GestureAction::SwipeRight); + const auto reverse = vertical + ? (direction < 0 ? GestureAction::SwipeDown : GestureAction::SwipeUp) + : (direction < 0 ? GestureAction::SwipeRight : GestureAction::SwipeLeft); + const auto none = int(GestureAction::None); + GestureTracker tracker; + arm(tracker); + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(point(0.2F), 1500))); + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(point(0.35F), 1600))); + TEST_ASSERT_EQUAL_INT(int(expected), int(tracker.update(point(0.5F), 1700))); + tracker.reset(true); // The accepted swipe's page change keeps swipes armed. + // The hand comes to rest in view; the stroke back is the hand returning. + for (unsigned t = 1800; t < 2400; t += 100) + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(point(0.65F), t))); + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(point(0.5F), 2400))); + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(point(0.35F), 2500))); + // The same direction repeats without the hand leaving. + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(point(0.5F), 2600))); + TEST_ASSERT_EQUAL_INT(int(expected), int(tracker.update(point(0.65F), 2700))); + tracker.reset(true); + // Motion that continues without a rest belongs to the same command. + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(point(0.8F), 2800))); + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(point(0.5F), 2900))); + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(point(0.2F), 3000))); + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(point(0.5F), 3100))); + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(point(0.8F), 3200))); + // A stationary hand issues nothing; the reverse direction works once the return window ends. + for (unsigned t = 3300; t < 4500; t += 100) + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(point(0.8F), t))); + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(point(0.65F), 4500))); + TEST_ASSERT_EQUAL_INT(int(reverse), int(tracker.update(point(0.5F), 4600))); + // A context change unrelated to a swipe requires the view to clear again. + tracker.reset(); + tracker.update(point(0.2F), 6100); + tracker.update(point(0.35F), 6200); + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(point(0.5F), 6300))); + } + } +} + +TEST_CASE("a wave too small for a swipe issues one command until the hand rests", "[gesture]") +{ + const auto none = int(GestureAction::None); + GestureTracker tracker; + arm(tracker); + // Frames arrive about every 125 ms; each leg covers 15% of the width. + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(hand(0.42F), 1500))); + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(hand(0.57F), 1625))); + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(hand(0.42F), 1750))); + TEST_ASSERT_EQUAL_INT(int(GestureAction::Wave), int(tracker.update(hand(0.57F), 1875))); + tracker.reset(true); + for (unsigned i = 0; i < 6; ++i) + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(hand(i % 2 ? 0.57F : 0.42F), 2000 + i * 125))); + for (unsigned t = 2750; t <= 3250; t += 125) + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(hand(0.57F), t))); + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(hand(0.42F), 3375))); + TEST_ASSERT_EQUAL_INT(none, int(tracker.update(hand(0.57F), 3500))); + TEST_ASSERT_EQUAL_INT(int(GestureAction::Wave), int(tracker.update(hand(0.42F), 3625))); +} + +TEST_CASE("a resting hand starts its stroke when it begins to move", "[gesture]") +{ + GestureTracker tracker; + arm(tracker); + for (unsigned t = 1500; t <= 2300; t += 100) + TEST_ASSERT_EQUAL_INT(int(GestureAction::None), int(tracker.update(hand(0.2F), t))); + tracker.update(hand(0.35F), 2400); + TEST_ASSERT_EQUAL_INT(int(GestureAction::SwipeRight), int(tracker.update(hand(0.5F), 2500))); +} + +TEST_CASE("pose classification waits for a hand that stays in place", "[gesture]") +{ + TEST_ASSERT_TRUE(hand_is_steady(hand(0.5F), hand(0.53F, 0.52F))); + TEST_ASSERT_FALSE(hand_is_steady(hand(0.5F), hand(0.6F))); + TEST_ASSERT_FALSE(hand_is_steady(hand(0.5F), hand(0.5F, 0.6F))); + TEST_ASSERT_FALSE(hand_is_steady({}, hand(0.5F))); +} + +TEST_CASE("OK requires a stable hold and release after context invalidation", "[gesture]") +{ + GestureTracker tracker; + arm(tracker); + for (unsigned t = 1500; t < 2000; t += 100) + TEST_ASSERT_EQUAL_INT(int(GestureAction::None), int(tracker.update(hand(0.5F, 0.5F, 0.95F), t))); + TEST_ASSERT_EQUAL_INT(80, tracker.hold_progress()); + TEST_ASSERT_EQUAL_INT(int(GestureAction::EnterClock), int(tracker.update(hand(0.5F, 0.5F, 0.95F), 2000))); + tracker.reset(true); + for (unsigned t = 2100; t <= 3000; t += 100) + TEST_ASSERT_EQUAL_INT(int(GestureAction::None), int(tracker.update(hand(0.5F, 0.5F, 0.95F), t))); + // The hand that made the pose never left: its stroke is ignored, its wave is not. + TEST_ASSERT_EQUAL_INT(int(GestureAction::None), int(tracker.update(hand(0.65F), 3100))); + TEST_ASSERT_EQUAL_INT(int(GestureAction::None), int(tracker.update(hand(0.8F), 3200))); + TEST_ASSERT_EQUAL_INT(int(GestureAction::None), int(tracker.update(hand(0.65F), 3300))); + TEST_ASSERT_EQUAL_INT(int(GestureAction::Wave), int(tracker.update(hand(0.8F), 3400))); +} + +TEST_CASE("external input during swipe cooldown requires release before another gesture", "[gesture]") +{ + GestureTracker tracker; + arm(tracker); + tracker.update(hand(0.2F), 1500); + tracker.update(hand(0.35F), 1600); + TEST_ASSERT_EQUAL_INT(int(GestureAction::SwipeRight), int(tracker.update(hand(0.5F), 1700))); + tracker.reset(true); + tracker.update(hand(0.65F), 1800); + tracker.reset(); + tracker.update(hand(0.5F), 2200); + tracker.update(hand(0.65F), 2300); + TEST_ASSERT_EQUAL_INT(int(GestureAction::None), int(tracker.update(hand(0.8F), 2400))); + tracker.update({}, 2500); + tracker.update({}, 2800); + tracker.update(hand(0.2F), 2900); + tracker.update(hand(0.35F), 3000); + TEST_ASSERT_EQUAL_INT(int(GestureAction::SwipeRight), int(tracker.update(hand(0.5F), 3100))); +} + +TEST_CASE("single dropouts keep a swipe and an OK hold alive", "[gesture]") +{ + HandObservation weak = hand(0.6F); + weak.score = 0.55F; + for (const HandObservation& dropout : {HandObservation{}, weak}) { + GestureTracker tracker; + arm(tracker); + tracker.update(hand(0.5F), 1500); + tracker.update(dropout, 1600); + tracker.update(hand(0.65F), 1700); + TEST_ASSERT_EQUAL_INT(int(GestureAction::SwipeRight), int(tracker.update(hand(0.8F), 1800))); + } + + GestureTracker tracker; + arm(tracker); + tracker.update(hand(0.5F, 0.5F, 0.95F), 1500); + tracker.update(hand(0.5F, 0.5F, 0.95F), 1600); + tracker.update(hand(0.5F, 0.5F, 0.95F), 1700); + tracker.update({}, 1800); + TEST_ASSERT_EQUAL_INT(40, tracker.hold_progress()); + tracker.update(hand(0.5F, 0.5F, 0.95F), 1900); + TEST_ASSERT_EQUAL_INT(int(GestureAction::EnterClock), int(tracker.update(hand(0.5F, 0.5F, 0.95F), 2000))); +} + +TEST_CASE("OK hold completes at a 300 ms frame interval", "[gesture]") +{ + GestureTracker tracker; + arm(tracker); + tracker.update(hand(0.5F, 0.5F, 0.95F), 1500); + tracker.update(hand(0.5F, 0.5F, 0.95F), 1800); + TEST_ASSERT_EQUAL_INT(int(GestureAction::EnterClock), int(tracker.update(hand(0.5F, 0.5F, 0.95F), 2100))); +} + +TEST_CASE("slow frames and position jumps restart tracking without requiring release", "[gesture]") +{ + GestureTracker tracker; + arm(tracker); + tracker.update(hand(0.2F), 1500); + tracker.update(hand(0.2F), 2000); // 500 ms gap: restart here. + tracker.update(hand(0.35F), 2100); + TEST_ASSERT_EQUAL_INT(int(GestureAction::SwipeRight), int(tracker.update(hand(0.5F), 2200))); + + tracker.update({}, 2300); + tracker.update({}, 2700); + tracker.update(hand(0.9F), 2800); + tracker.update(hand(0.3F), 2900); // Jump: restart here. + tracker.update(hand(0.45F), 3000); + TEST_ASSERT_EQUAL_INT(int(GestureAction::SwipeRight), int(tracker.update(hand(0.6F), 3100))); +} + +TEST_CASE("diagonal motion jitter ambiguous hands and stale samples do not trigger", "[gesture]") +{ + for (unsigned scenario = 0; scenario < 6; ++scenario) { + GestureTracker tracker; + arm(tracker); + for (unsigned i = 0; i < 7; ++i) { + HandObservation observation = hand(0.2F + i * 0.08F); + unsigned time = 1500 + i * 100; + if (scenario == 0) observation.y = observation.x; + if (scenario == 1) observation.x = i % 2 ? 0.55F : 0.45F; + if (scenario == 2) observation.count = 2; + if (scenario == 3) time = 1500 + i * 400; + if (scenario == 4) observation.x = std::numeric_limits::quiet_NaN(); + if (scenario == 5) { observation.x = 0.5F; observation.y = i % 2 ? 0.55F : 0.45F; } + TEST_ASSERT_EQUAL_INT(int(GestureAction::None), int(tracker.update(observation, time))); + } + } +} + +TEST_CASE("wake consumes the first swipe and protected states reject all commands", "[gesture]") +{ + TEST_ASSERT_EQUAL_INT(int(GestureEffect::Wake), int(route_gesture(GestureAction::SwipeLeft, true, true))); + TEST_ASSERT_EQUAL_INT(int(GestureEffect::Wake), int(route_gesture(GestureAction::SwipeRight, true, true))); + TEST_ASSERT_EQUAL_INT(int(GestureEffect::Next), int(route_gesture(GestureAction::SwipeLeft, true, false))); + TEST_ASSERT_EQUAL_INT(int(GestureEffect::Next), int(route_gesture(GestureAction::SwipeRight, true, false))); + TEST_ASSERT_EQUAL_INT(int(GestureEffect::Wake), int(route_gesture(GestureAction::SwipeUp, true, true))); + TEST_ASSERT_EQUAL_INT(int(GestureEffect::Wake), int(route_gesture(GestureAction::SwipeDown, true, true))); + TEST_ASSERT_EQUAL_INT(int(GestureEffect::Next), int(route_gesture(GestureAction::SwipeUp, true, false))); + TEST_ASSERT_EQUAL_INT(int(GestureEffect::Next), int(route_gesture(GestureAction::SwipeDown, true, false))); + TEST_ASSERT_EQUAL_INT(int(GestureEffect::Wake), int(route_gesture(GestureAction::Wave, true, true))); + TEST_ASSERT_EQUAL_INT(int(GestureEffect::Next), int(route_gesture(GestureAction::Wave, true, false))); + TEST_ASSERT_EQUAL_INT(int(GestureEffect::Clock), int(route_gesture(GestureAction::EnterClock, true, false))); + TEST_ASSERT_EQUAL_INT(int(GestureEffect::None), int(route_gesture(GestureAction::EnterClock, true, true))); + for (auto action : {GestureAction::SwipeLeft, GestureAction::SwipeRight, GestureAction::SwipeUp, + GestureAction::SwipeDown, GestureAction::Wave, GestureAction::EnterClock}) + TEST_ASSERT_EQUAL_INT(int(GestureEffect::None), int(route_gesture(action, false, false))); +} + +TEST_CASE("UYVY conversion respects padded stride mirror and four rotations", "[gesture]") +{ + // Four grayscale quadrants plus row padding make the geometric mapping observable. + const uint8_t source[] = {128, 16, 128, 80, 255, 255, 128, 160, 128, 235, 255, 255}; + auto& output = image_output; + const unsigned expected[4][4] = {{0, 74, 167, 255}, {167, 0, 255, 74}, + {255, 167, 74, 0}, {74, 255, 0, 167}}; + for (unsigned rotation = 0; rotation < 4; ++rotation) { + const auto size = gesture_image_size(2, 2, rotation); + const unsigned w = size.width; + const unsigned h = size.height; + TEST_ASSERT_TRUE(prepare_gesture_image(source, sizeof(source), 2, 2, 6, output.data(), rotation, false)); + const unsigned pixels[] = {0, w - 1, (h - 1) * w, h * w - 1}; + for (unsigned i = 0; i < 4; ++i) + TEST_ASSERT_INT_WITHIN(1, expected[rotation][i], output[pixels[i] * 3]); + } + TEST_ASSERT_TRUE(prepare_gesture_image(source, sizeof(source), 2, 2, 6, output.data(), 0, true)); + TEST_ASSERT_INT_WITHIN(1, 74, output[0]); + TEST_ASSERT_FALSE(prepare_gesture_image(source, 4, 2, 2, 6, output.data(), 0, false)); + TEST_ASSERT_FALSE(prepare_gesture_image(source, sizeof(source), 3, 2, 6, output.data(), 0, false)); +} + +TEST_CASE("full sensor frame preserves aspect ratio and edges through rotation", "[gesture]") +{ + const auto hd = gesture_image_size(1280, 720, 0); + TEST_ASSERT_EQUAL_UINT(320, hd.width); + TEST_ASSERT_EQUAL_UINT(180, hd.height); + const auto vga = gesture_image_size(640, 480, 0); + TEST_ASSERT_EQUAL_UINT(320, vga.width); + TEST_ASSERT_EQUAL_UINT(240, vga.height); + TEST_ASSERT_EQUAL_UINT(0, gesture_image_size(0, 720, 0).width); + TEST_ASSERT_EQUAL_UINT(0, gesture_image_size(1280, 720, 4).width); + + // A 16:9 frame with distinct corners verifies that fitting does not crop edges. + constexpr unsigned stride = 36; + std::array source{}; + for (unsigned y = 0; y < 9; ++y) { + for (unsigned x = 0; x < 16; x += 2) { + const uint8_t luma = y < 4 ? (x < 8 ? 16 : 80) : (x < 8 ? 160 : 235); + const unsigned offset = y * stride + x * 2; + source[offset] = source[offset + 2] = 128; + source[offset + 1] = source[offset + 3] = luma; + } + } + auto& output = image_output; + const unsigned corners[2][4][4] = { + {{0, 74, 167, 255}, {167, 0, 255, 74}, {255, 167, 74, 0}, {74, 255, 0, 167}}, + {{74, 0, 255, 167}, {255, 74, 167, 0}, {167, 255, 0, 74}, {0, 167, 74, 255}} + }; + for (unsigned mirror = 0; mirror < 2; ++mirror) { + for (unsigned rotation = 0; rotation < 4; ++rotation) { + output.fill(42); + const auto size = gesture_image_size(16, 9, rotation); + TEST_ASSERT_EQUAL_UINT(rotation & 1 ? 180 : 320, size.width); + TEST_ASSERT_EQUAL_UINT(rotation & 1 ? 320 : 180, size.height); + TEST_ASSERT_TRUE(prepare_gesture_image(source.data(), source.size(), 16, 9, + stride, output.data(), rotation, mirror)); + const unsigned pixels[] = {0, size.width - 1, (size.height - 1) * size.width, + size.width * size.height - 1}; + for (unsigned i = 0; i < 4; ++i) + TEST_ASSERT_INT_WITHIN(1, corners[mirror][rotation][i], output[pixels[i] * 3]); + TEST_ASSERT_EQUAL_UINT8(42, output[size.width * size.height * 3]); + } + } +} diff --git a/firmware/targets/esp_mosaico/test_apps/logic/main/test_panel_settings.cpp b/firmware/targets/esp_mosaico/test_apps/logic/main/test_panel_settings.cpp new file mode 100644 index 0000000..f168d12 --- /dev/null +++ b/firmware/targets/esp_mosaico/test_apps/logic/main/test_panel_settings.cpp @@ -0,0 +1,80 @@ +#include "panel_settings.hpp" +#include "nvs.h" +#include "nvs_flash.h" +#include "unity.h" + +#include + +using namespace usage_panel; +using namespace usage_panel::mosaico; + +TEST_CASE("panel preferences persist together and reject invalid updates", "[settings]") +{ + TEST_ASSERT_EQUAL(ESP_OK, nvs_flash_init()); + for (bool camera : {false, true}) { + const PanelSettings expected{{camera ? uint8_t{100} : uint8_t{1}, + camera ? 86400U : 0U}, camera}; + TEST_ASSERT_EQUAL(ESP_OK, save_panel_settings(expected)); + TEST_ASSERT_EQUAL(ESP_ERR_INVALID_ARG, save_panel_settings({{0, 60}, !camera})); + TEST_ASSERT_EQUAL(ESP_ERR_INVALID_ARG, save_panel_settings({{101, 60}, !camera})); + TEST_ASSERT_EQUAL(ESP_ERR_INVALID_ARG, save_panel_settings({{50, 86401}, !camera})); + TEST_ASSERT_EQUAL(ESP_OK, nvs_flash_deinit()); + TEST_ASSERT_EQUAL(ESP_ERR_NVS_NOT_INITIALIZED, save_panel_settings({{50, 60}, !camera})); + TEST_ASSERT_EQUAL(ESP_OK, nvs_flash_init()); + const auto actual = load_panel_settings({50, 300}); + TEST_ASSERT_EQUAL(expected.display.brightness, actual.display.brightness); + TEST_ASSERT_EQUAL(expected.display.clock_timeout_seconds, actual.display.clock_timeout_seconds); + TEST_ASSERT_EQUAL(expected.camera_enabled, actual.camera_enabled); + } +} + +TEST_CASE("missing or invalid panel records use defaults with capture off", "[settings]") +{ + TEST_ASSERT_EQUAL(ESP_OK, nvs_flash_init()); + nvs_handle_t handle; + TEST_ASSERT_EQUAL(ESP_OK, nvs_open("mosaico", NVS_READWRITE, &handle)); + // Invalid brightness and timeout must reject the entire record, including capture enablement. + for (uint32_t packed : {uint32_t{0x80}, uint32_t{(86401U << 8) | 0x80 | 50}}) { + TEST_ASSERT_EQUAL(ESP_OK, nvs_set_u32(handle, "settings", packed)); + TEST_ASSERT_EQUAL(ESP_OK, nvs_commit(handle)); + const auto actual = load_panel_settings({40, 600}); + TEST_ASSERT_EQUAL(40, actual.display.brightness); + TEST_ASSERT_EQUAL(600, actual.display.clock_timeout_seconds); + TEST_ASSERT_FALSE(actual.camera_enabled); + } + TEST_ASSERT_EQUAL(ESP_OK, nvs_erase_key(handle, "settings")); + TEST_ASSERT_EQUAL(ESP_OK, nvs_commit(handle)); + nvs_close(handle); + const auto actual = load_panel_settings({30, 60}); + TEST_ASSERT_EQUAL(30, actual.display.brightness); + TEST_ASSERT_EQUAL(60, actual.display.clock_timeout_seconds); + TEST_ASSERT_FALSE(actual.camera_enabled); +} + +TEST_CASE("display keys from earlier firmware apply until a panel record is saved", "[settings]") +{ + TEST_ASSERT_EQUAL(ESP_OK, nvs_flash_init()); + nvs_handle_t handle; + TEST_ASSERT_EQUAL(ESP_OK, nvs_open("mosaico", NVS_READWRITE, &handle)); + const esp_err_t erased = nvs_erase_key(handle, "settings"); + TEST_ASSERT_TRUE(erased == ESP_OK || erased == ESP_ERR_NVS_NOT_FOUND); + TEST_ASSERT_EQUAL(ESP_OK, nvs_commit(handle)); + nvs_close(handle); + TEST_ASSERT_EQUAL(ESP_OK, save_display_settings({70, 600})); + + auto actual = load_panel_settings({40, 300}); + TEST_ASSERT_EQUAL(70, actual.display.brightness); + TEST_ASSERT_EQUAL(600, actual.display.clock_timeout_seconds); + TEST_ASSERT_FALSE(actual.camera_enabled); + + TEST_ASSERT_EQUAL(ESP_OK, save_panel_settings({{20, 60}, true})); + actual = load_panel_settings({40, 300}); + TEST_ASSERT_EQUAL(20, actual.display.brightness); + TEST_ASSERT_EQUAL(60, actual.display.clock_timeout_seconds); + TEST_ASSERT_TRUE(actual.camera_enabled); + + TEST_ASSERT_EQUAL(ESP_OK, nvs_open("display", NVS_READWRITE, &handle)); + TEST_ASSERT_EQUAL(ESP_OK, nvs_erase_all(handle)); + TEST_ASSERT_EQUAL(ESP_OK, nvs_commit(handle)); + nvs_close(handle); +} diff --git a/firmware/targets/esp_mosaico/test_apps/logic/sdkconfig.defaults b/firmware/targets/esp_mosaico/test_apps/logic/sdkconfig.defaults new file mode 100644 index 0000000..3a329c6 --- /dev/null +++ b/firmware/targets/esp_mosaico/test_apps/logic/sdkconfig.defaults @@ -0,0 +1 @@ +CONFIG_IDF_TARGET="esp32s31" diff --git a/firmware/test_apps/panel_state/main/test_screensaver_state.cpp b/firmware/test_apps/panel_state/main/test_screensaver_state.cpp index 55c19bb..29a2673 100644 --- a/firmware/test_apps/panel_state/main/test_screensaver_state.cpp +++ b/firmware/test_apps/panel_state/main/test_screensaver_state.cpp @@ -4,6 +4,20 @@ using namespace usage_panel; +TEST_CASE("explicit clock entry is idempotent and wakes to the original page", "[screensaver]") +{ + ScreensaverController controller(0); + controller.reset(0, Page::Codex); + controller.enter_clock(1000, Page::Claude); + controller.enter_clock(2000, Page::Overview); + TEST_ASSERT_TRUE(controller.view(2000).active); + TEST_ASSERT_EQUAL_INT(int(Page::Claude), int(controller.restore_page())); + controller.note_touch_down(3000); + TEST_ASSERT_FALSE(controller.view(3000).active); + TEST_ASSERT_TRUE(controller.view(3000).consume_touch); + TEST_ASSERT_EQUAL_INT(int(Page::Claude), int(controller.restore_page())); +} + TEST_CASE("screensaver enters after five minutes and preserves page", "[screensaver]") { ScreensaverController controller;