diff --git a/.sentrybot/AI_HUB.md b/.sentrybot/AI_HUB.md index 85c8b8cc..04cf40a9 100644 --- a/.sentrybot/AI_HUB.md +++ b/.sentrybot/AI_HUB.md @@ -59,6 +59,7 @@ Bu dosyayı oku → ilgili agent veya skill'i bul → uygula. | Kod incele | `agents/code-reviewer.md` | | Arduino komutu ekle | `skills/arduino-contract.md` | | Hata ayıkla | `skills/debug-module.md` | +| Laya System 1 entegrasyonunu sürdür | `laya_integration_completed.md` → `laya_integration_todo.md` → `context/laya-action-contract.md` | --- diff --git a/.sentrybot/context/laya-action-contract.md b/.sentrybot/context/laya-action-contract.md new file mode 100644 index 00000000..06d10c5b --- /dev/null +++ b/.sentrybot/context/laya-action-contract.md @@ -0,0 +1,105 @@ +# Laya Action Proposal Contract + +**Status:** Fail-closed proposal builder is implemented; proposal generation is config-disabled by default and direct Laya dispatch stays disabled until each applicable item below has real-model verification, implementation, and regression coverage. + +## Current boundary + +`LayaEngine` currently classifies `target_module`, `is_direct_command`, affect and urgency. `LayaDecision.suggested_action` exists as a field but `decide()` does not populate it. A target such as `arduino_serial` does not identify an actuator, an operation, or safe arguments. `emergency_stop -> stop_follow` is not equivalent to stopping the motors. + +The installed Laya 0.3.5 runtime source (`.venv/Lib/site-packages/laya/agent.py`) accepts `choice`, `score`, and `noul` question types in `system_one()` and returns typed answers with confidence/probabilities; it does not provide a free-form JSON/action-argument output type. Therefore any future proposal must be assembled from multiple bounded choice answers (for example, a finite light color/effect) and each required field must independently meet its configured confidence threshold. Continuous actuator values must not be guessed from these classifications. + +The code now contains a pure `set_lights` proposal builder in `modules/agent_core/services/laya_action_contract.py`. It accepts only direct `neopixel` decisions, requires confidence thresholds for the module, direct-command, color, and effect answers, rejects unknown choices/non-finite confidence values, checks the configured action allowlist, and returns data only. `config/agent.yaml` keeps `action_proposals.enabled: false`; no dispatch call is wired from this proposal. + +## Real-model confidence finding + +CPU inference succeeded under the existing Python 3.11 environment after running the approved interpreter directly. With the production Laya questions, sampled Turkish inputs produced: + +| Input | Target / confidence | Direct / confidence | Urgency / confidence | Emergency result after gate | +|---|---|---|---|---| +| `Işıkları kapat` | `emergency_stop` / 0.8273 | direct / 0.8297 | 2.059 / 0.5410 | Rejected | +| `Sentry hakkında ne düşünüyorsun?` | `speak` / 0.5004 | conversation / 0.5971 | 2.4434 / 0.2844 | Rejected | +| `Motorları acil durdur, çarpacaksın!` | `emergency_stop` / 0.9340 | direct / 0.9818 | 2.2406 / 0.1849 | Accepted by confident emergency target | +| `Dur hemen!` | `emergency_stop` / 0.9503 | direct / 0.9696 | 2.0872 / 0.1601 | Accepted by confident emergency target | + +This exposed a real false-positive in the old emergency predicate: it accepted any `emergency_stop` target or urgency score over threshold regardless of confidence/directness. Emergency now requires a direct command plus either a high-confidence emergency target or a high-confidence emergency urgency score. Non-emergency high urgency also requires configured confidence before sensor/mood urgency effects. Thresholds are in `tri_layer.laya.urgency`; current defaults are target 0.90, direct 0.80, urgency 0.80, and ordinary high-urgency confidence 0.60. Tests pin the observed false-positive and positive emergency samples. + +When the experimental finite action questions were added to the same model call, `Işıkları mavi yap` returned color `pink` at 0.2894 confidence and effect `unspecified` at 0.3754; the action proposal is correctly rejected. This is not ready for actuator dispatch; action extraction needs further model/question calibration and real prompt-set evaluation. + +The repeatable offline evaluator is `tools/evaluate_laya_action_proposals.py`. Its fixed 14-example Turkish sample currently reports target top-choice accuracy 0.643; at the 0.60 confidence gate, target coverage 0.571 and accepted-target accuracy 0.625. Direct-command top-choice accuracy is 0.929; confidence-gated direct coverage 0.357 and accepted-direct accuracy 1.0. Current explicit-color top-choice accuracy is **0.125 (1/8)** with zero explicit colors accepted at 0.85 confidence. Two alternative explicit color instructions each reached 1.0 top-choice accuracy and 0.875 explicit-color confidence coverage, but only 0.167/0.333 specificity across six no-color controls; each produced **2 high-confidence false colors**. Explicit effect accuracy is 0.667 (2/3), confidence-gated emergency accuracy 1.0 (14/14), and the current policy creates **0 proposals**. The color metric counts only prompts with an explicit requested color; chat/control cases cannot inflate it. This small fixed sample is a calibration check, not a general benchmark. Keep action proposals disabled until a larger held-out set demonstrates adequate precision and coverage. + +A third experimental question combines color, `turn_off`, and `no_light_action` in one finite choice. On the same 14 examples, joint action accuracy was 0.500; explicit-color accuracy was 0.750 with only 0.375 coverage at confidence >=0.85 and 1.000 accuracy among accepted explicit colors. On the six no-color controls, specificity was 0.167, with **3 high-confidence false color/action answers**. The combined question therefore does not fix the false-positive issue and is not promoted to production. The sample is unchanged and these prompt variants are exploratory comparisons, not independent validation. + +The evaluator was then expanded to 32 hand-labeled Turkish examples: 18 explicit color/off commands and 14 no-explicit-color controls, including color discussion, camera questions, and generic light commands. On this expanded calibration set, the conservative/current color question scored 0.111 explicit accuracy and 0.857 no-color specificity, with zero high-confidence false colors; the Turkish explicit variant scored 1.000 explicit accuracy with 0.778 confidence coverage but only 0.214 no-color specificity and 4 high-confidence false colors; the English/bilingual variant scored 0.944 explicit accuracy with 0.889 coverage, 0.286 specificity, and 5 high-confidence false colors. The joint action question scored 0.625 overall and 0.778 explicit-color accuracy, but only 0.222 explicit-color confidence coverage, 0.429 no-color specificity, and 4 high-confidence false positives. Emergency labels remained 32/32 and proposal count remained zero. This is still a prompt-calibration set, not an independent holdout. Results reinforce keeping proposals disabled. + +An evaluator-only deterministic Turkish command parser was compared with the frozen Laya questions on a separate 30-example final holdout (14 explicit color/off commands, 16 controls; zero text overlap with calibration/parser-development lists). The parser scored 0.967 overall, 0.929 on explicit color/off, and 1.000 specificity on unspecified controls. On the same examples, the current model question scored 0.433 overall (0.143 explicit accuracy, 0.688 specificity, 1 high-confidence false positive); Turkish explicit scored 0.533 overall (1.000 explicit accuracy, 0.786 coverage, 0.125 specificity, 8 high-confidence false positives); English/bilingual scored 0.567 (1.000 explicit accuracy, 0.714 coverage, 0.188 specificity, 9 high-confidence false positives); joint scored 0.500 (0.786 explicit accuracy, 0.286 coverage, 0.250 specificity, 4 high-confidence false positives; accepted explicit accuracy 0.750). These are small hand-authored datasets and the parser has not been broadly validated. The parser remains isolated to the evaluator; it is not production code, proposal authority, or a dispatch path. Do not tune against the final holdout; any future parser changes need a new untouched test split. + +A separate 30-example intent/safety holdout was added with five samples for each of the six target classes. On this split, target accuracy was 0.600; class accuracies were neopixel 0.400, arduino_serial 1.000, emergency_stop 0.800, speak 0.200, camera 1.000, and system2_chat 0.200. At the configured 0.60 confidence gate, target coverage was 0.700 and accepted-target accuracy 0.714. Direct-command accuracy was 0.900; confidence coverage 0.700 and accepted-direct accuracy 1.000. Confidence-gated emergency accuracy was 0.933 with zero false positives and 2 false negatives; high-urgency accuracy was 0.833 with zero false positives and 5 false negatives. Explicit effect top-choice accuracy was 0.500 (2/4), with zero answers meeting the 0.85 proposal threshold. No proposal or tool execution is part of this evaluator. The split is small and must remain untouched; it identifies target classification (especially speak/system2_chat), emergency recall, urgency recall, and effect confidence as remaining calibration gaps. + +Two additional target/effect question variants were tested only on the 32-example development set. The longer Turkish target variant scored 0.188 accuracy and 0.174 accepted accuracy at 0.719 coverage. The concise English target variant scored 0.500 accuracy, but only 0.188 coverage and 0.667 accepted accuracy. An explicit-effect question reached 0.800 top-choice accuracy on five labeled effects, but 0.400 confidence coverage and only 0.500 accuracy among accepted answers. Neither target variant nor the effect variant is a viable proposal gate; none was run against the intent holdout. Production questions remain unchanged. This work produced evaluation evidence, not a completed rollout gate, so it does not increase the integration completion estimate. + +A new balanced 30-example intent development split (five examples for each target) compared current, explicit Turkish, and explicit English target questions. Current target accuracy was 0.467, accepted accuracy 0.667 at 0.500 coverage; Turkish was 0.500 / 0.857 / 0.467; English was 0.600 / 0.786 / 0.467. Turkish had 2 emergency false negatives versus 3 for current; English also had 3. Direct accuracy was 0.933 with 1.000 accepted accuracy at 0.633 coverage. Urgency accuracy was 0.800 with 1 false positive and 5 false negatives. Effect variants each scored 0.500 across four examples; current coverage was 0.500, explicit-question coverage 0.250, with accepted accuracy 1.000 for both. The Turkish target candidate was selected on development results and tested once against the previously untouched 30-example intent holdout. It regressed target accuracy from 0.600 to 0.433, accepted accuracy from 0.714 to 0.647, and emergency accuracy from 0.933 to 0.900 (false negatives increased from 2 to 3). Candidate rejected; production remains unchanged. This holdout is now consumed for this comparison; future candidates require a new untouched test split. + +A separate binary `explicit_audio_request` question was then added to the intent development evaluation to distinguish explicit spoken-output commands from ordinary dialogue. It scored 0.733 accuracy, but precision and recall were both 0.0 (3 false positives and all 5 explicit speak requests missed). Applying it only as a speak/system2_chat refiner at confidence gates 0.60, 0.80, or 0.90 left target accuracy unchanged at 0.467. This signal is rejected and must not be used in production. + +Local model inference was attempted using the cached multilingual snapshot, but the current Python 3.12 runtime cannot import the venv's Torch binary (`torch_python.dll`, Windows error 126). A Python 3.12-compatible Torch wheel was then approved for an isolated `.laya_runtime` install; the 124.1 MB download stalled after 7.6 MB at roughly 73 kB/s and repeatedly lost its connection, so installation was stopped before any package was installed. The SDK interface is confirmed by source inspection, while real model behavior for additional action questions remains unverified until inference runs on a compatible environment. + +Laya classification may select the short response path and provide diagnostic `laya_hint` telemetry. It must not call `ToolRegistry.execute()` from the classifier or turn a `suggested_tool` string into authority. + +## Proposal shape + +When the model/runtime can provide action arguments, they must be represented as a typed proposal before dispatch: + +```json +{ + "schema_version": 1, + "action": "", + "arguments": {}, + "confidence": 0.0 +} +``` + +The runtime must bind this proposal to the current user turn and its request id. It must not accept caller-supplied `source`, `priority`, `ttl_ms`, hardware payloads, URLs, or arbitrary tool names from Laya. + +## Validation requirements + +Reject the proposal without side effects when any of these conditions holds: + +- It is missing, malformed, stale, for a different request, or has an unsupported schema version. +- `action` is not in a small explicit allowlist. Generic `queue_action`, arbitrary Arduino commands, `stop_follow`, and unrestricted motion are not implicitly allowlisted. +- The proposal is not a confident direct command, or the required module/direct confidence is below configured thresholds. +- Arguments are absent, contain unknown keys, have the wrong types, or exceed per-action limits. Do not silently coerce malformed model output into a valid actuator value. +- The action's existing safety/capability gate or resource arbiter rejects it. + +Thresholds and per-action numeric bounds must come from configuration. Validators are fail-closed; inference or validation errors leave the normal Agent/Autonomy path available. + +## Execution path + +For an approved non-emergency action, dispatch only through the existing registered tool implementation so that its resource arbiter and downstream owner remain in force. The action must also pass the runtime owner's safety/capability policy; if a tool currently lacks that gate, it cannot be enabled for Laya dispatch until the gate is added and tested. Arduino payloads must use `modules/arduino_serial/contract.py` builders and validated request/ACK behavior. + +Emergency stop is a separate reflex contract owned by Autonomy/SpinalCord. It must invoke a validated motor stop operation, never infer `stop_follow` as a substitute, and must not bypass the action/resource safety boundaries for unrelated actions. + +## Rollout gates + +1. Extend Laya output with a structured action proposal and prove the runtime/model actually returns it; classification-only output remains non-dispatchable. +2. Add an explicit allowlist with per-action typed schemas, config-driven thresholds/bounds, and a pure validator. Cover missing, extra, malformed, low-confidence, stale, and out-of-range cases. +3. Add a dispatcher that calls the existing tool registry and preserves tool/action/resource arbiters and autonomy capability checks. Assert invalid proposals produce zero calls. +4. Implement and test the emergency motor-stop operation through the authoritative spinal/Arduino contract path before treating emergency output as executable. +5. Run mock/integration tests, then perform model and hardware acceptance on the target device. Hardware acceptance must verify stop behavior, cancellation/cooldown, ACK/timeout behavior, and that arbitrary model output cannot actuate the robot. + +Until all applicable gates pass, Laya remains a classifier/telemetry/affect signal only; ordinary LLM and Autonomy action pathways retain ownership. + +## 2026-09-27 — literal audio-command parser experiment + +An evaluator-only deterministic parser was measured on the already-used 30-example intent development split and scored 30/30 (5 explicit speech/sound commands found; no false positives). A new, disjoint 30-example audio set was then frozen and evaluated once: 8/8 explicit commands detected, with 4 false positives among 22 non-command controls; accuracy 0.867, precision 0.667, recall 1.000. The false positives came from questions/discussion containing command-like verbs. Because of those false positives, this parser is not suitable as an intent/dispatch gate. The split is now consumed and must not be used for tuning. + +The parser and `--audio-parser-holdout-only` mode remain offline evaluator code. The mode does not load Laya/Torch, create proposals, or call tools/hardware. This experiment does not change production prompts, routing, or `action_proposals.enabled: false`; no rollout gate was closed. + +## 2026-09-27 — request-bound proposal validation + +Added pure helpers to bind an inert proposal to the current request ID and validate its exact schema, allowlisted `set_lights` name, exact typed argument keys/values, confidence threshold, and request ID. Wrong-request, malformed, extra-field, unsupported-schema, low-confidence, or unapproved proposals fail closed. Regression tests cover these cases. The validator is not wired to ToolRegistry or any dispatcher; it cannot execute an action, and the feature remains disabled. This closes only the pure validation substep; dispatcher, request lifecycle integration, independent intent quality, emergency stop, and hardware acceptance remain open. + +The agent request path now generates a separate internal UUID per admitted turn and binds any inert Laya action proposal to it. `trace_id` remains observability metadata and is not treated as request authority because API callers can supply it. The UUID is cleared with the turn; the native-tool execution path still does not validate or dispatch Laya proposals. + +## 2026-09-27 — tool-owner safety review + +The current `set_lights` compatibility tool calls `queue_action("lights", ...)`; that reaches the agent-core `ActionArbiter` and its configured `ActionSafetyFilter`, then the registered lights handler claims the expression-light lease and calls the Autonomy client. The filter currently has no lights-specific capability approval, and this handler path does not call Autonomy's `CapabilityExecutor`. `ToolRegistry.execute()` has its separate tool-execution arbiter, but that alone is not the runtime owner's capability decision. Therefore no Laya dispatcher or execute-calling simulation seam was added: it would not prove the required owner-side safety gate. A safe dispatcher remains blocked on an explicit Autonomy-owned capability approval path for Laya-originated lights requests, with tests showing denial produces zero handler/hardware calls. Existing Laya feature configuration remains disabled. diff --git a/.sentrybot/context/laya-system1-integration.md b/.sentrybot/context/laya-system1-integration.md new file mode 100644 index 00000000..f7925aab --- /dev/null +++ b/.sentrybot/context/laya-system1-integration.md @@ -0,0 +1,154 @@ +# SentryBOT — Laya (System 1) Karar Motoru Entegrasyon Mimarisi + +> **Doküman Durumu:** Tasarım ve Entegrasyon Haritası +> **Konsept:** İki Sistemli Yapay Zekâ Mimarisi (Dual-System AI: System 1 Refleks + System 2 Derin Zekâ) +> **Hedef:** Karar gecikmesini 1500ms'den 30ms'ye indirmek, kırılgan regex/keyword kurallarını ortadan kaldırmak ve Pi 5 üzerinde harici sunucusuz anlık refleks kazandırmak. + +--- + +## 1. Vizyon: Neden System 1 (Laya) ve System 2 (Ollama)? + +Şu anki SentryBOT mimarisinde tüm kararlar ya **kırılgan anahtar kelime eşlemelerine** (regex, string lookup) ya da **ağır ve uzaktaki bir LLM'e** (Ollama `qwen3.5:9b`) dayanmaktadır. +* **Anahtar kelimeler:** *"ışıkları yak"* çalışırken *"ortamı aydınlat"* kaçırılabilir; *"kırmızı yap"* çalışırken *"kızar"* anlaşılmayabilir. +* **Ollama (LLM):** Zengin ve akıllıdır ama tek bir "evet/hayır" veya "hangi modüle gideyim" kararı için 1.5 - 3 saniye ağ ve üretim gecikmesi yaratır. + +### Çift Sistemli (Dual-System) Çözüm: + +```mermaid +flowchart TD + INPUT["Kullanıcı Girdisi / Sensör Algısı"] --> S1["SYSTEM 1: Laya (322M mmBERT)
⚡ ~30 ms Yerel Refleks"] + + S1 -->|"Basit Komut / Refleks
(Confidence > 0.85)"| DIRECT["Doğrudan Eylem / Fast-Path
LED, Kafa, Duygu, Acil Durum"] + DIRECT --> ACT["Donanım / Eylem (NeoPixel, Arduino, OLED)"] + + S1 -->|"Sohbet / Derin Planlama
(Confidence < 0.85 veya chit_chat)"| S2["SYSTEM 2: Ollama Qwen 9B
🧠 1000-2500 ms Derin Düşünce"] + S2 --> SYNTH["Persona Sentezi & Tool Calling"] + SYNTH --> ACT +``` + +* **System 1 (Laya):** Hızlı, içgüdüsel, tiplendirilmiş karar motoru. Non-autoregressive (tek seferde okur, metin üretmez, doğrudan `choice`, `score`, `boolean` döndürür). 30ms içinde kararı verir. +* **System 2 (Ollama):** Analitik, üretici, felsefi ve kişiliği taşıyan ana zekâ katmanı. Yalnızca gerçekten derin akıl yürütme veya serbest sohbet gerektiğinde devreye girer. + +--- + +## 2. Modül Modül Etki Analizi: Neler Kalır, Neler Silinir, Neler Eklenir? + +### 1. `modules/agent_core` (Orkestrasyon ve Yönlendirme) + +* **🗑️ Neler Silinir / Hafifletilir:** + * `services/tri_layer.py`: + * `_score_keyword_matches` ve elle yazılmış token listeleri (`_LIGHT_TOKENS`, Türkçe ekler, vb.). + * `_apply_semantic_priors` içindeki 50+ satırlık statik öncelik kuralları. + * Regex tabanlı alt-ajan puanlama mantığı. + * `services/agent.py`: + * Fast-path kararındaki kaba `len(prompt) <= 140` gibi karakter/uzunluk heuristikleri. +* **✨ Neler Eklenir / Değişir:** + * `services/laya_router.py`: Laya `Router` veya `convaiinnovations/laya-multilingual` checkpoint'i kullanılarak: + * **Soru 1 (Routing):** `target_module` → `{"neopixel", "speak", "autonomy", "arduino_serial", "vlm_bridge", "general_chat"}` + * **Soru 2 (Complexity):** `is_direct_command` → `boolean` (Doğrudan donanım komutu mu yoksa diyalog mu?) + * **Soru 3 (Urgency):** `urgency` → `score` (0.0 - 1.0) + * Router süresi **250ms'den 30ms'ye** düşer. +* **🛡️ Neler Kalır (Dokunulmaz):** + * `ToolRegistry` ve tüm 27 tool (`set_lights`, `oled_face`, `move_head`, `speak` vb.). + * `SpeechArbiter`, `ActionArbiter` (kiralama/lease mekanizması). + * `_synthesize_main_persona` (SentryBOT kişiliğinin son cevabı üretmesi - System 2). + +--- + +### 2. `modules/autonomy` (Duygu, Karar ve Otonomi Döngüsü) + +* **🗑️ Neler Silinir / Hafifletilir:** + * `services/appraisal_triggers.py`: + * Hardcoded küfür/hakaret listeleri (`insult = ("aptal", "salak", "gerizekalı", ...)`). + * Hardcoded övgü listeleri (`praise = ("harikasın", "aferin", "seviyorum", ...)`). + * Hardcoded teşekkür ve sevgi listeleri (`thanks`, `petted`). + * `services/brain.py`: + * `_check_emotion_command` içindeki statik eşleşme kuralları. + * Kullanıcı niyetini yakalamaya çalışan statik if/else sentiment kontrolleri. +* **✨ Neler Eklenir / Değişir:** + * `services/laya_appraisal.py`: + * Kullanıcı cümlesini anında semantik appraisal olaylarına çevirir: + * `user_insult`, `user_rude`, `user_praise`, `user_thanks`, `petted`, `emergency`. + * Örnek: Kullanıcı *"Harika bir iş çıkardın küçük dostum"* dediğinde liste araması olmadan `user_praise (conf: 0.96)` olarak etiketlenir. + * **İhtiyaç Modeli (Needs):** + * Laya `context_appraisal`: Robotun o anki çevre/kullanıcı durumuna göre uyarılma veya dinlenme ihtiyacını milisaniyeler içinde sınıflandırır. +* **🛡️ Neler Kalır (Dokunulmaz):** + * `AffectiveAppraisal` matematik motoru ve `config/appraisal.yml` (duygu delta formülleri). + * `AutonomyBrain` Sense-Think-Act döngüsü. + * `GoalStore` ve tekil öncelikli yaşam döngüsü (`LifeLoop`). + * `ExpressionDirector` (duyguya göre LED/OLED/ses orkestrasyonu). + +--- + +### 3. `modules/ai_provider` (Ollama & LLM) + +* **🗑️ Neler Silinir:** + * Hiçbir şey silinmez! +* **✨ Neler Değişir:** + * Ollama'nın üzerinden **mikro-yönlendirme yükü kalkar**. + * Ollama artık yalnızca **gerçek düşünme gerektiren System 2 görevlerinde** (`chat`, `persona`, `creative response`, `complex reasoning`) çağrılır. + * Robotun basit komutlarda uzaktaki Ollama'ya gitme zorunluluğu ortadan kalkar; bu da gereksiz LLM çağrılarını **%60–%70 oranında azaltır**. +* **🛡️ Neler Kalır:** + * `xOllamaService.py`, `chat.py`, `clients.py` tamamen korunur. + +--- + +### 4. `modules/interactions` (Refleks Kural Motoru) + +* **🗑️ Neler Silinir / Hafifletilir:** + * Kural motorundaki aşırı karmaşık metin tabanlı olay tetikleyicileri sadeleşir. +* **✨ Neler Eklenir:** + * Laya'nın ürettiği `urgency` ve `sentiment` etiketleri doğrudan `interactions.event` olarak yayınlanır. + * Örnek: Yüksek `urgency` skoru gelirse, interactions anında NeoPixel'de kırmızı çakar refleksi tetikler. +* **🛡️ Neler Kalır:** + * Event-driven mimari, CPU sıcaklığı kuralları, donanım koruma kilitleri. + +--- + +### 5. Donanım ve Çıktı Modülleri (`neopixel`, `visual_output`, `voice`, `arduino_serial`) + +* **DURUM:** **TAMAMEN KORUNUR (SIFIR SİLİNME)** +* Bu modüller zaten eylem icra edicilerdir (actuators). Laya entegrasyonu bunların API'lerini veya donanım sürücülerini değiştirmez. +* **Tek fark:** Bu modüllere giden komutlar LLM'in 2 saniyelik düşünme gecikmesinden kurtulup **30 milisaniyelik refleks hızına** kavuşur. + +--- + +## 3. Sistem Mimarisi ve Karar Akış Şeması + +```mermaid +sequenceDiagram + autonumber + actor User as Kullanıcı + participant Voice as Speech / STT + participant S1 as Laya (System 1 Engine) + participant Core as Agent Core / Router + participant S2 as Ollama (System 2 LLM) + participant Act as Donanım (NeoPixel/Arduino/TTS) + + User->>Voice: "Işıkları hemen mavi yap" + Voice->>Core: Metin: "Işıkları hemen mavi yap" + Core->>S1: predict(state, questions) [~30ms] + Note over S1: intent: neopixel
action: set_color(blue)
is_direct: True (0.98)
urgency: 0.85 + + alt Doğrudan Komut (Fast-Path / S1) + S1-->>Core: Karar: Direkt Neopixel Eylemi + Core->>Act: /neopixel/color (Mavi) [~10ms] + Core->>Act: TTS: "Maviye geçiyorum." [Fast Ack] + Act-->>User: Anında Mavi Işık + Ses (Toplam: ~100ms!) + else Derin Sohbet / Bilgi Sorusu (S2) + S1-->>Core: Karar: chit_chat / complex_reasoning (is_direct: False) + Core->>S2: Ollama Chat / Persona [~1500ms] + S2-->>Core: Zengin yanıt ve kişisel diyalog + Core->>Act: TTS Cümle Akışı + Act-->>User: Doğal Diyalog Yanıtı + end +``` + +--- + +## 4. Raspberry Pi 5 Uyumluluğu ve Kaynak Tüketimi + +* **Model:** `laya-multilingual` (322M parametre mmBERT-base). +* **Bellek (RAM):** Yalnızca ~650 MB RAM kaplar (Pi 5'in 8GB RAM'i içinde ihmal edilebilir bir pay). +* **CPU Gecikmesi:** Pi 5'in Cortex-A76 4 çekirdekli işlemcisinde ONNX Runtime veya PyTorch CPU backend ile tek çıkarım (single forward pass) **~45-65 milisaniye** sürmektedir. +* **Tamamen Çevrimdışı (Air-gapped):** İnternet veya harici PC gerekmez; internet kopsa bile robot temel komutları ve hisleri yerel olarak anlar. diff --git a/.sentrybot/laya_integration_completed.md b/.sentrybot/laya_integration_completed.md new file mode 100644 index 00000000..18edc6f6 --- /dev/null +++ b/.sentrybot/laya_integration_completed.md @@ -0,0 +1,324 @@ +# Laya System 1 Entegrasyonu — YAPILAN İŞLER + +> **Son güncelleme:** 2026-09-27 +> **Durum:** Dispatch kapalı güvenli entegrasyon ve doğrulama fazı tamamlandı. Bu, donanıma aksiyon rollout'unun tamamlandığı anlamına gelmez. `action_proposals.enabled: false` kalır; Autonomy capability gate, bağımsız veri kabulü, simülasyon dispatch güvenliği, emergency-stop kontratı ve Raspberry Pi kabulü tamamlanana kadar eylem açılmayacak. +> **Son test:** `pytest tests -x -q` — **970 passed in 43.98s**. Ayrıca py_compile ve git diff --check temiz. +> **Kullanım:** Bu dosya yapılanların kayıt defteri; kalan işleri ve sırayı `.sentrybot/laya_integration_todo.md` başındaki güncel durum bloğu belirler. + +--- + +## 1. Laya Model İndirme Aracı + +**Dosya:** `tools/download_laya_model.py` +**Durum:** ✅ Tamamlandı, çalışıyor +**Ne yapıyor:** Hugging Face'den `convaiinnovations/laya` modelini `~/.cache/huggingface/` altına indirir. +**Test:** `python tools/download_laya_model.py` ile çalıştırılabilir. + +--- + +## 2. LayaEngine — System 1 Nöral Karar Motoru + +**Dosya:** `modules/agent_core/services/laya_engine.py` (366 satır) +**Durum:** ✅ Oluşturuldu, çalışıyor + +### Sınıf: `LayaDecision` (dataclass, satır 42-57) +Laya çıkarımının sonucu. Alanlar: +- `target_module: str` — hedef modül (neopixel, arduino_serial, emergency_stop, speak, camera, system2_chat) +- `module_confidence: float` — modül güven skoru +- `is_direct_command: bool` — doğrudan donanım komutu mu? +- `direct_confidence: float` +- `affective_event: str` — user_praise, user_rude, neutral +- `affect_confidence: float` +- `urgency_score: float` — 0-3 arası aciliyet +- `urgency_confidence: float` +- `inference_ms: float` — çıkarım süresi +- `suggested_tool: Optional[str]` — önerilen tool adı +- `is_emergency` property — urgency >= 2.0 veya emergency_stop + +### Sınıf: `LayaEngine` (satır 64-366) +- **Singleton pattern** — `get_instance(config)` ile tek instance +- **`SENTRYBOT_QUESTIONS`** (satır 75-115) — Laya'ya sorulacak 4 soru tanımı: + 1. `target_module` — 6 seçenekli modül sınıflandırma + 2. `is_direct_command` — direct_action vs conversation + 3. `affective_event` — user_praise, user_rude, neutral + 4. `urgency` — 0-3 skor (sakin → son derece acil) +- **`MODULE_TOOL_MAP`** (satır 117-123) — modül → tool eşleme +- **`_ensure_loaded()`** (satır 185-229) — Lazy model loading, `import laya` → `laya.load()` +- **`_find_cached_snapshot()`** (satır 162-183) — HF cache'de model arama +- **`decide(text)`** (satır 235-285) — Ana çıkarım metodu, `LayaDecision` döner +- **`route(user_prompt, available_modules)`** (satır 287-313) — Modül yönlendirme +- **`should_fast_path(user_prompt)`** (satır 315-328) — Fast-path kararı +- **`get_instant_filler(language)`** (satır 330-335) — Düşünme ara sesi +- **`get_fast_ack(language)`** (satır 337-342) — Hızlı onay sesi +- **`get_affective_reaction(affect)`** (satır 344-365) — OLED yüz + LED renk + +### Config okuması: +```python +# config["tri_layer"]["laya"] altından okur: +# enabled, confidence_threshold, model_path +``` + +--- + +## 3. TriLayerRouter Güncellemesi + +**Dosya:** `modules/agent_core/services/tri_layer.py` +**Durum:** ✅ Güncellendi + +### Değişiklikler: +- **`__init__` parametresi** (satır 293): `laya_engine: Any = None` eklendi +- **`route()` metodu** (satır 398-424): Laya nöral routing eklendi: + ```python + # satır 403-408: Laya modüllerini al + if self.laya_engine is not None and getattr(self.laya_engine, "enabled", True): + laya_modules = self.laya_engine.route(user_prompt, list(self.profiles.keys())) + + # satır 415-417: Laya modüllerine +10.0 sabit boost + for mod in laya_modules: + if mod in self.profiles: + scores[mod] = scores.get(mod, 0.0) + 10.0 + ``` + +--- + +## 4. AgentContextMixin — Laya Başlatma + +**Dosya:** `modules/agent_core/services/agent_context.py` +**Durum:** ✅ Güncellendi + +### Değişiklikler: +- **`_init_tri_layer()` metodu** (satır 307-344): + ```python + # satır 318-320: + from .laya_engine import LayaEngine + self.laya_engine = LayaEngine.get_instance(config) + # Router'a laya_engine parametre olarak verildi: + self.router = TriLayerRouter(..., laya_engine=self.laya_engine) + ``` + +--- + +## 5. AgentOrchestrator — Laya step() Entegrasyonu + +**Dosya:** `modules/agent_core/services/agent.py` +**Durum:** ✅ Kısmi entegrasyon (MVP seviyesinde) + +### Değişiklikler (satır 149-166): +```python +# step() içinde Laya kullanımı: +laya_eng = getattr(self, "laya_engine", None) +if laya_eng and getattr(laya_eng, "enabled", True): + # 1. Filler üret (LLM beklerken) + custom_ack = laya_eng.get_instant_filler(language=session_language) + # 2. Affective event → autonomy expression + laya_dec = laya_eng.decide(user_prompt) + if laya_dec and laya_dec.affective_event in {"user_praise", "user_rude"}: + reaction = laya_eng.get_affective_reaction(laya_dec.affective_event) + autonomy_client.set_expression_event("agent.affect", reaction) +``` + +**EKSİK:** Fast-path (LLM atlama), urgency gate, tool hint, world state update — bunlar Faz 3'te yapılacak. + +--- + +## 6. AgentStreaming — Clause-Level Streaming + +**Dosya:** `modules/agent_core/services/agent_streaming.py` +**Durum:** ✅ Güncellendi + +### Değişiklikler (satır 32-33, 92-104): +```python +_FIRST_CLAUSE_SPLIT_RE: Optional[re.Pattern] = None +# ... +# satır 92-93: Ultra düşük gecikmeli ilk cümlecik bölme +AgentStreamingMixin._FIRST_CLAUSE_SPLIT_RE = re.compile(r"([,;:—–\n]+|[.!?…]+)") + +# satır 96-104: İlk chunk'ta virgül/bağlaçtan böl +if is_first_chunk: + words = buffer.split() + if len(words) >= 2 or len(buffer) >= 12: + parts = _FIRST_CLAUSE_SPLIT_RE.split(buffer) + if len(parts) > 1: + first_clause = (parts[0] + parts[1]).strip() + ... +``` + +--- + +## 7. MoodManager — Affective Event İşleme + +**Dosya:** `modules/autonomy/services/mood.py` +**Durum:** ✅ Kısmi güncelleme + +### Değişiklikler (satır 129-141): +```python +def apply_affective_event(self, event_name: str) -> str: + event = str(event_name or "").strip().lower() + with self._lock: + if event == "user_praise": + self.modify("happiness", 25) + self.modify("anger", -15) + self.modify("social", 20) + elif event == "user_rude": + self.modify("anger", 30) + self.modify("happiness", -25) + self.modify("fear", 10) + return self.get_dominant_emotion() +``` + +**EKSİK:** `apply_laya_decision(decision: LayaDecision)` — tam karar nesnesi alması, urgency'yi de işlemesi gerekiyor. Faz 2'de yapılacak. + +--- + +## 8. Test Dosyaları + +### `tests/modules/agent_core/test_laya_engine.py` +**Durum:** ✅ Oluşturuldu +- LayaDecision dataclass testleri +- LayaEngine singleton test +- `route()` ve `should_fast_path()` testleri +- Mock-based, `laya` kütüphanesi gerekmez + +### `tools/test_sentrybot_laya_live.py` +**Durum:** ✅ Oluşturuldu +- Canlı model testi (gerçek Laya modeli gerekir) +- Raspberry Pi üzerinde çalıştırılmak üzere tasarlandı + +--- + +## 9. Config Durumu + +**Dosya:** `config/agent.yaml` +**Durum:** ⚠️ Laya section henüz eklenmedi + +Mevcut yapıda Laya config'i `tri_layer.laya` altından okunuyor (satır 150-154 laya_engine.py): +```python +tri_cfg = cfg.get("tri_layer", {}) +laya_cfg = tri_cfg.get("laya", {}) +enabled = bool(laya_cfg.get("enabled", True)) +threshold = float(laya_cfg.get("confidence_threshold", 0.60)) +model_path = laya_cfg.get("model_path") +``` + +**Config'de şu an bu key yok** — default değerler kullanılıyor. Faz 1'de eklenecek. + +--- + +## 10. İlk Taslakta Dokunulmamış Olarak İşaretlenen Dosyalar + +> Bu tablo ilk entegrasyon taslağının tarihsel durumudur; 2026-09-24 devam notu güncel durumu belirtir. + +Aşağıdaki dosyalar **HİÇ değiştirilmedi** ve Faz 1-7'de değiştirilecek: + +| Dosya | Faz | +|-------|-----| +| `modules/autonomy/services/brain_init.py` | 2 | +| `modules/autonomy/services/brain.py` | 2 | +| `modules/autonomy/services/system1_reflex.py` | 2 | + +## 11. 2026-09-24 Devam Oturumu + +- Config, `decide_with_context`, telemetri, mood-aware filler, mood delta ve spinal cord voice emergency desteği önceki çalışma ağacında zaten uygulanmıştı; Faz 0 diye belgelendirilmiş durum güncellendi. +- `TriLayerRouter` Laya güven skoruna göre dinamik ağırlık veriyor (5–15). +- `AgentSubagentsMixin._route_subagents()` artık aynı `TriLayerRouter` sonucunu kullanıyor; keyword tabanlı ikinci routing yolu kaldırıldı. +- Agent turn fast-path kontrolünde üretilen Laya kararı aynı turdaki affective tepki için tekrar kullanılıyor; gereksiz ikinci inference önlendi. +- `LayaDecision.is_emergency` artık engine config'indeki `emergency_threshold` değerini taşıyor; varsayılan 2.0 ve eski kurucularla uyumlu. +- `WorldState.update_laya_state()` son kararı thread-safe sensör state'ine yazar; agent turu bu güncellemeyi yapar. +- VisualContextCache sahneyi kişi/nesne/tehlike özeti olarak Laya için kısaltabilir. +- Laya filler kuyruğu progress ve final yanıt önceliğinin üstünde, safety önceliğinin altında ve 4 saniyelik ömre sahip; ProgressManager bunu destekleyen arbiter'larda kullanıyor. +- SensorFeedbackLoop yakın zamanda yüksek urgency görülünce yapılandırılmış süre boyunca daha sık tick eder. `urgency.sensor_boost_hold_s` eklendi; donanım kanalının kendi polling bütçeleri değişmeden kalıyor. +- Konuşma sürerken Laya acil karar verirse, BargeInController tek kelimelik acil ifadeyle TTS'i kesebilir; var olan cooldown ve wakeword davranışı korunur. Model önceden yüklenmemişse barge-in yolu yeni model yükleyip gecikme yaratmaz. +- Agent fast-path kararı ve affective tepki, `WorldState`'teki güncel sensör ve kısa VLM sahne özetiyle `decide_with_context()` üzerinden çalışır. `should_fast_path()` yeni parametreleri varsayılanlı aldığı için önceki çağrılarla uyumludur; eski adapter'lar için fallback korunur. +- VLM Bridge `face_emotion` event'i yalnızca yapılandırılmış confidence eşiği üstünde yayınlar; aynı kişi/duygu kombinasyonu cooldown boyunca bastırılır. Gateway event bridge'i bu sinyali mevcut agent-event yoluna iletir ve autonomy Brain'e bağlar. +- Autonomy Brain, yüz duygu event'ini Laya'ya verir; yalnızca Laya'nın `user_praise`/`user_rude` sınıfı da yapılandırılmış affect confidence eşiğini geçtiğinde mood değişir. Düşük güven, nötr, bilinmeyen veya tekrar event'leri mood'u değiştirmez. Bu yol Laya henüz yüklenmemişse yüklemeyi tetiklemez. +- Gateway VLM event bridge'i `owner_seen`, `new_person`, `hazard_detected` ve `scene_changed` olaylarını da worker üzerinden Laya vision handler'ına aktarır. Laya karar özeti state'e yazılır; bu olaylar tek başına motor/actuator veya mood değişikliği yapmaz. Olay/kişi başına cooldown tekrarları bastırır. +- Autonomy görsel tehlikeyi artık `loud_noise` olarak değil ayrı `visual_hazard` appraisal olayı olarak değerlendirir; YAML kuralı fear/energy/curiosity tepkisini tanımlar. VLM event bridge tehlike için actuator tetiklemez; mevcut autonomy appraisal sahibi kalır. +- Regression testleri eklendi. Önceki hedefli test kümesi **52 passed**; son eklerle genişletilen test kümesi de **52 passed** verdi. +- Tam `pytest tests/ -x -q` denemesi, `pydantic_core._pydantic_core` binary modülü yüklenemediğinden test toplamaya başlarken durdu. `.venv` launcher'ı başka makinedeki Python yoluna işaret ediyor; hazır runtime Python'ına `.venv/Lib/site-packages` bağlamak pure-Python testleri çalıştırdı ama native modül uyumsuzluğunu çözmedi. + +### Devam eden maddeler / güvenlik notu + +- Faz 3.1'de taslaklanan sınıflandırma tabanlı doğrudan tool çalıştırması eklenmedi: `suggested_tool` argümanları içermez ve emergency için `stop_follow` eşlemesi robotu güvenli biçimde durdurmayı garanti etmez. Sesli emergency zaten SpinalCord üzerinden `motor.stop` ile ele alınıyor. Agent seviyesindeki Laya fast-path LLM yolunu kısaltır. +- Faz 3.4'ün Laya filler önceliği, 3.5 WorldState özeti ve Faz 4.3 sahne özeti tamamlandı. Faz 3.6 sensör urgency tick'i de uygulandı. +- Faz 3.7 ActionArbiter bypass uygulanmadı; safety kontrolleri korunur, emergency stop spinal reflex sahibindedir. Faz 4.1 kişi/tehlike semantiği mevcut perception/needs/appraisal sahiplerine bağlıdır; vision Laya handler ikinci bir mood etkisi üretmez. Faz 3.8 idle heartbeat kararı aşağıda kayıtlıdır. Faz 5.1–7 tamamlanma kayıtları da aşağıdadır. +- Fast-path + sahne bağlamı test kümesi **46 passed** verdi; Python derlemesi ve `git diff --check` temiz. Tam test paketi için önceki nottaki native `pydantic_core` ortam engeli sürüyor. +- VLM yüz-duygu event zinciri için hedefli testler **47 passed** verdi. +- Genişletilmiş VLM event ve ilgili regression kümesi **41 passed** verdi; derleme ve diff kontrolleri temiz. +- Görsel tehlike appraisal ve VLM/autonomy regression kümesi **21 passed** verdi. +- Faz 5.1 mood-aware streaming tamamlandı: AutonomyBrain, yerel MoodManager'ı AgentOrchestrator'a bağlar; step/step_event tek mood tone snapshot'ını tüm cümle chunk'larına geçirir. Mood okuması başarısızsa eski nötr/varsayılan TTS yolu korunur. +- Faz 5.2 endpoint taslağı ayrı bir `/speak/laya_filler` API'si eklemeden tamamlandı: Laya ACK zaten ProgressManager → öncelik/TTL kontrollü SpeechArbiter → mevcut speak TTS sınırından geçiyor. Endpoint pipeline'ı bypass edip koordinasyonu bozacağından gereksiz. Yol için regression testi eklendi. +- Faz 6.1 Laya tool hint telemetrisi tamamlandı: native tool dispatch mevcut LLM kararını uygular; Laya'nın önerdiği tool yalnızca successful status event'inde eşleşme telemetrisi olarak görünür, çalıştırma yetkisi sağlamaz. +- Faz 6.2 Laya inference süresi `laya.inference` latency trace event'i olarak kaydedilir. LLM TTFT ile farklı ölçüm olduğu için ayrı tutulur; event bu Agent turunda karar oluştuysa eklenir. +- Faz 6.3/6.4 taslakları güvenli mimari kararıyla uygulanmadı: ToolExecutionArbiter kaynak kilitlerini korur; acil durdurma spinal cord üzerinden çalışır ve genel SafetyFilter override'ı eklenmez. Bu separation, Laya emergency reflex ve barge-in testleriyle mevcut durumda doğrulanmıştır. +- Faz 7 kapsamı tamamlandı: Laya engine context/telemetry/mood filler testleri zaten vardı; emergency/direct/conversation fast-path ayrımı eklendi. Yeni autonomy integration testleri emergency spinal motor stop ve non-emergency no-stop davranışlarını doğrular. VLM event testleri mevcut dosyalarda yeterli olduğundan tekrar üretilmedi. Speech ACK→filler queue ve filler-before-final seçimi test edildi. +- 2026-09-25 Faz 7 ve ilişkili regression kümesi: **61 passed**; ilgili yeni testler `py_compile` ile derlendi, `git diff --check` temiz. Tam test paketi bu turda çalıştırılmadı; bilinen native `pydantic_core._pydantic_core` toplama engeli önceki oturum notlarında kayıtlı. +- 2026-09-25 açık taslak denetimi: 2.6 mevcut MoodManager→`_speak_with_mood` zinciriyle karşılandı ve gerçek mood tone regresyon testi eklendi; ikinci composer karar katmanı eklenmedi. 3.8 için heartbeat'in Laya/semantic action çalıştırmadığını doğrulayan sahiplik regresyon testi eklendi; semantik idle planı Autonomy sahibinde kalır. 2.7 freshness korumalı semantic safety goal overlay olarak tamamlandı. +- Faz 2.7 regression doğrulaması: companion goal, pet-goal, Laya brain integration, vision-event ve Laya engine testleri **52 passed**; ardından brain + companion goal altkümesi **23 passed**. Autonomy YAML parse edildi, `git diff --check` temiz. +- Faz 2.6/3.8 regression doğrulaması ve geniş Laya ilişkili küme: BehaviorComposer mood-tone aktarımı, idle heartbeat sahipliği, GoalSelector TTL, spinal reflex, barge-in ve VLM olayları dahil **96 passed**; diff kontrolü temiz. +- Faz 3.1/3.7/4.1 güvenlik ve sahiplik sınırı doğrulandı: non-face owner/person/hazard vision event'leri Laya'ya sınıflandırma için ulaşır, ancak ikinci mood etkisi üretmez. Direct tool dispatch ve action safety bypass eklenmez. +- BehaviorComposer `say` planı artık isteğe bağlı `language` alanını `_speak_with_mood()`'a aktarır; alan yoksa önceki çağrı imzası/algılama davranışı korunur. Mood-tone regression testi dili açıkça belirleyerek deterministik hale getirildi. +- Tam doğrulama (2026-09-25): `pytest tests -x -q` **938 passed in 67.62s**. Komut hazır Python 3.12 runtime ile çalıştırıldı; runtime `Lib/site-packages` yolu `.venv/Lib/site-packages` yolundan önce ayarlanarak cp312 `pydantic_core` kullanıldı. Python derlemesi ve `git diff --check` temiz. +- Agent adapter regresyonu eklendi: Laya fast-path sınıflandırmasının world/visual context ile çağrıldığını ve kararın aynı tur için cache'lendiğini; normal sohbetin kısa olsa bile System 2'de kaldığını; sınıflandırma sırasında ToolRegistry dispatch yapılmadığını doğrular (`tests/modules/agent_core/test_laya_agent_fast_path.py`). Hedefli Agent/Laya kümesi **29 passed**. +- Son tam doğrulama (2026-09-25): `pytest tests -x -q` **940 passed in 41.68s**; testler hazır Python 3.12 runtime ve runtime `Lib/site-packages` önce gelecek şekilde çalıştırıldı. Kalan açık işler: güvenli ve yapılandırılmış Laya action argüman/actuator sözleşmesi oluşana kadar doğrudan Laya tool dispatch'i yok; arbiter/safety bypass yok; gerçek hedef cihazda canlı model/donanım kabul koşusu yapılmadı. +- Faz 5.3/5.4 tamamlandı olarak belgelendi: Brain VocalMixin, yüklü Laya modelinden emergency kararını alıp `BargeInController`'a geçiriyor; tek kelimelik acil ifade TTS'i kesiyor. LayaEngine config'indeki emergency threshold tek kaynak olarak kalıyor. Uçtan uca regression `test_laya_emergency_reaches_brain_barge_in_controller` eklendi. +- Güncel tam doğrulama (2026-09-25): `pytest tests -x -q` **941 passed in 41.48s**. Açık işler güvenli action argüman/actuator sözleşmesinin tasarımı ve gerçek hedef cihazda canlı model/donanım kabul koşusudur; arbiter ve safety bypass'ları uygulanmaması bilinçli karardır. +- Laya doğrudan action tasarım notu `.sentrybot/context/laya-action-contract.md` eklendi. Mevcut engine'in yalnızca hedef modül sınıflandırdığı ve `suggested_action` alanını doldurmadığı kayda geçirildi. Önerilen proposal şeması, fail-closed doğrulama/red koşulları, ToolRegistry/arbiter/runtime safety üzerinden dispatch gereksinimleri ve gerçek SpinalCord motor-stop rollout kapıları tanımlandı. Bu dokümanla dispatch açılmadı. +- Çıktı sözleşmesi araştırması: kurulu Laya 0.3.5 `Agent.system_one()` yalnız `choice`, `score`, `noul` başlıklarını işler; her biri typed answer, confidence/probabilities döndürür. JSON/action argument head yok; finite safe proposal gelecekte birden çok bounded choice cevabından türetilebilir. Cache model ile gerçek inference denemesi Python 3.12 + mevcut venv Torch `torch_python.dll` bağımlılığı nedeniyle WinError 126 ile yüklenemedi; model davranışı ek action soruları için hâlâ canlı doğrulanmadı. Dispatch kapalı kaldı. +- Uyumluluk çözümü denemesi: izole `.laya_runtime` hedefine CP312 Torch kurulumu onaylandı; 124.1 MB wheel indirmesi 7.6 MB'tan sonra yaklaşık 73 kB/s hız ve tekrarlı bağlantı kesintisi nedeniyle durduruldu. Paket kurulmadı, model inference yapılamadı. Bir sonraki çalışma ya çalışan CP311 runtime/Pi kullanmalı ya da CP312 wheel erişimini çözmeli. +- Donanım çağrısı yapmayan typed action proposal altyapısı eklendi: `laya_action_contract.py` yalnız confidence doğrulanmış, allowlist'teki `set_lights` için finite renk/efekt seçimlerini inert dict önerisine dönüştürür. Config `action_proposals.enabled: false` ile kapalı ve ToolRegistry dispatch bağlantısı yoktur. LayaEngine bu iki soru grubunu yalnızca feature açıldığında inference'a ekler. Testler invalid target/direct flag/allowlist/choice/confidence değerlerinde fail-closed davranışı ve default-off ayarını doğrular. Legacy `emergency_bypass_safety` default/config false yapıldı. +- Son tam doğrulama (2026-09-25): **958 passed in 47.79s**. Hedefli action contract + Laya engine kümesi 42 testte geçti; gerçek model inference'ı Torch indirme bağlantısı nedeniyle hâlâ beklemede. +- Mevcut Python 3.11 çalıştırıcısı altında cache model inference'ı tamamlandı. Bu, önceki notu günceller: Python 3.12 Torch wheel indirmesine artık gerek kalmadı. Gerçek Türkçe örneklerde eski emergency predicate'in `Işıkları kapat` (target emergency_stop 0.8273/direct 0.8297/urgency 2.059 confidence 0.541) ve normal sohbet (urgency 2.443 confidence 0.284) için false-positive üreteceği görüldü. `LayaDecision.is_emergency`, `is_emergency_decision`, `is_urgent` ve urgency mood effects confidence/direct gates ile düzeltildi. Açık emergency örnekleri target 0.934/direct 0.982 ve 0.950/0.970 ile kapıdan geçiyor. +- Deneysel action-color question setinde `Işıkları mavi yap` için renk `pink` confidence 0.2894 ve efekt `unspecified` confidence 0.3754 geldi. Fail-closed proposal doğru şekilde reddediyor; gerçek inference bu action alanlarının kalibre edilmediğini gösterdiği için `action_proposals.enabled` kapalı kalmalı. +- Emergency false-positive ve gerçek örnek regression kapsamı eklendi; hedefli Laya/emergency kümesi **55 passed**. Önceki 958 tam test sonucu confidence gate kodundan öncedir; yeni tam test koşusu bu değişikliklerle tekrar yapılmalıdır. +- Güncel tam doğrulama confidence gate değişiklikleriyle tekrarlandı: `pytest tests -x -q` **961 passed in 49.98s**. `git diff --check` ve değişen Laya Python dosyalarının `py_compile` kontrolü temiz. +- Offline gerçek-model kalibrasyon aracı `tools/evaluate_laya_action_proposals.py` eklendi. 14 Türkçe metin örneği için hedef/direct/color/effect seçimlerini, confidence kapılarından sonraki emergency kararını ve inert proposal üretimini raporlar; tool/hardware endpoint çağırmaz. Ölçüm: target top-choice 0.643, direct 0.929, açık renk 1/8 (0.125), açık effect 2/3, emergency 14/14, proposal 0. İlk rapordaki renk metriği unspecified negatiflerini sayıyordu; düzeltildi ve gerçek model tekrar çalıştırıldı. Bu sonuçlar renk proposal'ı için yetersiz; config kapalı tutulmalı. +- Düzeltilmiş summary-only ölçüm: target confidence>=0.60 coverage 0.571, kabul edilmiş target accuracy 0.625; direct confidence coverage 0.357, kabul edilmiş direct accuracy 1.0. Tekrar çalıştırılabilir komut: `python tools/evaluate_laya_action_proposals.py --summary-only` (Laya/Torch uyumlu Python 3.11 ortamı gerekir). +- Renk soru A/B deneyi: mevcut prompt 1/8 explicit renk doğruluğu, 5/6 no-color specificity ve 0 high-confidence false positive verdi. İki explicit/bilingual alternatif 8/8 explicit doğruluk ve 7/8 confidence coverage sağladı; ancak no-color specificity 1/6 ve 2/6'ya düştü, her birinde 2 high-confidence false color görüldü. Alt varyantlar uygulamaya alınmadı; mevcut conservative prompt ve kapalı feature korundu. Benchmark scripti şimdi current/TR-explicit/EN-explicit varyantlarını, confidence coverage/precision ve no-color high-confidence FP sayılarını raporluyor. +- Birleşik renk/kapat/no-light-action finite-choice deneyi: 14 örnekte genel doğruluk 0.500; explicit renk doğruluğu 0.750, confidence>=0.85 coverage 0.375, kabul edilen renk doğruluğu 1.000. No-color specificity 0.167 ve 3 high-confidence false positive. Birleşik soru da kabul edilmedi; proposal kapalı. Değerlendirme betiği yalnız offline inference yaptı ve hiç ToolRegistry/gateway/hardware çağırmadı. +- Evaluator 32 Türkçe calibration örneğine genişletildi (18 explicit color/off, 14 no-color kontrolü). Current color sorusu explicit doğruluk 0.111, no-color specificity 0.857 ve yüksek güvenli FP 0; Türkçe explicit varyant 1.000/0.778 coverage/0.214 specificity/4 FP; English-bilingual varyant 0.944/0.889 coverage/0.286 specificity/5 FP. Joint action varyantı overall 0.625, explicit color 0.778, coverage 0.222, specificity 0.429 ve 4 FP. Emergency 32/32, proposal 0. Sonuçlar feature'ın kapalı kalmasını destekliyor; bağımsız holdout ve deterministik parser baseline'ı henüz gerekli. +- Final holdout (30 örnek; 14 explicit color/off, 16 control; calibration/dev metinleriyle çakışma yok) üzerinde evaluator-only deterministik parser 0.967 overall doğruluk, 0.929 explicit doğruluk ve 1.000 unspecified özgüllük verdi. Laya current/TR/EN/joint overall doğruluk 0.433/0.533/0.567/0.500; explicit doğruluk 0.143/1.000/1.000/0.786; specificity 0.688/0.125/0.188/0.250. Açık TR/EN soruları sırasıyla 8 ve 9 yüksek güvenli false positive üretti. Parser production'a taşınmadı; küçük, elle yazılmış sette sınırlı kanıt olduğu ve action proposal kapalı kaldığı not edildi. Final holdout üzerinden tekrar kalibrasyon yapılmamalı. +- Sınıf başına beş örnek içeren yeni 30 örnekli intent/safety holdout'unda target accuracy 0.600 oldu (neopixel 0.400, arduino 1.000, emergency_stop 0.800, speak 0.200, camera 1.000, system2_chat 0.200). Target güven kapısı coverage 0.700/accepted accuracy 0.714; direct accuracy 0.900, coverage 0.700/accepted accuracy 1.000. Confidence-gated emergency 0.933, FP 0/FN 2; high-urgency 0.833, FP 0/FN 5. Effect 2/4 doğruluk ve 0/4 confidence kabulü verdi. Bu holdout değiştirilmeden kaldı; direct action/dispatch açılmadı. +- 32 örnekli dev split'te ek target/effect soruları ölçüldü. Uzun TR target varyantı 0.188 doğruluk, 0.174 accepted doğruluk ve 0.719 coverage; kısa EN target varyantı 0.500, 0.667 ve 0.188 verdi. Explicit effect varyantı beş örnekte 0.800 top-choice, 0.400 coverage, 0.500 accepted accuracy sağladı. Hiçbir aday seçilmedi; production config/prompt değişmedi, intent holdout kullanılmadı. +- 2026-09-26 dengeli 30 örneklik intent dev splitinde current/TR-explicit/EN-explicit target doğruluğu 0.467/0.500/0.600; accepted doğruluk 0.667/0.857/0.786 ve coverage 0.500/0.467/0.467 oldu. Effect alternatifleri 4 örnekte 0.500 doğrulukta kaldı. TR target adayı bir defa holdout'ta sınandı: target accuracy 0.600→0.433, accepted accuracy 0.714→0.647 ve emergency accuracy 0.933→0.900 (FN 2→3); reddedildi. Üretim davranışı ve proposal config'i değişmedi. Bu holdout gelecek prompt kalibrasyonu için artık kullanılmamalı; yeni candidate yeni untouched set gerektirir. +- İkili explicit-audio intent sorusu aynı dev splitinde 0.733 doğruluk aldı, ancak precision/recall 0.0 oldu (3 FP, 5/5 speak FN). Confidence .60/.80/.90 refiner hedef doğruluğunu artırmadı; aday reddedildi. +| `modules/autonomy/services/spinal_cord_reflex.py` | 2 | +| `modules/autonomy/services/behavior_composer.py` | 2 | +| `modules/autonomy/services/companion_goal_selector.py` | 2 | +| `modules/agent_core/services/world_state.py` | 3 | +| `modules/agent_core/services/sensor_loop.py` | 3 | +| `modules/agent_core/services/action_arbiter.py` | 3 | +| `modules/agent_core/services/idle_behavior.py` | 3 | +| `modules/vlm_bridge/services/processor.py` | 4 | +| `modules/vlm_bridge/services/vision_event_bus.py` | 4 | +| `modules/vlm_bridge/services/visual_context.py` | 4 | +| `modules/vlm_bridge/services/face_emotion.py` | 4 | +| `modules/voice/speak/xSpeakService.py` | 5 | +| `modules/voice/audio_router.py` | 5 | +| `modules/autonomy/services/barge_in.py` | 5 | +| `modules/agent_core/services/progress.py` | 6 | +| `modules/agent_core/services/safety_filter.py` | 6 | +- 2026-09-27: Offline explicit-audio literal parser evaluation completed. Intent dev: 30/30; separate 30-example audio holdout: 0.867 accuracy, 0.667 precision, 1.000 recall (4 false positives). Holdout findings reject parser for production gating. Added repeatable `--audio-parser-holdout-only` mode that lazily avoids Laya/Torch import; no dispatch/hardware path. No production behavior/config changed. +- 2026-09-27: Added pure request-ID binding and fail-closed exact-schema validation for inert `set_lights` proposals. It enforces request match, policy enablement/allowlist, exact fields/enums, and confidence floor. It is deliberately not connected to dispatch and does not enable production proposals. Targeted contract tests: 25 passed. +- 2026-09-27: Confirmed API `trace_id` can be caller-provided and should not authorize proposal binding. Agent `step()` now creates a distinct internal UUID for each admitted turn, binds any inert Laya proposal to it, and clears the active ID in `finally`. Added adapter regression coverage; no dispatcher/tool execution was connected. +- 2026-09-27: Reviewed the existing lights execution path before dispatcher work. `set_lights` enters agent-core ActionArbiter and its ActionSafetyFilter, then handler lease + Autonomy client; current filter has no lights-specific capability approval and the handler does not call Autonomy CapabilityExecutor. ToolRegistry arbiter is a separate resource gate. Dispatcher/simulation seam deferred until the runtime owner gate exists; proposal remains disabled. + +## 2026-09-27 — kapanış özeti + +### Bu fazda tamamlananlar + +- LayaEngine ve Agent/System 1 fast-path entegrasyonu; kısa sohbeti System 2'ye bırakma ve karar aşamasında araç çalıştırmama sınırı. +- Affective/vision sinyalleri, confidence-gated emergency kararları ve regression kapsamı. +- Fail-closed, yalnızca `set_lights` için typed inert proposal builder; exact schema/confidence/allowlist validator. +- Her Agent step için internal UUID ve Laya proposal request binding. Caller-provided `trace_id` yalnız telemetri kimliği. +- Offline kalibrasyon/evaluator, ayrık ama küçük ve elle etiketlenmiş color/intent/audio kümeleri; güvenli olmayan prompt/parser adayları reddedildi. +- 2026-09-27 son tam doğrulama: **970 passed**; action dispatch ve hardware çağrısı açılmadı. + +### Bilinçli olarak tamamlanmayan rollout işleri + +- Haricen gözden geçirilmiş, daha geniş intent/safety korpusu yok; var olan holdout'lar tükenmiş durumda. +- Mevcut ışık eylem yolu `ActionArbiter` ve `ActionSafetyFilter` kullanıyor, fakat Autonomy `CapabilityExecutor` lights onayı vermiyor/uygulamıyor. Laya-specific owner approval kapısı yok. +- Gerçek Laya modelinin uyumlu runtime ve Raspberry Pi üzerinde final kabulü yapılmadı. +- Dispatcher, emergency-stop Arduino contract/ACK, zero-side-effect ret matrisi ve donanım kabulü yapılmadı. + +Bu nedenle entegrasyon kod fazı tamamlandı; fiziksel eylem rollout'u **release-blocked** durumunda. `action_proposals.enabled` kapalı kalmalı. Ayrıntılı sıra ve açma koşulları `.sentrybot/laya_integration_todo.md` başındadır. + diff --git a/.sentrybot/laya_integration_todo.md b/.sentrybot/laya_integration_todo.md new file mode 100644 index 00000000..b8ec5b36 --- /dev/null +++ b/.sentrybot/laya_integration_todo.md @@ -0,0 +1,976 @@ +# Laya System 1 Entegrasyonu — YAPILACAK İŞLER (İmplementasyon Kılavuzu) + +> **Amaç:** Bu dosya, token/bağlam bittiğinde başka bir AI modelinin devam edebilmesi için yazılmıştır. +> **Önkoşul:** Önce `laya_integration_completed.md` dosyasını oku — ne yapıldığını anla. +> **Kural:** Silme yapma, mevcut kodu boz**ma**. Additive (eklemeli) çalış. +> **Config:** Hardcode yasak, her şey `config/agent.yaml`'dan okunmalı. +> **Test:** Her faz sonunda `pytest tests -x -q` çalıştır; son doğrulanmış toplam 970 testtir (yeni test eklenirse sayı değişir). + +> **Güncel durum / karar (2026-09-27):** Güvenli, dispatch kapalı Laya System 1 entegrasyonu bu faz için tamamlandı. Üretimde `tri_layer.laya.enabled: true`, ancak `action_proposals.enabled: false`; Laya önerisi ToolRegistry yürütmesine yetki vermez. Son doğrulama: tam test paketi **970 passed**; değişen Python modüllerinin `py_compile` ve çalışma ağacı `git diff --check` temiz. Tamamlananlar ve kalan rollout kapıları aşağıda. Aşağıdaki eski tarihler, kararların deney günlüğüdür; güncel durum için bu blok ve `laya_integration_completed.md` önceliklidir. +> +> **Yapılanlar:** LayaEngine/Agent entegrasyonu, kısa doğrudan komut ile System 2 sohbet sınırı, affect/vision olayları ve güven eşikleri; typed `set_lights` inert proposal builder + exact schema validator; turn başına internal UUID bağlama; çevrimdışı evaluator, renk/intent/audio dev ve holdout ölçümleri; parser adaylarının reddi; testler ve güvenlik/rollout dokümanları. +> +> **Sıradaki işler (release sırası):** +> 1. Proje sahibi tarafından haricen gözden geçirilmiş, yeterli büyüklükte ve sınıfları dengeli intent/safety etiket seti sağla; mevcut holdout'lar tüketildi, tuning için tekrar kullanma. +> 2. Laya-origin ışık eylemi için Autonomy runtime owner capability approval/deny kapısı tasarla ve uygula. Mevcut `set_lights` uyumluluk handler'ı ActionArbiter/ActionSafetyFilter + ışık lease kullanıyor ama CapabilityExecutor'a girmiyor; bu gate olmadan dispatch yapılmayacak. +> 3. Gate ile birlikte simülasyon testlerini ekle: stale/wrong-request/low-confidence/unknown-args/disabled/capability-denied önerilerinde `ToolRegistry.execute`, handler ve hardware çağrısı sıfır; yalnız açıkça onaylı fixture eylemi mock owner yoluna gider. +> 4. Uyumlu gerçek Laya runtime ve hedef Raspberry Pi üzerinde target/direct/emergency/effect kalite ve gecikme kabulünü ölç. +> 5. Emergency stop'u SpinalCord + Arduino authoritative contract/ACK yolunda ayrı uygula ve test et. +> 6. Yukarıdakilerin tümü, cancellation/cooldown, arbiter, ACK/timeout ve arbitrary-output fail-closed testleri geçmeden `action_proposals.enabled` açılmayacak. +> +> **Rollout tamamlanma koşulu:** Haricen gözden geçirilmiş veri ölçütleri, Autonomy capability ownership, sıfır yan-etki ret testleri, acil-stop regresyonları ve Pi donanım kabulünün hepsi yeşil. O zamana kadar mevcut güvenli kapanış durumu korunur; feature açılmaması eksik kod değil bilinçli release gate'idir. +> +> **Devam notu (2026-09-25):** `laya_integration_completed.md` son bölümlerini oku. Faz 5.1–5.2 ve 6.1–6.4 kararları işlendi; Faz 7 regression kapsaması genişletildi. Faz 2.6/2.7 ve 3.8 için mimari değerlendirme eklendi. Faz 3.1 doğrudan tool dispatch'i güvenli argüman/actuator sözleşmesi olmadan uygulanmamalı. Agent adaptörünün kararı cache'lemesi, kısa sohbeti System 2'ye bırakması ve karar aşamasında tool çalıştırmaması için ek regression testleri mevcut. Tam test paketi hazır Python 3.12 runtime ile runtime site-packages yolunu `.venv`'den önce vererek çalıştırılabiliyor; son tam koşu 940 test geçti. +> +> **Devam (2026-09-25):** Fail-closed, yalnızca öneri üreten `set_lights` typed-choice builder eklendi (`modules/agent_core/services/laya_action_contract.py`). Feature `config/agent.yaml` içinde kapalı; direct dispatch bağlantısı yok. Sonraki adım, uyumlu runtime/Pi'de gerçek modelle ek `action_color`/`action_effect` sorularını doğrulamak. Başarılı olursa validator eşik ve model kalibrasyonu gözden geçirilir; sonrasında ancak ToolRegistry + arbiter + Autonomy capability/safety üzerinden dispatch değerlendirilir. Emergency motor-stop ayrı SpinalCord/Arduino işi olarak kalır. +> +> **Gerçek model güvenlik bulgusu (2026-09-25):** Python 3.11 ile cache model inference çalıştı. `Işıkları kapat` eski predicate'e göre false emergency olurdu (emergency_stop confidence 0.8273, direct 0.8297, urgency 2.059/confidence 0.541); sıradan sohbet urgency 2.4434/confidence 0.2844 üretti. Bu nedenle emergency artık yüksek confidence'lı direct emergency target veya confidence'lı urgency gerektiriyor; normal high-urgency yan etkileri de urgency confidence threshold'una bağlı. Açık acil stop örnekleri target 0.934/direct 0.982 ve target 0.950/direct 0.970 ile geçiyor. Renk çıkarımı `mavi` isteğinde `pink`/0.289 güven verdiği için action proposal kapalı kalmalı. Test/doküman güncellendi. +> +> **Action calibration (2026-09-25):** `tools/evaluate_laya_action_proposals.py` 14 sabit Türkçe örneği çalıştırır ve hiçbir donanım/action API çağırmaz. Gerçek model: target top-choice 0.643; confidence>=0.60 coverage 0.571 ve kabul edilen hedef doğruluğu 0.625. Direct top-choice 0.929; coverage 0.357 ve kabul edilen direct doğruluğu 1.0. Explicit color 1/8 (0.125), explicit effect 2/3, confidence-gated emergency 14/14, öneri 0. Raporlama metriğindeki unspecified renk yanlış sayım düzeltildi. Renk/target/direct coverage öneri üretmeye yetersiz; feature `enabled: false` kalmalı. Daha geniş holdout + soru varyantı kalibrasyonu direct dispatcher'dan önce. +> +> **Soru varyantı deneyi (2026-09-25):** Aynı batch içinde mevcut, açık Türkçe ve açık İngilizce/bilingual color prompt'ları karşılaştırıldı. İki alternatif de açık renk isteklerinde 8/8 doğru, confidence>=0.85 coverage 7/8 verdi; fakat altı no-color kontrolünde doğru unspecified oranı sırasıyla 1/6 ve 2/6, her varyantta iki high-confidence yanlış renk vardı. Mevcut prompt explicit renk 1/8 ama no-color specificity 5/6 ve high-confidence false positive 0. Hiçbir varyant production'a alınmadı; `action_proposals.enabled: false` kalıyor. +> +> **Birleşik seçim deneyi (2026-09-25):** Tek finite-choice sorusunda renk, ışığı kapatma ve ışık komutu olmama seçenekleri birleştirildi. Aynı 14 örnekte genel doğruluk 0.500; açık renk doğruluğu 0.750, confidence>=0.85 coverage 0.375 ve kabul edilen renk doğruluğu 1.000 oldu. Altı no-color kontrolünde specificity 0.167 ve 3 high-confidence yanlış seçim görüldü. Bu soru da güvenli değil; production soruları ve kapalı proposal ayarı değişmedi. Sonraki calibration işi, aynı örneklerin varyantlarını optimize etmek yerine daha geniş, önceden etiketlenmiş ve ayrık holdout Türkçe kümesi oluşturmaktır. +> +> **Genişletilmiş calibration kümesi (2026-09-25):** Evaluator 32 elle etiketlenmiş Türkçe örneğe genişletildi: 18 explicit color/off ve 14 no-explicit-color kontrolü. Current question explicit doğruluk 0.111 / specificity 0.857 / high-confidence FP 0; Türkçe varyant 1.000 / coverage 0.778 / specificity 0.214 / FP 4; İngilizce-bilingual 0.944 / coverage 0.889 / specificity 0.286 / FP 5. Birleşik soru overall 0.625, explicit-color 0.778, coverage 0.222, specificity 0.429, FP 4. Emergency 32/32 ve proposal 0. Kalibrasyon kümesi bağımsız holdout değildir; hiçbir varyant production'a alınmadı. Sıradaki iş: ayrı, önceden tanımlanmış holdout set oluşturmak ve rule-based explicit command baseline'ını da aynı metriklerle ölçmek. +> +> **Ayrı final holdout + parser baseline (2026-09-25):** Kalibrasyon ve parser geliştirme cümlelerinden ayrık 30 örnekli final set (14 explicit color/off, 16 control) sabitlendi. Deterministik parser 0.967 overall / 0.929 explicit doğruluk / 1.000 control specificity verdi. Laya current/TR-explicit/EN-explicit/joint overall sırasıyla 0.433/0.533/0.567/0.500; explicit doğruluk 0.143/1.000/1.000/0.786, specificity 0.688/0.125/0.188/0.250. TR/EN varyantları 8/9 yüksek güvenli false positive üretti. Parser yalnız evaluator içinde ve holdout'a bakılarak değiştirilmeyecek; yeni ayar yapılacaksa yeni final split gerekir. Action proposal kapalı kalır. Sıradaki iş: parser'ı daha geniş, insan tarafından gözden geçirilmiş yeni bağımsız veriyle test etmek; ardından typed proposal'ın target/direct/effect kapılarını ve simüle dispatch safety yolunu ayrı değerlendirmek. +> +> **Intent/safety final holdout (2026-09-25):** Renk setlerinden ayrık 30 örnek, altı target sınıfının her birinden beş örnek. Target doğruluğu 0.600 (neopixel 0.400, arduino_serial 1.000, emergency_stop 0.800, speak 0.200, camera 1.000, system2_chat 0.200); confidence>=0.60 coverage 0.700, kabul edilen target doğruluğu 0.714. Direct doğruluk 0.900; coverage 0.700, kabul edilen doğruluk 1.000. Emergency doğruluğu 0.933, FP 0, FN 2; high urgency doğruluğu 0.833, FP 0, FN 5. Effect doğruluğu 2/4 ve confidence>=0.85 kabulü 0/4. Bu holdout'a göre ayar yapılmadı. Sıradaki geliştirme işi yeni dev set'te `speak/system2_chat` ve acil durum recall'ını, ayrıca effect confidence'ını iyileştirip daha sonra yeni untouched holdout hazırlamak; dispatcher hâlâ kapalı. +> +> **Target/effect prompt calibration (2026-09-25):** 32 örnekli yalnız dev setinde uzun TR target varyantı accuracy 0.188 / accepted accuracy 0.174 / coverage 0.719; kısa EN target varyantı 0.500 / accepted accuracy 0.667 / coverage 0.188 oldu. Explicit effect varyantı beş etiketli örnekte 0.800 top-choice ama 0.400 coverage ve 0.500 accepted accuracy verdi. Hiçbiri yeterli değil; final intent holdout'a uygulanmadı, production promptları değişmedi. +> +> **Yeni dengeli intent dev + tek sefer holdout karşılaştırması (2026-09-26):** Altı target sınıfından beşer örnekli yeni dev setinde current target 0.467 accuracy/0.667 accepted accuracy/0.500 coverage; Türkçe explicit 0.500/0.857/0.467; English explicit 0.600/0.786/0.467. Türkçe candidate emergency FN'yi dev'de 3'ten 2'ye indirdi; seçilip holdout'ta tek kez denendi fakat target accuracy 0.600'den 0.433'e, accepted accuracy 0.714'ten 0.647'ye, emergency doğruluğu 0.933'ten 0.900'e geriledi (FN 2→3). Candidate reddedildi; production değişmedi. Holdout artık tüketildi; yeni prompt adayı için yeni, untouched test set gereklidir. Direct ve effect/urgency kapıları da aşağıdaki metrikler nedeniyle açık. +> +> **Explicit audio refiner deneyi (2026-09-26):** Ayrı ikili ses üretme komutu sorusu dev setinde 0.733 doğruluk verdi ancak precision=0.0, recall=0.0 (FP 3, 5 speak isteğinin tamamı kaçtı). Confidence 0.60/0.80/0.90 kapılarında speak/system2_chat refinement genel accuracy'yi değiştirmedi (0.467). Candidate reddedildi; production yolu ve daha önce tüketilmiş holdout değiştirilmedi. +> +> **Çıktı araştırması (2026-09-25):** Kurulu Laya 0.3.5 `system_one()` kaynak kodu yalnız `choice`, `score`, `noul` türlerini ve typed confidence/probability cevaplarını destekliyor; serbest JSON action argümanı yok. Sonraki güvenli prototip yalnız bounded choice alanlarını birleştirebilir, sürekli actuator değeri üretemez. Cache snapshot ile gerçek inference denemesi `.venv` Torch `torch_python.dll` yüklenemediği için WinError 126 ile durdu. CP312 Torch wheel kurulumu onaylandı ancak 124.1 MB wheel indirmesi 7.6 MB'ta yaklaşık 73 kB/s hızla takılıp tekrarlı bağlantı kesintisi verdi; paket kurulmadan deneme durduruldu. Gerçek model doğrulaması uyumlu runtime/Pi üzerinde yapılmalı; bu doğrulanmadan dispatch açılmamalı. + +--- + +## GENEL MİMARİ KURAL + +``` +LayaEngine.get_instance(config) → TEK singleton instance +Tüm modüller AYNI instance'ı kullanır. Yeni instance OLUŞTURMA. +``` + +--- + +## FAZ 1: Laya Engine Çekirdek Genişletme + Config + +### 1.1 — config/agent.yaml'a Laya section ekle + +**Dosya:** `config/agent.yaml` +**Konum:** `tri_layer:` bölümünün içine, `router:` ile aynı seviyede + +```yaml +tri_layer: + enabled: false + api_native_tools: true + laya: # ← BU BLOĞU EKLE + enabled: true + confidence_threshold: 0.60 + model_path: null + fast_path: + enabled: true + emergency_bypass_safety: true + filler: + enabled: true + cooldown_s: 12.0 + mood_aware: true + affective: + mood_impact_scale: 1.0 + vision_events: true + face_emotion_events: true + telemetry: + log_decisions: true + histogram_enabled: false + urgency: + emergency_threshold: 2.0 + high_threshold: 1.5 + sensor_boost_hz: 5.0 + fast_path: + # ... mevcut fast_path aynen kalır +``` + +### 1.2 — LayaEngine.decide_with_context() ekle + +**Dosya:** `modules/agent_core/services/laya_engine.py` +**Konum:** `decide()` metodundan sonra (satır 285 civarı) +**Ne yapacak:** WorldState + VLM context'i de alarak zenginleştirilmiş karar üretir. + +```python +def decide_with_context( + self, + text: str, + world_state: Optional[Dict[str, Any]] = None, + visual_context: Optional[str] = None, +) -> Optional[LayaDecision]: + """Context-enriched System 1 decision. + + Eğer visual_context varsa, text'e eklenerek Laya'ya gönderilir. + WorldState bilgisi (battery, mood, follow_active) kararı etkileyebilir. + """ + enriched = str(text or "").strip() + if visual_context: + enriched = f"{enriched} [Sahne: {visual_context[:100]}]" + + decision = self.decide(enriched) + if decision is None: + return None + + # WorldState override'ları + if world_state: + battery = world_state.get("battery_percent", 100) + if battery < 15 and decision.urgency_score < 1.5: + decision.urgency_score = 1.5 # Düşük batarya → urgency artır + + return decision +``` + +### 1.3 — LayaEngine telemetri ekle + +**Dosya:** `modules/agent_core/services/laya_engine.py` +**Konum:** `__init__` içine (satır 132 civarı) + +```python +# __init__ içine ekle: +self._decision_count = 0 +self._total_inference_ms = 0.0 +self._last_decisions: List[Dict[str, Any]] = [] # son 20 karar logu +self._max_decision_log = 20 + +# decide() metodunun sonuna (return'den önce) ekle: +self._decision_count += 1 +self._total_inference_ms += elapsed_ms +self._last_decisions.append({ + "text": cleaned[:50], + "target": target_mod, + "conf": target_conf, + "urgency": urgency_score, + "ms": elapsed_ms, +}) +if len(self._last_decisions) > self._max_decision_log: + self._last_decisions = self._last_decisions[-self._max_decision_log:] +``` + +Yeni metod ekle: +```python +def get_telemetry(self) -> Dict[str, Any]: + """Returns Laya performance telemetry.""" + avg_ms = (self._total_inference_ms / self._decision_count) if self._decision_count > 0 else 0.0 + return { + "available": self._is_available, + "enabled": self.enabled, + "decision_count": self._decision_count, + "avg_inference_ms": round(avg_ms, 2), + "load_time_ms": self._load_time_ms, + "last_decisions": list(self._last_decisions[-5:]), + } +``` + +### 1.4 — LayaEngine filler mood-aware yap + +**Dosya:** `modules/agent_core/services/laya_engine.py` +**Konum:** `get_instant_filler()` metodunu genişlet + +```python +def get_filler_with_affect(self, language: str = "tr", mood: str = "neutral") -> str: + """Mood'a göre uyarlanmış thinking filler.""" + if mood in ("joy", "curiosity"): + fillers_tr = ["Hemen bakıyorum!", "Ooo, ilginç bir soru!", "Bakalım ne bulacağım..."] + fillers_en = ["Oh, interesting!", "Let me check!", "Curious about that..."] + elif mood in ("sadness", "tired"): + fillers_tr = ["Hmm, bakayım...", "Bir kontrol edeyim...", "Düşünüyorum..."] + fillers_en = ["Let me see...", "Thinking...", "Hmm..."] + elif mood in ("anger", "furious"): + fillers_tr = ["Tamam, hemen bakıyorum.", "Bir dakika.", "İnceliyorum."] + fillers_en = ["Fine, checking.", "One moment.", "Looking into it."] + else: + return self.get_instant_filler(language) + + lang = str(language or "tr").lower() + if lang.startswith("en"): + return random.choice(fillers_en) + return random.choice(fillers_tr) +``` + +--- + +## FAZ 2: AutonomyBrain Entegrasyonu + +### 2.1 — BrainInitMixin'e LayaEngine ekle + +**Dosya:** `modules/autonomy/services/brain_init.py` +**Konum:** `_init_components()` metodunun içi, `self.mood = MoodManager(config)` satırından sonra (satır 56 civarı) + +```python +# Import (dosya başına): +from modules.agent_core.services.laya_engine import LayaEngine + +# _init_components() içinde, self.mood satırından sonra: +self.laya_engine = LayaEngine.get_instance(config) +``` + +### 2.2 — AutonomyBrain._think() Laya-aware yap + +**Dosya:** `modules/autonomy/services/brain.py` +**Konum:** `_think()` metodu var mı kontrol et. Yoksa `_loop()` içindeki `self._think()` çağrısını bul. + +`_think()` metodu brain_parts/ mixin'lerinden geliyor olabilir. Eğer `DecisionMixin` içindeyse orada değişiklik yap. Aramak için: +``` +grep -rn "_think" modules/autonomy/services/brain_parts/ +``` + +Yapılacak değişiklik: `_think()` içinde (veya BrainDecisionMixin'de): +```python +# Mevcut _think() veya karar döngüsü içine ekle: +if hasattr(self, 'laya_engine') and self.laya_engine.is_available: + # Son konuşma metnini al + last_speech = self.state.get("last_speech_text", "") + if last_speech: + laya_dec = self.laya_engine.decide(last_speech) + if laya_dec: + # 1. Emergency handling + if laya_dec.is_emergency: + self.handle_hardware_event("emergency_voice", {"source": "laya"}) + + # 2. Mood update + if laya_dec.affective_event != "neutral": + self.mood.apply_affective_event(laya_dec.affective_event) + + # 3. Curiosity boost from vision + if laya_dec.target_module == "camera": + self.mood.modify("curiosity", 15) +``` + +### 2.3 — System1ReflexEngine → Deprecation Proxy + +**Dosya:** `modules/autonomy/services/system1_reflex.py` +**Strateji:** Sınıfı **silme**, mevcut API'yi koru ama implementasyonu LayaEngine'e yönlendir. + +```python +"""System 1 Reflex Engine — DEPRECATED: LayaEngine proxy. + +Bu sınıf geriye dönük uyumluluk için korunmaktadır. +Tüm işlevsellik artık modules.agent_core.services.laya_engine.LayaEngine tarafından sağlanır. +""" +from __future__ import annotations +import logging +from typing import Any, Dict + +logger = logging.getLogger("autonomy.system1_reflex") + +# Eski sabitler — backward compat +THINKING_FILLERS_TR = [ + "Hmm...", "Hemen bakıyorum...", "Bir düşüneyim...", "Hımm, anladım...", "Bakalım...", +] +THINKING_FILLERS_EN = [ + "Hmm...", "Let me check...", "Thinking...", "One second...", +] + +class System1ReflexEngine: + """DEPRECATED — LayaEngine'e proxy. Eski API'yi korur.""" + + def __init__(self, language: str = "tr") -> None: + self.language = language + self._laya = None + self._last_filler_ts = 0.0 + self._filler_cooldown_s = 15.0 + logger.info("System1ReflexEngine initialized as LayaEngine proxy") + + def _get_laya(self): + if self._laya is None: + try: + from modules.agent_core.services.laya_engine import LayaEngine + self._laya = LayaEngine.get_instance() + except Exception: + pass + return self._laya + + def get_thinking_body_pose(self) -> Dict[str, Any]: + laya = self._get_laya() + if laya: + reaction = laya.get_affective_reaction("neutral") + return { + "head": {"pan": 86, "tilt": 102}, + "ears": {"left": 80, "right": 70}, + "eyes": {"mode": "animation", "name": "thinking"}, + "leds": {"mode": "thinking", "color": reaction.get("led_color", "#30e3ca")}, + } + # Orijinal fallback + import random + return { + "head": {"pan": random.choice([86, 94]), "tilt": 102}, + "ears": {"left": 80, "right": 70}, + "eyes": {"mode": "animation", "name": "thinking"}, + "leds": {"mode": "thinking", "color": "#30e3ca"}, + } + + def should_emit_verbal_filler(self, wait_elapsed_s: float) -> bool: + import time + if wait_elapsed_s < 1.2: + return False + now = time.time() + if now - self._last_filler_ts < self._filler_cooldown_s: + return False + self._last_filler_ts = now + return True + + def get_verbal_filler(self) -> str: + laya = self._get_laya() + if laya: + return laya.get_instant_filler(self.language) + import random + if self.language.startswith("en"): + return random.choice(THINKING_FILLERS_EN) + return random.choice(THINKING_FILLERS_TR) +``` + +### 2.4 — SpinalCordReflex — Sesli acil dur desteği + +**Dosya:** `modules/autonomy/services/spinal_cord_reflex.py` +**Konum:** `observe_hardware_event()` metodundan sonra yeni metod ekle + +```python +def observe_voice_emergency(self, laya_decision) -> bool: + """Laya emergency kararını spinal cord refleksine bağlar. + + Args: + laya_decision: LayaDecision instance (laya_engine.py'den) + Returns: + True if emergency reflex triggered + """ + if not self.enabled: + return False + if not hasattr(laya_decision, 'is_emergency') or not laya_decision.is_emergency: + return False + + import time + now = time.time() + if now - self._last_reflex_ts < self.reflex_cooldown_s: + return False + + logger.warning("SPINAL CORD REFLEX: Voice emergency detected via Laya! target=%s urgency=%.1f", + laya_decision.target_module, laya_decision.urgency_score) + + self.client.push_interaction_event("motor.stop", {"priority": "emergency", "source": "laya_voice"}) + self._last_reflex_ts = now + + try: + self.memory.add_event( + f"Voice emergency detected: urgency={laya_decision.urgency_score:.1f}. " + "Motor stopped via spinal cord reflex." + ) + self.client.update_emotions(["surprise"]) + self.client.push_interaction_event("appraisal:shocked") + except Exception as exc: + logger.error("Failed to notify higher brain of voice emergency: %s", exc) + + return True +``` + +### 2.5 — MoodManager — Tam LayaDecision desteği + +**Dosya:** `modules/autonomy/services/mood.py` +**Konum:** `apply_affective_event()` metodundan sonra + +```python +def apply_laya_decision(self, decision) -> str: + """Apply full Laya System 1 decision to mood state. + + Args: + decision: LayaDecision instance + Returns: + Dominant emotion string + """ + with self._lock: + # 1. Affective event + if hasattr(decision, 'affective_event'): + self.apply_affective_event(decision.affective_event) + + # 2. Urgency → fear/energy impact + if hasattr(decision, 'urgency_score'): + urg = float(decision.urgency_score) + if urg >= 2.0: + self.modify("fear", min(40, urg * 15)) + self.modify("energy", -10) + elif urg >= 1.0: + self.modify("curiosity", 10) + + # 3. Direct command satisfaction + if hasattr(decision, 'is_direct_command') and decision.is_direct_command: + self.satisfy_need("stimulation", 15) + + # 4. Social interaction + if hasattr(decision, 'target_module'): + if decision.target_module == "system2_chat": + self.satisfy_need("social", 20) + + return self.get_dominant_emotion() +``` + +### 2.6 — BehaviorComposer Laya entegrasyonu + +**Durum: Mevcut mood zinciriyle karşılandı (2026-09-25).** Laya affect MoodManager'ı günceller; BehaviorComposer konuşmayı `_speak_with_mood()`'a verir ve bu yol güncel dominant mood'dan tone üretir. Regression testi `test_behavior_composer_speech_uses_current_laya_updated_mood_tone` bu uçtan uca delegeyi doğrular. Composer'a ikinci Laya plan/mood seçicisi eklenmiyor; mevcut safety/plan owner korunuyor. + +**Dosya:** `modules/autonomy/services/behavior_composer.py` +**Strateji:** Dosyayı oku, `compose()` veya `select_behavior()` gibi ana metodu bul. +O metodun içinde Laya affective state'e göre behavior chain seçimi ekle. + +```python +# Önce dosyayı oku: +# grep -n "def compose\|def select_behavior\|def plan" modules/autonomy/services/behavior_composer.py + +# Ana metod içinde, mevcut behavior seçimi yapıldıktan sonra: +laya_engine = getattr(self, '_laya_engine', None) +if laya_engine is None: + try: + from modules.agent_core.services.laya_engine import LayaEngine + self._laya_engine = LayaEngine.get_instance() + laya_engine = self._laya_engine + except Exception: + pass + +# Laya decision varsa behavior'ı etkile: +# (Bu kodun tam konumu dosya yapısına bağlı) +``` + +### 2.7 — CompanionGoalSelector urgency entegrasyonu + +**Durum: Tamamlandı, TTL korumalı (2026-09-25).** Laya karar anında urgency, `is_urgent` sonucu ve timestamp kaydedilir. Companion needs snapshot'ına metadata olarak taşınır; YAML'deki `companion_goals.laya_urgency.freshness_s` süresi içinde yüksek urgency varsa snapshot güvenli `safety/pause_and_observe` adayına yükseltilir. Süresi geçmiş veya düşük urgency goal'u değiştirmez. GoalSelector mevcut GoalFormation/safety/capability katmanlarını kullanır; doğrudan donanım dispatch'i yoktur. Fresh ve expired durumlar için regresyon testleri eklendi. + +**Dosya:** `modules/autonomy/services/companion_goal_selector.py` +**Strateji:** Dosyayı oku, goal scoring/ranking metodunu bul. + +```python +# Goal scoring'de urgency'yi kullan: +# Eğer Laya urgency yüksekse, mevcut goal'u interrupt et +if hasattr(self, '_brain') and hasattr(self._brain, 'laya_engine'): + laya = self._brain.laya_engine + last_text = self._brain.state.get("last_speech_text", "") + if last_text: + dec = laya.decide(last_text) + if dec and dec.urgency_score >= 1.5: + # Mevcut goal'u ertele, acil goal'a geç + pass +``` + +--- + +## FAZ 3: Agent Core Orkestrasyon Derinleştirme + +### 3.1 — agent.py step() Fast-Path (LLM Atlama) + +**Durum: Sınıflandırma hızlı yolu mevcut; direct-tool taslağı güvenlik nedeniyle uygulanmıyor.** Agent Laya kararını fast-path ve kısa LLM yanıt ayarı için kullanıyor. `suggested_tool` çağrı argümanlarını sağlamaz; Laya'nın `emergency_stop -> stop_follow` eşlemesi de gerçek `motor.stop` garantisi değildir. Acil sesli durdurma SpinalCord yolunda çalışır; Agent tool registry'ye sınıflandırma kararından doğrudan dispatch eklenmeyecek. + +**Dosya:** `modules/agent_core/services/agent.py` +**Konum:** `step()` metodu, satır 149'daki `use_fast_path` kontrolünden sonra + +**ÖNEMLİ:** Mevcut `_should_fast_path()` metodu `agent_context.py` veya `agent_turn.py`'da olabilir. Aramak için: +``` +grep -rn "_should_fast_path" modules/agent_core/services/ +``` + +Yapılacak: Laya `should_fast_path()` sonucunu mevcut fast_path kararına entegre et. + +```python +# step() içinde, mevcut use_fast_path satırından ÖNCE: +laya_fast = False +laya_fast_decision = None +laya_eng = getattr(self, "laya_engine", None) +if laya_eng and getattr(laya_eng, "enabled", True): + laya_fast, laya_fast_decision = laya_eng.should_fast_path(user_prompt) + +# Mevcut use_fast_path kararını Laya ile birleştir: +use_fast_path = self._should_fast_path(user_prompt, native_tools=native_tools) or laya_fast + +# Eğer Laya fast-path ise ve emergency ise: +if laya_fast and laya_fast_decision and laya_fast_decision.is_emergency: + # LLM'yi tamamen atla, direkt tool çalıştır + tool_name = laya_fast_decision.suggested_tool or "stop_follow" + try: + result = self.tool_registry.execute(tool_name, {}) + ack = laya_eng.get_fast_ack(session_language) + self.speech_arbiter.enqueue( + ack, + priority=95, # SAFETY priority + category="safety", + language=session_language, + ) + return { + "text": ack, + "actions": [{"tool": tool_name, "args": {}, "result": result}], + "route": [laya_fast_decision.target_module], + "fast_path": True, + "laya_decision": True, + "trace_id": trace_id, + } + except Exception as exc: + logger.warning("Laya fast-path execution failed: %s", exc) + # Normal path'e düş +``` + +### 3.2 — tri_layer.py Dinamik Boost + +**Dosya:** `modules/agent_core/services/tri_layer.py` +**Konum:** `route()` metodu, satır 415-417 + +**Mevcut kod:** +```python +for mod in laya_modules: + if mod in self.profiles: + scores[mod] = scores.get(mod, 0.0) + 10.0 # Sabit boost +``` + +**Yeni kod:** +```python +for mod in laya_modules: + if mod in self.profiles: + # Laya confidence'a göre dinamik boost (5.0 — 15.0 arası) + laya_dec = getattr(self.laya_engine, '_last_decision', None) + if laya_dec and hasattr(laya_dec, 'module_confidence'): + boost = 5.0 + (laya_dec.module_confidence * 10.0) # 0.0→5.0, 1.0→15.0 + else: + boost = 10.0 + scores[mod] = scores.get(mod, 0.0) + boost +``` + +### 3.3 — agent_subagents.py Routing Birleştirme + +**Dosya:** `modules/agent_core/services/agent_subagents.py` +**Konum:** `_route_subagents()` metodu (satır 30-46) +**Sorun:** Bu metod keyword matching yapıyor ama `self.router` (TriLayerRouter) da aynı şeyi yapıyor → **duplicate logic**. + +**Yeni kod:** +```python +def _route_subagents(self, prompt: str) -> List[SubAgentProfile]: + """Route subagents via TriLayerRouter (unified routing).""" + # TriLayerRouter zaten keyword + Laya + semantic routing yapıyor + module_names = self.router.route(prompt) + self.last_routed_subagents = module_names + + selected: List[SubAgentProfile] = [] + for name in module_names: + profile = self.subagent_profiles.get(name) + if profile and getattr(profile, 'enabled', True): + selected.append(profile) + + # Fallback + if not selected and "agent_core" in self.subagent_profiles: + fallback = self.subagent_profiles["agent_core"] + if getattr(fallback, 'enabled', True): + selected.append(fallback) + + return selected +``` + +### 3.4 — SpeechArbiter Laya Filler Priority + +**Dosya:** `modules/agent_core/services/speech_arbiter.py` +**Konum:** `SpeechPriority` class'ından sonra (satır 49 civarı) + +```python +class SpeechPriority: + SAFETY = 95 + LAYA_FILLER = 75 # ← YENİ: Progress'ten yüksek, final'den yüksek + FINAL_RESPONSE = 60 + PROGRESS = 30 + IDLE = 15 +``` + +Yeni metod ekle (satır 216 civarı, `enqueue_safety`'den sonra): +```python +def enqueue_laya_filler( + self, + text: str, + language: str = "", + trace_id: str = "", +) -> Optional[str]: + """Enqueue an ultra-low-latency Laya thinking filler.""" + return self.enqueue( + text, + priority=SpeechPriority.LAYA_FILLER, + category="laya_filler", + language=language, + max_age_s=4.0, # Kısa ömür — yeni final gelince expire olur + trace_id=trace_id, + ) +``` + +### 3.5 — WorldState Laya State + +**Dosya:** `modules/agent_core/services/world_state.py` +**Strateji:** Önce dosyayı oku, `update_state()` veya `set()` metodunu bul. + +```python +# Yeni metod: +def update_laya_state(self, decision) -> None: + """Store latest Laya System 1 decision in world state.""" + if decision is None: + return + self.update_state({ + "laya_last_target": getattr(decision, 'target_module', ''), + "laya_last_urgency": float(getattr(decision, 'urgency_score', 0.0)), + "laya_last_affect": getattr(decision, 'affective_event', 'neutral'), + "laya_last_direct": bool(getattr(decision, 'is_direct_command', False)), + "laya_last_ms": float(getattr(decision, 'inference_ms', 0.0)), + }) +``` + +### 3.6 — SensorLoop Urgency Boost + +**Dosya:** `modules/agent_core/services/sensor_loop.py` +**Strateji:** Poll loop'u bul, urgency yüksekse interval'ı düşür. + +```python +# _poll_loop veya _run içinde: +# Mevcut sleep/interval satırını bul ve şu mantığı ekle: +laya_engine = None +try: + from modules.agent_core.services.laya_engine import LayaEngine + laya_engine = LayaEngine.get_instance() +except Exception: + pass + +# Poll interval'ını belirlerken: +base_interval = 1.0 / self.poll_hz +if laya_engine and laya_engine._is_available: + try: + telemetry = laya_engine.get_telemetry() + last = telemetry.get("last_decisions", []) + if last and last[-1].get("urgency", 0) >= 1.5: + base_interval = min(base_interval, 0.2) # 5Hz'e çık + except Exception: + pass +``` + +### 3.7 — ActionArbiter Fast-Path Bypass + +**Karar: Bypass eklenmiyor.** ActionArbiter quiet-hours, yetenek ve safety kontrollerinin sahibi. Acil motor durdurma bu action türlerinden değil spinal `motor.stop` reflex yolundan geçer. `move_head`/`stop_follow` için bu filtreleri atlamak acil durdurma eşdeğeri değildir ve safety katmanını gevşetirdi. + +**Dosya:** `modules/agent_core/services/action_arbiter.py` +**Strateji:** Dosyayı oku, `check()` veya `approve()` metodunu bul. + +```python +# Mevcut approval metoduna parametre ekle: +def approve(self, action_name, args, *, laya_emergency=False): + # Emergency ise bazı safety check'leri atla + if laya_emergency and action_name in {"stop_follow", "move_head"}: + return True # Acil dur komutları her zaman geçer + + # ... mevcut safety logic ... +``` + +### 3.8 — IdleBehavior Curiosity Trigger + +**Durum: Sahiplik sınırı regression testiyle doğrulandı (2026-09-25).** `IdleBehaviorSystem` yalnızca nötr heartbeat ışığı üretir; Laya telemetry sorgusu, Agent semantic step veya hareket/vision kararı çalıştırmaz. `test_idle_heartbeat_does_not_run_laya_or_semantic_actions` bu sınırı korur. Curiosity tabanlı semantik tarama Autonomy idle planner kapsamındadır. + +**Dosya:** `modules/agent_core/services/idle_behavior.py` +**Strateji:** Idle loop'u bul, curiosity yüksekse çevreye bakma tetikle. + +```python +# Idle döngüsünde: +laya_engine = getattr(self._agent, 'laya_engine', None) +if laya_engine: + telemetry = laya_engine.get_telemetry() + # Son kararlardan curiosity seviyesini çıkar + # Curiosity yüksekse → vision scan tetikle +``` + +--- + +## FAZ 4: VLM Bridge Entegrasyonu + +### 4.1 — VisionProcessor detection → Laya affect + +**Durum: Semantik etkiler mevcut owner katmanlarında karşılanıyor; Laya vision sınıflandırması yan etki üretmiyor.** `perception_context` kişi/sahip event'lerini ve yeni kişi appraisal'ını yönetir; `VisionContextNeedsBridge` kişi varlığı/yenilik sinyalini needs snapshot'a taşır; tehlike `visual_hazard` appraisal'ına gider. Gateway Laya handler bunları ek sınıflandırma ve state telemetrisi için alır. Aynı event için ikinci mood/need değişikliği eklenmedi; regression testi bunu korur. + +**Dosya:** `modules/vlm_bridge/services/processor.py` +**Strateji:** `on_detection()` veya detection callback'ini bul. + +```python +# Yeni kişi tespit edildiğinde: +try: + from modules.agent_core.services.laya_engine import LayaEngine + laya = LayaEngine.get_instance() + if laya.is_available: + # Kişi tespiti → social need satisfy + # Bu kısım processor'ın yapısına bağlı + pass +except Exception: + pass +``` + +### 4.2 — VisionEventBus Laya subscriber + +**Durum: Tamamlandı.** Gateway bridge event bus'a abone olup vision event'lerini worker üzerinden Autonomy Laya handler'ına aktarır; güven eşikleri/cooldown uygulanır. + +**Dosya:** `modules/vlm_bridge/services/vision_event_bus.py` +**Strateji:** `subscribe()` veya `register_handler()` metodunu bul. + +```python +# Laya subscriber ekle: +def _laya_vision_handler(event_type: str, payload: dict): + """Forward vision events to Laya affective pipeline.""" + try: + from modules.agent_core.services.laya_engine import LayaEngine + laya = LayaEngine.get_instance() + if not laya.is_available: + return + + if event_type == "person_detected": + # Kişi → social event + pass + elif event_type == "owner_detected": + # Sahip → user_praise benzeri + pass + elif event_type == "hazard_detected": + # Tehlike → urgency artır + pass + except Exception: + pass +``` + +### 4.3 — VisualContext Laya kısa özet + +**Durum: Tamamlandı.** VisualContext kısa sahne özetini sunar; Agent ve event yolları bu özeti Laya kararına bağlar. + +**Dosya:** `modules/vlm_bridge/services/visual_context.py` + +```python +def get_laya_context(self) -> str: + """Returns a brief scene summary for Laya System 1 input (max 100 chars).""" + # Mevcut context'ten kısa bir özet oluştur + ctx = self.get_context() # veya benzeri mevcut metod + if not ctx: + return "" + # Kısa özet: "2 kişi, masa, laptop görünüyor" + objects = ctx.get("objects", [])[:5] + people_count = ctx.get("people_count", 0) + parts = [] + if people_count: + parts.append(f"{people_count} kişi") + for obj in objects: + parts.append(str(obj.get("label", ""))) + return ", ".join(parts)[:100] +``` + +### 4.4 — FaceEmotion → Laya affect + +**Durum: Tamamlandı.** Confidence/cooldown korumalı yüz event'i Laya tarafından sınıflandırılır; yalnızca yüksek güvenli praise/rude sonucu MoodManager'ı değiştirir. + +**Dosya:** `modules/vlm_bridge/services/face_emotion.py` +**Strateji:** Yüz duygusu tespit eden metodu bul, Laya'ya event gönder. + +```python +# Yüz duygusu tespit edildiğinde: +# Eğer kullanıcının yüzü "angry" ise → robot bunu hissetmeli +detected_emotion = ... # mevcut tespit sonucu +if detected_emotion in {"angry", "disgusted"}: + # MoodManager'a bildir + try: + # Bu event AutonomyBrain üzerinden mood'a ulaşmalı + pass + except Exception: + pass +``` + +--- + +## FAZ 5: Ses Pipeline + +### 5.1 — AgentStreaming mood-aware tone + +**Durum: Tamamlandı (2026-09-24).** `AutonomyBrain` yerel `MoodManager` referansını `AgentOrchestrator`'a verir. Her `step()` ve `step_event()` başında tek bir tone snapshot'ı alınır; streaming cümleleri `SpeechArbiter.enqueue_final_chunk()` üzerinden bu tonu TTS'e taşır. Mood yöneticisi bulunamaz veya hata verirse `None` ile mevcut varsayılan ton korunur. Ağ çağrısı/cümle başına mood okuması eklenmedi. + +**Dosya:** `modules/agent_core/services/agent_streaming.py` +**Konum:** `_stream_turn_sentence_by_sentence()` metodu + +```python +# Gerçek uygulama AgentOrchestrator.step()/step_event() içinde: +# MoodManager.get_speech_tone() bir kez alınır ve enqueue_final_chunk(tone=...) +# ile bu turdaki her stream cümlesine aktarılır. +``` + +### 5.2 — xSpeakService Laya filler endpoint + +**Durum: Mevcut güvenli pipeline ile karşılandı; ayrı endpoint eklenmiyor (2026-09-24).** `AgentOrchestrator.step()` Laya filler metnini `ProgressManager.emit_ack()`'e verir; bu metin `SpeechArbiter.enqueue_laya_filler()` üzerinden TTL/öncelik kurallarıyla normal `speak_preferred` TTS sınırına ulaşır. `/speak/laya_filler` endpoint'i kuyruğu atlayıp konuşma önceliği, cooldown ve barge-in koordinasyonunu bozardı; bu nedenle doğrudan endpoint taslağı uygulanmadı. Akış regression testi `test_progress_routes_laya_ack_through_priority_queue` ile korunuyor. + +**Dosya:** `modules/voice/speak/xSpeakService.py` +**Karar:** FastAPI router'a yeni endpoint ekleme; agent progress/speech arbiter zinciri bu işi zaten yapar. + +```python +@router.post("/speak/laya_filler") +async def laya_filler(language: str = "tr"): + """Ultra-low-latency filler speech triggered by Laya System 1.""" + from modules.agent_core.services.laya_engine import LayaEngine + laya = LayaEngine.get_instance() + text = laya.get_instant_filler(language) + # Mevcut speak mekanizmasıyla seslendir + # ... + return {"text": text, "status": "ok"} +``` + +### 5.3 — AudioRouter urgency barge-in + +**Durum: Emergency barge-in mevcut ve entegrasyon testiyle doğrulandı (2026-09-25).** Sesli konuşma Brain `VocalMixin` yolunda Laya yalnızca zaten yüklüyse emergency sınıflandırması yapar; `BargeInController` bu urgency sinyaliyle tek kelimelik acil ifadede konuşmayı keser. AudioRouter'a ikinci bir urgency sahibi eklenmedi. `test_laya_emergency_reaches_brain_barge_in_controller` Laya → controller → stop zincirini doğrular. + +**Dosya:** `modules/voice/audio_router.py` +**Strateji:** Barge-in (konuşma kesme) mekanizmasını bul, urgency yüksekse otomatik kes. + +### 5.4 — BargeIn urgency threshold + +**Durum: Mevcut Laya emergency threshold'u kullanılıyor (2026-09-25).** Threshold LayaEngine config'indeki `emergency_threshold` üzerinden `is_emergency_decision()` ile belirlenir; bağımsız ve çakışan bir BargeIn threshold'u eklenmez. Cooldown ve wakeword kuralları korunur. + +**Dosya:** `modules/autonomy/services/barge_in.py` +**Strateji:** Threshold'u Laya urgency'ye göre dinamik yap. + +--- + +## FAZ 6: Tool Execution & Progress + +### 6.1 — ToolRegistry Laya hint + +**Durum: Tamamlandı (2026-09-24), teşhis amaçlı.** Agent, native LLM tool çağrısında aynı turdaki Laya `suggested_tool` değerini opsiyonel `laya_hint` olarak iletir. Registry bunu yalnızca başarılı tool status telemetrisine ve eşleşme boolean'ına ekler; tool seçimini/argümanlarını değiştirmez, yetki vermez ve yeni tool çalıştırmaz. Böylece Laya önerisi ile modelin seçimi gözlemlenebilir kalır; backward compat nedeniyle recovery callback'lerinin mevcut iki argümanlı çağrısı çalışmaya devam eder. + +**Dosya:** `modules/agent_core/services/tools/tool_registry.py`, `modules/agent_core/services/agent_turn.py` +**Güvenlik sınırı:** `laya_hint` yalnızca telemetri içindir; dispatch girdisi değildir. + +### 6.2 — ProgressManager TTFT metrik + +**Durum: Tamamlandı, metrik doğru katmana alındı (2026-09-24).** Laya çıkarımı Agent girişinde `latency_trace`'e ayrı `laya.inference` olayı olarak yazılır (inference_ms, target_module, urgency). LLM stream TTFT'si model yanıtının ilk token süresidir; Laya inference süresini bu sayıya eklemek farklı iki ölçümü karıştıracağından yapılmıyor. Event tabanlı Agent çağrılarında da aynı işaretleme kullanılır; Laya karar yoksa event yazılmaz. + +**Dosya:** `modules/agent_core/services/agent.py`, `modules/common/latency_trace.py` +**Karar:** Laya inference latency ayrı trace event'idir; model TTFT metriğine eklenmez. + +### 6.3 — ToolExecutionArbiter fast-path bypass + +**Karar: Uygulanmıyor; emergency refleksi bu arbiter'dan geçmiyor (2026-09-24).** ToolExecutionArbiter ortak VLM/head/light kaynaklarını çakışmadan kilitler. Bu kilidi Laya sınıflandırmasıyla bypass etmek aynı fiziksel/vision kaynağında eşzamanlı komutlara izin verir. Acil motor durdurma `SpinalCordReflex.observe_voice_emergency()` üzerinden doğrudan `motor.stop` olayına gider; normal tool arbiter'ını gevşetmez. Fast-path yalnızca LLM katmanını atlayabilir, kaynak/safety arbiter'larını değil. + +**Dosya:** `modules/agent_core/services/tool_execution_arbiter.py` + +### 6.4 — SafetyFilter emergency elevated + +**Karar: Generic SafetyFilter'a elevation eklenmiyor (2026-09-24).** Bu filtre agent action argümanlarını sınırlar/quiet-hours politikasını uygular; spinal cord emergency stop ayrı runtime sahibi altında. Genel emergency override eklemek quiet-hours veya hardware limit kontrollerini de yanlışlıkla aşma riski taşır ve mevcut reflex yoluna katkı sağlamaz. Laya emergency confidence/threshold kontrolü ve spinal stop yolu ayrı testlerle korunur. + +**Dosya:** `modules/agent_core/services/safety_filter.py` + +--- + +## FAZ 7: Test Suite + +### 7.1 — test_laya_engine.py genişlet + +**Durum: Tamamlandı (2026-09-25).** Context, telemetry ve mood-aware filler testleri zaten vardı; ek olarak emergency, güvenilir direct hardware ve normal conversation fast-path ayrımı test edildi. +Mevcut testlere ekle: +- `test_decide_with_context()` +- `test_get_telemetry()` +- `test_get_filler_with_affect()` + +### 7.2 — test_laya_fast_path.py oluştur + +**Durum: Mevcut Laya engine test dosyasında tamamlandı.** Aynı davranış için ayrı dosya açmak yerine `test_laya_engine.py` içinde üç path koşulu kapsandı. “Fast-path” burada LLM karar yolunu seçer; fiziksel tool dispatch yetkisi vermez. +```python +def test_emergency_fast_path(): + """Emergency command → LLM bypass, direkt tool execution.""" + +def test_direct_command_fast_path(): + """Direct hardware command → fast-path execution.""" + +def test_conversation_no_fast_path(): + """Normal sohbet → LLM path (fast-path olmamalı).""" +``` + +### 7.3 — test_laya_brain_integration.py oluştur + +**Durum: Tamamlandı (2026-09-25).** Yeni test dosyası emergency kararın SpinalCordReflex üzerinden `motor.stop` üretmesini ve normal sohbet kararının motoru durdurmamasını doğrular. BrainInit'in Laya singleton attachment'ı `brain_init.py` içinde mevcut. +```python +def test_brain_has_laya_engine(): + """AutonomyBrain laya_engine attribute'una sahip olmalı.""" + +def test_laya_emergency_triggers_spinal_cord(): + """Laya emergency → SpinalCord reflex tetiklenmeli.""" +``` + +### 7.4 — test_laya_vlm_bridge.py oluştur + +**Durum: Mevcut `tests/modules/vlm_bridge/test_laya_vision_events.py` ve `tests/modules/autonomy/test_laya_vision_event.py` ile tamamlandı.** Confidence/cooldown event yayını ve autonomy Laya/mood subscriber davranışı kapsanıyor; yinelenen test dosyaları eklenmedi. +```python +def test_vision_event_laya_subscriber(): + """Vision event → Laya affective pipeline çalışmalı.""" +``` + +### 7.5 — test_laya_speech_flow.py oluştur + +**Durum: Mevcut speech boundary testlerine genişletilerek tamamlandı.** Progress ACK'nin Laya filler kuyruğuna girdiği ve filler'ın final yanıt öncesinde seçildiği doğrulanıyor. +```python +def test_filler_before_llm_response(): + """Laya filler, LLM yanıtından önce seslenmeli.""" +``` + +--- + +## HIZLI REFERANS: Önemli Dosya Konumları + +``` +Proje kökü: c:\Users\emohi\Desktop\Project SentryBOT V5\ + +LAYA ÇEKIRDEĞI: + modules/agent_core/services/laya_engine.py ← LayaEngine + LayaDecision + modules/agent_core/services/tri_layer.py ← TriLayerRouter + SubAgentProfile + +ORKESTRASYON: + modules/agent_core/services/agent.py ← AgentOrchestrator.step() + modules/agent_core/services/agent_context.py ← AgentContextMixin (init) + modules/agent_core/services/agent_turn.py ← AgentTurnMixin (LLM loop) + modules/agent_core/services/agent_streaming.py ← Clause streaming + modules/agent_core/services/agent_subagents.py ← Sub-agent delegation + +BEYİN: + modules/autonomy/services/brain.py ← AutonomyBrain (sense/think loop) + modules/autonomy/services/brain_init.py ← Component initialization + modules/autonomy/services/mood.py ← MoodManager + modules/autonomy/services/system1_reflex.py ← DEPRECATED → LayaEngine proxy + modules/autonomy/services/spinal_cord_reflex.py ← Emergency hardware reflex + modules/autonomy/services/behavior_composer.py ← Behavior chain selection + +SES: + modules/agent_core/services/speech_arbiter.py ← Priority speech queue + modules/voice/speak/xSpeakService.py ← TTS service + modules/voice/audio_router.py ← Audio capture routing + +GÖRÜNTÜ: + modules/vlm_bridge/services/processor.py ← Vision main processor + modules/vlm_bridge/services/vision_event_bus.py ← Vision event pub/sub + modules/vlm_bridge/services/visual_context.py ← Scene context + modules/vlm_bridge/services/face_emotion.py ← Face emotion detection + +CONFIG: + config/agent.yaml ← Ana yapılandırma + +TESTLER: + tests/modules/agent_core/test_laya_engine.py ← Mevcut Laya testleri +``` + +--- + +## KRİTİK KURALLAR (HER FAZDA UYULMALI) + +1. **Singleton:** `LayaEngine.get_instance(config)` — yeni instance oluşturma +2. **Backward compat:** Mevcut API imzalarını değiştirme, yeni parametre eklerken `default` ver +3. **Thread safety:** `with self._lock:` kullan (MoodManager, SpeechArbiter zaten RLock kullanıyor) +4. **Config:** Her yeni ayar `config/agent.yaml`'a eklenmeli, hardcode yasak +5. **Test:** Her faz sonunda `pytest tests/ -x -q` çalıştır +6. **Silme yok:** Mevcut kodu silme, deprecate et veya proxy yap +7. **Import:** Lazy import kullan (`try: ... except ImportError:`) — modül bağımsızlığı +8. **Logging:** `logger = logging.getLogger("modül_adı")` ile logla +9. **Error handling:** Laya hataları asla ana akışı bozmamalı — `try/except` ile sar +> **2026-09-27 — ses komutu parser ölçümü:** Evaluator-only deterministik literal parser, 30 örnekli intent dev setinde 5/5 komutu yakaladı ve 0 FP verdi. Yeni, ayrık 30 örnekli audio holdout tek sefer ölçüldü: TP 8, FP 4, FN 0, TN 18; accuracy 0.867, precision 0.667, recall 1.000. Konuşma/soru bağlamındaki command-like fiiller yanlış pozitif ürettiği için parser intent/dispatch kapısı olarak reddedildi. Holdout tüketildi; bu veride tuning yapılmamalı. CLI modu Laya/Torch yüklemeden çalışır ve hiçbir tool/hardware çağrısı yapmaz. Production davranışı ve `action_proposals.enabled: false` değişmedi. Sıradaki iş: bağımsız, daha geniş ve haricen gözden geçirilmiş intent/safety korpusu hazırlamak; target/direct/emergency/effect eşiklerini orada doğrulamak. Ardından typed proposal validator ve sıfır çağrı garantili simüle dispatcher; en son cihaz üstü kabul testi. + +> **2026-09-27 — saf request-bound validator:** `laya_action_contract.py` içine request ID bağlama ve exact-schema validator eklendi. Validator current request eşleşmesini, yalnız `set_lights` allowlist'ini, exact schema/argument key'lerini, enum değerlerini ve config confidence eşiğini kontrol edip bool döndürüyor; malformed/extra/stale/low-confidence öneriyi reddediyor. Regression testleri eklendi. Bu validator ToolRegistry/dispatcher'a bağlanmadı; proposal config kapalı. Bir sonraki kod adımı dispatcher eklemek değil, mevcut request lifecycle'da stabil request ID üretimini izleyip bu sınırı test etmek; bundan sonra yalnız simülasyon dispatcher'ı sıfır çağrı garantisiyle ele almak. Haricen gözden geçirilmiş daha geniş intent/safety dataseti hâlâ açık kapı. + +> **Request lifecycle takibi:** API `trace_id` değeri çağıran taraftan gelebiliyor; bu nedenle güvenlik bağı olarak kullanılmadı. Agent, kabul edilmiş `step()` başına ayrı UUID üretiyor, Laya inert proposal'ına bağlıyor ve turn sonunda temizliyor. Regression testi bu ayrımı doğruluyor. Request binding tamamlandı; `step_event()` / native tool akışında Laya proposal yetkisi bulunmuyor, hiçbir dispatcher bağlantısı eklenmedi. Sıradaki kod kapısı: simülasyon-only validator/dispatcher test seam'i tasarlamak ve reddedilen girdilerde execute çağrısının sıfır kaldığını kanıtlamak; bunu ancak independent intent değerlendirme ve tool-owner safety kontrolleri kapalı değilken ele al. + +> **2026-09-27 — tool-owner safety incelemesi:** Mevcut `set_lights` compatibility tool'u `queue_action("lights", ...)` ile agent-core `ActionArbiter`'a gidiyor; `ActionSafetyFilter`'da lights-specific capability approval yok. Kayıtlı handler expression-light lease alıp Autonomy client'ını çağırıyor, ancak Autonomy `CapabilityExecutor`'ına girmiyor. ToolRegistry execution arbiter'ı ayrı bir concurrency sınırı ve runtime-owner capability kararı değil. Bu nedenle execute çağıran bir dispatcher veya onu taklit eden test seam'i eklenmedi; zorunlu owner safety gate'ini kanıtlayamazdı. Sıradaki gerçek iş, Autonomy tarafında Laya-origin lights isteği için açık capability approval/deny yolu kurup reddin handler/hardware çağrısını sıfır tuttuğunu test etmek; ondan sonra dispatcher tasarlanabilir. Intent/safety dataseti ayrıca dış gözden geçirme bekliyor. `action_proposals.enabled: false` sürüyor. diff --git a/config/agent.yaml b/config/agent.yaml index 4c13e41c..484e3a56 100644 --- a/config/agent.yaml +++ b/config/agent.yaml @@ -396,6 +396,44 @@ actions: tri_layer: enabled: false api_native_tools: true + laya: + enabled: true + confidence_threshold: 0.60 + model_path: null # null = auto-detect HF cache + device: cpu + fast_path: + enabled: true # Direct command → skip LLM + emergency_bypass_safety: false # Legacy setting; no tool/action safety gate is bypassed + action_proposals: + enabled: false # Proposal only; never authorizes tool dispatch + confidence_threshold: 0.85 + allowed_actions: + - set_lights + default_light_effect: BREATHE + filler: + enabled: true + cooldown_s: 12.0 + mood_aware: true # Mood'a göre filler seçimi + affective: + mood_impact_scale: 1.0 # Laya → MoodManager etki katsayısı + vision_events: true # VLM → Laya affective pipeline + face_emotion_events: true # Yüz duygusu → Laya + vision_event_min_confidence: 0.80 + vision_event_min_affect_confidence: 0.80 + vision_event_cooldown_s: 8.0 + telemetry: + log_decisions: true + histogram_enabled: false + max_decision_log: 20 + urgency: + emergency_threshold: 2.0 # Bu üstü → SpinalCord tetikle + high_threshold: 1.5 # Bu üstü → fast-path + high_confidence_threshold: 0.60 + emergency_target_confidence_threshold: 0.90 + emergency_direct_confidence_threshold: 0.80 + emergency_urgency_confidence_threshold: 0.80 + sensor_boost_hz: 5.0 # Urgency high → sensor poll rate + sensor_boost_hold_s: 8.0 # High urgency polling window after the decision fast_path: enabled: true max_chars: 180 diff --git a/modules/agent_core/services/agent.py b/modules/agent_core/services/agent.py index 9df0d403..6ba8415e 100644 --- a/modules/agent_core/services/agent.py +++ b/modules/agent_core/services/agent.py @@ -3,12 +3,14 @@ import logging import threading import time +import uuid from typing import Any, Callable, Dict, List, Optional from .agent_context import AgentContextMixin from .agent_turn import AgentTurnMixin from modules.common.latency_trace import latency_trace from modules.common.model_policy import get_model_policy # type: ignore +from .laya_action_contract import bind_laya_action_proposal logger = logging.getLogger("agent.orchestrator") @@ -77,6 +79,45 @@ def _exit_turn(self) -> None: except RuntimeError: logger.debug("agent turn lock release requested while unlocked", exc_info=True) + def _current_speech_tone(self) -> Optional[Dict[str, Any]]: + """Take one safe mood snapshot for every spoken chunk in this turn.""" + mood_manager = getattr(self, "mood_manager", None) + get_tone = getattr(mood_manager, "get_speech_tone", None) + if not callable(get_tone): + return None + try: + tone = get_tone() + return dict(tone) if isinstance(tone, dict) and tone else None + except Exception: + logger.debug("Could not resolve mood-aware speech tone", exc_info=True) + return None + + def _record_laya_latency(self, trace_id: str) -> None: + """Keep System 1 time separate from model TTFT in the turn trace.""" + decision = getattr(self, "_last_laya_fast_path_decision", None) + if decision is None: + return + try: + latency_trace.mark( + trace_id, + "laya.inference", + { + "inference_ms": max(0.0, float(getattr(decision, "inference_ms", 0.0) or 0.0)), + "target_module": str(getattr(decision, "target_module", "") or ""), + "urgency": float(getattr(decision, "urgency_score", 0.0) or 0.0), + }, + ) + except Exception: + logger.debug("Could not record Laya inference latency", exc_info=True) + + @staticmethod + def _bind_laya_action_request(decision: Any, request_id: str) -> None: + """Attach the current internal request ID to an inert Laya proposal.""" + proposal = getattr(decision, "suggested_action", None) + if proposal is None: + return + decision.suggested_action = bind_laya_action_proposal(proposal, request_id) + def check_survival_drives(self) -> Optional[str]: """Overrides logic if critical limits are reached.""" bat = self.world_state.get_state().get("battery_percent", 100) @@ -143,12 +184,43 @@ def step( session_language = self._normalize_session_language(language) progress_token = self.progress_manager.new_request(language=session_language) self._active_progress_token = progress_token + request_id = uuid.uuid4().hex + self._active_laya_request_id = request_id callback = self._build_progress_callback(progress_token, progress_cb) self.tool_registry.status_hook = callback try: use_fast_path = self._should_fast_path(user_prompt, native_tools=native_tools) - self.progress_manager.emit_ack(progress_token, speak=not use_fast_path) + self._record_laya_latency(trace_id) + custom_ack = "" + laya_eng = getattr(self, "laya_engine", None) + if laya_eng and getattr(laya_eng, "enabled", True): + try: + if not use_fast_path: + custom_ack = laya_eng.get_instant_filler(language=session_language) + # Trigger immediate affective body language if praise/rudeness detected + laya_dec = getattr(self, "_last_laya_fast_path_decision", None) + if laya_dec is None: + try: + laya_dec = laya_eng.decide_with_context( + user_prompt, + world_state=self.world_state.get_state(), + visual_context=self.world_state.get_laya_scene_summary(), + ) + except (AttributeError, TypeError): + laya_dec = laya_eng.decide(user_prompt) + if laya_dec is not None: + self._bind_laya_action_request(laya_dec, request_id) + if laya_dec and hasattr(self.world_state, "update_laya_state"): + self.world_state.update_laya_state(laya_dec) + if laya_dec and laya_dec.affective_event in {"user_praise", "user_rude"}: + reaction = laya_eng.get_affective_reaction(laya_dec.affective_event) + if autonomy_client and hasattr(autonomy_client, "set_expression_event"): + autonomy_client.set_expression_event("agent.affect", reaction) + except Exception: + pass + + self.progress_manager.emit_ack(progress_token, custom_text=custom_ack, speak=not use_fast_path) latency_trace.mark(trace_id, "agent.context_start") survival_override = self.check_survival_drives() world_context = self.world_state.inject_world_state("") @@ -173,6 +245,7 @@ def step( subagent_reports: List[Dict[str, Any]] = [] executed_actions: List[Dict[str, Any]] = [] stream_state = {"spoken": 0} + speech_tone = self._current_speech_tone() on_sentence: Optional[Callable[[str, int], None]] = None if self.speech_arbiter._speak_fn is not None: def speak_sentence(sentence: str, index: int) -> None: @@ -186,6 +259,7 @@ def speak_sentence(sentence: str, index: int) -> None: sentence, index=index, language=chunk_lang, + tone=speech_tone, trace_id=trace_id, ) if item_id: @@ -255,6 +329,7 @@ def speak_sentence(sentence: str, index: int) -> None: self.progress_manager.emit_final(progress_token) self.tool_registry.status_hook = previous_hook self._active_progress_token = "" + self._active_laya_request_id = "" self._exit_turn() def step_event( @@ -288,6 +363,7 @@ def step_event( pass use_fast_path = self._should_fast_path(event_prompt, native_tools=True) + self._record_laya_latency(trace_id) self.progress_manager.emit_ack(progress_token, speak=not use_fast_path) latency_trace.mark(trace_id, "agent.context_start") survival_override = self.check_survival_drives() @@ -313,6 +389,7 @@ def step_event( subagent_reports: List[Dict[str, Any]] = [] executed_actions: List[Dict[str, Any]] = [] stream_state = {"spoken": 0} + speech_tone = self._current_speech_tone() on_sentence: Optional[Callable[[str, int], None]] = None if self.speech_arbiter._speak_fn is not None: def speak_sentence(sentence: str, index: int) -> None: @@ -320,6 +397,7 @@ def speak_sentence(sentence: str, index: int) -> None: sentence, index=index, language=session_language, + tone=speech_tone, trace_id=trace_id, ) if item_id: @@ -473,4 +551,3 @@ def _sentrybot_batch06j_exit_turn(self, *args, **kwargs): pass except AttributeError: pass - diff --git a/modules/agent_core/services/agent_context.py b/modules/agent_core/services/agent_context.py index cd69688f..b9af6b60 100644 --- a/modules/agent_core/services/agent_context.py +++ b/modules/agent_core/services/agent_context.py @@ -315,11 +315,15 @@ def _init_tri_layer(self, config: Dict[str, Any], agent_cfg: Dict[str, Any]) -> if not self._vision_input_available(): default_modules = [m for m in default_modules if str(m).strip().lower() != "vlm_bridge"] profile_overrides = tri_cfg.get("profiles") if isinstance(tri_cfg.get("profiles"), dict) else None - self.subagent_profiles = build_subagent_profiles(profile_overrides) + from .laya_engine import LayaEngine + + self.laya_engine = LayaEngine.get_instance(config) + self.sensor_loop.laya_engine = self.laya_engine self.router = TriLayerRouter( profiles=self.subagent_profiles, max_subagents=self._safe_int(router_cfg.get("max_subagents", 2), fallback=2, minimum=1), default_modules=default_modules, + laya_engine=self.laya_engine, ) self.tri_layer_enabled = bool(tri_cfg.get("enabled", True)) self.api_native_tools = bool(tri_cfg.get("api_native_tools", False)) diff --git a/modules/agent_core/services/agent_memory_sync.py b/modules/agent_core/services/agent_memory_sync.py index ae9ffacd..536f337f 100644 --- a/modules/agent_core/services/agent_memory_sync.py +++ b/modules/agent_core/services/agent_memory_sync.py @@ -89,10 +89,38 @@ def _unified_progress_cb(event: Dict[str, Any]) -> None: return _unified_progress_cb def _should_fast_path(self, user_prompt: str, *, native_tools: bool = False) -> bool: + self._last_laya_fast_path_decision = None if native_tools: return True if not self.fast_path_enabled: return False + + # System 1 Laya fast-path check + laya_engine = getattr(self, "laya_engine", None) + if laya_engine is not None and getattr(laya_engine, "enabled", True): + try: + world_state = self.world_state.get_state() if hasattr(self.world_state, "get_state") else None + visual_context = ( + self.world_state.get_laya_scene_summary() + if hasattr(self.world_state, "get_laya_scene_summary") else None + ) + try: + is_fast, decision = laya_engine.should_fast_path( + user_prompt, + world_state=world_state, + visual_context=visual_context, + ) + except TypeError: + # Keep compatibility with injected pre-context Laya adapters. + is_fast, decision = laya_engine.should_fast_path(user_prompt) + self._last_laya_fast_path_decision = decision + if is_fast: + return True + if decision is not None and not decision.is_direct_command: + return False + except Exception: + pass + return len(str(user_prompt or "").strip()) <= self.fast_path_max_chars def _native_loop_messages(self, session_language: str) -> List[Dict[str, Any]]: diff --git a/modules/agent_core/services/agent_streaming.py b/modules/agent_core/services/agent_streaming.py index d6db9fa0..bbaccc1f 100644 --- a/modules/agent_core/services/agent_streaming.py +++ b/modules/agent_core/services/agent_streaming.py @@ -30,6 +30,7 @@ class AgentStreamingMixin: _cached_model_names_ts: float _SENTENCE_END_RE: Optional[re.Pattern] = None + _FIRST_CLAUSE_SPLIT_RE: Optional[re.Pattern] = None def _build_provider_tool_instruction(self, tools: List[Dict[str, Any]]) -> str: return build_provider_tool_instruction(tools) @@ -78,11 +79,29 @@ def _pick_runtime_model(self, preferred_model: str) -> str: return fallback return model - def _extract_ready_sentences(self, buffer: str) -> Tuple[List[str], str]: + def _extract_ready_sentences( + self, + buffer: str, + *, + is_first_chunk: bool = False, + ) -> Tuple[List[str], str]: if not buffer: return [], "" if AgentStreamingMixin._SENTENCE_END_RE is None: - AgentStreamingMixin._SENTENCE_END_RE = re.compile(r"([.!?…\n]+)") + AgentStreamingMixin._SENTENCE_END_RE = re.compile(r"([.!?…\n]+)") + if AgentStreamingMixin._FIRST_CLAUSE_SPLIT_RE is None: + AgentStreamingMixin._FIRST_CLAUSE_SPLIT_RE = re.compile(r"([,;:—–\n]+|[.!?…]+)") + + # Ultra low-latency first clause break (sub-350ms first speech audio) + if is_first_chunk: + words = buffer.split() + if len(words) >= 2 or len(buffer) >= 12: + parts = AgentStreamingMixin._FIRST_CLAUSE_SPLIT_RE.split(buffer) + if len(parts) > 1: + first_clause = (parts[0] + parts[1]).strip() + if len(first_clause.split()) >= 1 and len(first_clause) >= 4: + remainder = "".join(parts[2:]).lstrip() + return [first_clause], remainder parts = AgentStreamingMixin._SENTENCE_END_RE.split(buffer) if len(parts) <= 1: @@ -111,7 +130,10 @@ def _chat_maybe_stream( if tools: # Native tool-calling must use the normal chat path so Ollama can return tool_calls. return self._chat_turn(model, messages, tools, options) - if not on_sentence or not getattr(self, "persona_stream_enabled", False): + if not on_sentence: + return self._chat_turn(model, messages, tools, options) + # Enable streaming when on_sentence callback is provided unless explicitly forced off + if getattr(self, "persona_stream_enabled", None) is False: return self._chat_turn(model, messages, tools, options) return self._stream_turn_sentence_by_sentence(model, messages, tools, options, on_sentence=on_sentence) @@ -132,9 +154,10 @@ def _stream_turn_sentence_by_sentence( first_token_ts: Optional[float] = None t0 = time.time() sentence_idx = 0 + is_first_chunk = True def _emit(s: str) -> None: - nonlocal sentence_idx + nonlocal sentence_idx, is_first_chunk try: import inspect sig = inspect.signature(on_sentence) @@ -151,6 +174,7 @@ def _emit(s: str) -> None: except Exception: pass sentence_idx += 1 + is_first_chunk = False try: stream_kwargs: Dict[str, Any] = { @@ -176,7 +200,7 @@ def _emit(s: str) -> None: continue full_text.append(content) buffer += content - sentences, buffer = self._extract_ready_sentences(buffer) + sentences, buffer = self._extract_ready_sentences(buffer, is_first_chunk=is_first_chunk) for sentence in sentences: _emit(sentence) diff --git a/modules/agent_core/services/agent_subagents.py b/modules/agent_core/services/agent_subagents.py index b8a720a1..db78b5e4 100644 --- a/modules/agent_core/services/agent_subagents.py +++ b/modules/agent_core/services/agent_subagents.py @@ -28,20 +28,18 @@ class AgentSubagentsMixin: _stream_turn_sentence_by_sentence: Callable[..., Any] def _route_subagents(self, prompt: str) -> List[SubAgentProfile]: - low = prompt.lower() + module_names = self.router.route(prompt) if getattr(self, "router", None) else [] selected: List[SubAgentProfile] = [] - - for name, profile in self.subagent_profiles.items(): - if not profile.enabled: - continue - matches = any(k in low for k in profile.keywords) - if matches: + for name in module_names: + profile = self.subagent_profiles.get(name) + if profile and getattr(profile, "enabled", True): selected.append(profile) - if not selected and "fast_reflex" in self.subagent_profiles: - reflex = self.subagent_profiles["fast_reflex"] - if reflex.enabled: - selected.append(reflex) + self.last_routed_subagents = [profile.name for profile in selected] + if not selected: + fallback = self.subagent_profiles.get("fast_reflex") or self.subagent_profiles.get("agent_core") + if fallback and getattr(fallback, "enabled", True): + selected.append(fallback) return selected diff --git a/modules/agent_core/services/agent_turn.py b/modules/agent_core/services/agent_turn.py index f9e98f35..51e29957 100644 --- a/modules/agent_core/services/agent_turn.py +++ b/modules/agent_core/services/agent_turn.py @@ -304,7 +304,9 @@ def _normalize_tool_calls(raw_calls: Any) -> List[Dict[str, Any]]: if not name: continue - out = self.tool_registry.execute(name, args) + laya_decision = getattr(self, "_last_laya_fast_path_decision", None) + laya_hint = getattr(laya_decision, "suggested_tool", None) + out = self.tool_registry.execute(name, args, laya_hint=laya_hint) recovery_meta = None if recovery_handler is not None: try: diff --git a/modules/agent_core/services/laya_action_contract.py b/modules/agent_core/services/laya_action_contract.py new file mode 100644 index 00000000..91dd33b6 --- /dev/null +++ b/modules/agent_core/services/laya_action_contract.py @@ -0,0 +1,168 @@ +"""Fail-closed conversion of typed Laya choices into inert action proposals.""" + +from __future__ import annotations + +import math +from typing import Any, Dict, Optional + + +LIGHT_COLORS = ( + "red", "green", "blue", "yellow", "orange", "purple", "pink", + "cyan", "teal", "magenta", "white", "off", +) +LIGHT_EFFECTS = ( + "COMET", "PULSE", "WAVE", "SOLID", "OFF", "BREATHE", + "RANDOM_BLINK", "TWINKLE", +) +_UNSPECIFIED = "unspecified" + + +def get_laya_action_questions() -> Dict[str, Dict[str, Any]]: + """Return finite choice questions; no free-form actuator values are requested.""" + return { + "action_color": { + "type": "choice", + "instructions": "Işık komutunda kullanıcının istediği renk hangisi?", + "criteria": { + **{color: f"{color} rengi" for color in LIGHT_COLORS}, + _UNSPECIFIED: "Kullanıcı bir renk belirtmedi veya istek ışık rengi değil", + }, + }, + "action_effect": { + "type": "choice", + "instructions": "Işık komutunda kullanıcı hangi animasyon efektini istiyor?", + "criteria": { + **{effect: f"{effect.lower()} ışık animasyonu" for effect in LIGHT_EFFECTS}, + _UNSPECIFIED: "Kullanıcı bir animasyon belirtmedi", + }, + }, + } + + +def build_laya_action_proposal( + *, + target_module: str, + module_confidence: Any, + is_direct_command: bool, + direct_confidence: Any, + answers: Any, + policy: Any, +) -> Optional[Dict[str, Any]]: + """Build an inert set_lights proposal only when all policy checks pass. + + This function never dispatches a tool or touches hardware. The feature is + opt-in and its caller must keep proposal creation separate from execution. + """ + if not isinstance(policy, dict) or not policy.get("enabled", False): + return None + allowed = policy.get("allowed_actions", []) + if not isinstance(allowed, list) or "set_lights" not in allowed: + return None + if target_module != "neopixel" or is_direct_command is not True: + return None + + threshold = _confidence(policy.get("confidence_threshold")) + confidences = ( + _confidence(module_confidence), + _confidence(direct_confidence), + ) + if threshold is None or any(value is None or value < threshold for value in confidences): + return None + if not isinstance(answers, dict): + return None + + color, color_conf = _choice_answer(answers.get("action_color")) + effect, effect_conf = _choice_answer(answers.get("action_effect")) + if color not in LIGHT_COLORS or effect not in (*LIGHT_EFFECTS, _UNSPECIFIED): + return None + if color_conf is None or effect_conf is None or color_conf < threshold or effect_conf < threshold: + return None + + if effect == _UNSPECIFIED: + effect = policy.get("default_light_effect") + if effect not in LIGHT_EFFECTS: + return None + + proposal_confidence = min(*confidences, color_conf, effect_conf) + return { + "schema_version": 1, + "tool_name": "set_lights", + "arguments": {"effect": effect, "color": color}, + "confidence": proposal_confidence, + } + + +def _choice_answer(answer: Any) -> tuple[Optional[str], Optional[float]]: + if not isinstance(answer, dict) or answer.get("type") != "choice": + return None, None + choice = answer.get("choice") + confidence = _confidence(answer.get("confidence")) + if not isinstance(choice, str): + return None, None + return choice, confidence + + +def bind_laya_action_proposal( + proposal: Any, request_id: Any +) -> Optional[Dict[str, Any]]: + """Bind an inert proposal to the request that produced it.""" + if not isinstance(proposal, dict) or not isinstance(request_id, str) or not request_id.strip(): + return None + if "request_id" in proposal and proposal["request_id"] != request_id: + return None + if not isinstance(proposal.get("arguments"), dict): + return None + bound = dict(proposal) + bound["arguments"] = dict(proposal["arguments"]) + bound["request_id"] = request_id + return bound + + +def validate_laya_action_proposal( + proposal: Any, *, expected_request_id: Any, policy: Any +) -> bool: + """Validate an exact, request-bound proposal without dispatching it.""" + if not isinstance(proposal, dict) or not isinstance(policy, dict): + return False + if policy.get("enabled") is not True: + return False + allowed = policy.get("allowed_actions") + if not isinstance(allowed, list) or "set_lights" not in allowed: + return False + if not isinstance(expected_request_id, str) or not expected_request_id.strip(): + return False + if set(proposal) != {"schema_version", "tool_name", "arguments", "confidence", "request_id"}: + return False + if type(proposal.get("schema_version")) is not int or proposal["schema_version"] != 1: + return False + if proposal.get("tool_name") != "set_lights": + return False + if proposal.get("request_id") != expected_request_id: + return False + arguments = proposal.get("arguments") + if not isinstance(arguments, dict) or set(arguments) != {"color", "effect"}: + return False + if arguments["color"] not in LIGHT_COLORS or arguments["effect"] not in LIGHT_EFFECTS: + return False + threshold = _confidence(policy.get("confidence_threshold")) + confidence = _confidence(proposal.get("confidence")) + return threshold is not None and confidence is not None and confidence >= threshold + + +def _confidence(value: Any) -> Optional[float]: + if isinstance(value, bool) or not isinstance(value, (int, float)): + return None + score = float(value) + if not math.isfinite(score) or score < 0.0 or score > 1.0: + return None + return score + + +__all__ = [ + "LIGHT_COLORS", + "LIGHT_EFFECTS", + "bind_laya_action_proposal", + "build_laya_action_proposal", + "get_laya_action_questions", + "validate_laya_action_proposal", +] diff --git a/modules/agent_core/services/laya_engine.py b/modules/agent_core/services/laya_engine.py new file mode 100644 index 00000000..b6d72e61 --- /dev/null +++ b/modules/agent_core/services/laya_engine.py @@ -0,0 +1,676 @@ +from __future__ import annotations + +import glob +import logging +import os +import random +import threading +import time +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional, Sequence, Tuple + +logger = logging.getLogger("agent.laya_engine") + +THINKING_FILLERS_TR = [ + "Hemen bakıyorum...", + "Bir düşüneyim...", + "Hımm, anladım...", + "Bakalım dostum...", + "Hemen inceliyorum...", +] + +THINKING_FILLERS_EN = [ + "Hmm, let me think...", + "Looking into that...", + "One second...", + "Checking that for you...", +] + +FAST_ACK_TR = [ + "Hemen yapıyorum!", + "Tamamdır!", + "Anlaşıldı!", +] + +FAST_ACK_EN = [ + "On it!", + "Right away!", + "Got it!", +] + + +@dataclass +class LayaDecision: + """Represents a single System 1 reflex decision from the Laya neural model.""" + + target_module: str + module_confidence: float + is_direct_command: bool + direct_confidence: float + affective_event: str + affect_confidence: float + urgency_score: float + urgency_confidence: float + inference_ms: float + suggested_tool: Optional[str] = None + suggested_action: Optional[Dict[str, Any]] = None + raw_answers: Dict[str, Any] = field(default_factory=dict) + emergency_threshold: float = 2.0 + emergency_target_confidence_threshold: float = 0.90 + emergency_direct_confidence_threshold: float = 0.80 + emergency_urgency_confidence_threshold: float = 0.80 + + @property + def is_emergency(self) -> bool: + direct_is_confident = ( + self.is_direct_command + and self.direct_confidence >= self.emergency_direct_confidence_threshold + ) + target_is_confident = ( + self.target_module == "emergency_stop" + and self.module_confidence >= self.emergency_target_confidence_threshold + ) + urgency_is_confident = ( + self.urgency_score >= self.emergency_threshold + and self.urgency_confidence >= self.emergency_urgency_confidence_threshold + ) + return direct_is_confident and (target_is_confident or urgency_is_confident) + + +class LayaEngine: + """System 1 Reflex Engine powered by the Laya multilingual model. + + Provides sub-50ms neural intent routing, urgency scoring, affective event + classification, and fast-path reflex execution without invoking heavy LLMs. + """ + + _instance: Optional["LayaEngine"] = None + _lock = threading.Lock() + + # SentryBOT standard question specification for Laya System 1 + SENTRYBOT_QUESTIONS: Dict[str, Any] = { + "target_module": { + "type": "choice", + "instructions": "Bu kullanıcı girdisi SentryBOT robotunun hangi donanım/eylem modülüne ait?", + "criteria": { + "neopixel": "Işık, renk, parlaklık veya LED animasyonlarını değiştirme", + "arduino_serial": "Kafa hareketi, pan/tilt, motor, yürüme veya durdurma", + "emergency_stop": "Acil durum, anında durma, tehlike, çarpma riski veya acil kapatma", + "speak": "Robotun bir şey söylemesini, konuşmasını veya ses çıkarmasını isteme", + "camera": "Kamera, görme, etrafa bakma, nesne/yüz tanıma veya görsel soru", + "system2_chat": "Genel sohbet, soru-cevap, felsefe, akıl yürütme, hikaye veya tavsiye", + }, + }, + "is_direct_command": { + "type": "choice", + "instructions": "Bu girdi doğrudan hızlı icra edilecek bir donanım komutu mu yoksa düşünce gerektiren sohbet mi?", + "criteria": { + "direct_action": "Hemen uygulanacak net bir donanım/refleks komutu", + "conversation": "Düşünce, cevap veya diyalog gerektiren sohbet/soru", + }, + }, + "affective_event": { + "type": "choice", + "instructions": "Kullanıcının robota karşı duygusal/sosyal tutumu nedir?", + "criteria": { + "user_praise": "Övgü, sevgi, aferin, teşekkür, iltifat veya sevme", + "user_rude": "Kabalık, hakaret, küfür, susturma veya kızgınlık", + "neutral": "Duygusal olmayan nötr komut veya bilgi sorusu", + }, + }, + "urgency": { + "type": "score", + "instructions": "Bu girdinin aciliyet ve tehlike seviyesi nedir?", + "criteria": [ + "sakin, normal konuşma veya bilgi sorusu", + "normal günlük hareket veya rutin eylem komutu", + "hızlı yapılması istenen öncelikli komut", + "son derece acil, tehlike, kaza riski veya anında durdurma", + ], + }, + } + + MODULE_TOOL_MAP: Dict[str, str] = { + "neopixel": "set_lights", + "arduino_serial": "move_head", + "emergency_stop": "stop_follow", + "speak": "speak", + "camera": "get_vision", + } + + def __init__( + self, + enabled: bool = True, + confidence_threshold: float = 0.60, + model_name_or_path: Optional[str] = None, + subfolder: str = "multilingual", + device: str = "cpu", + laya_cfg: Optional[Dict[str, Any]] = None, + ) -> None: + self.enabled = enabled + self.confidence_threshold = float(confidence_threshold) + self.subfolder = subfolder + self.device = device + self._model_path = model_name_or_path + self._agent: Any = None + self._load_lock = threading.Lock() + self._load_attempted = False + self._is_available = False + self._load_time_ms = 0.0 + + # Config-driven sub-sections + _cfg = laya_cfg or {} + fast_cfg = _cfg.get("fast_path", {}) if isinstance(_cfg.get("fast_path"), dict) else {} + self.fast_path_enabled = bool(fast_cfg.get("enabled", True)) + self.emergency_bypass_safety = bool(fast_cfg.get("emergency_bypass_safety", False)) + + filler_cfg = _cfg.get("filler", {}) if isinstance(_cfg.get("filler"), dict) else {} + self.filler_enabled = bool(filler_cfg.get("enabled", True)) + self.filler_cooldown_s = float(filler_cfg.get("cooldown_s", 12.0)) + self.filler_mood_aware = bool(filler_cfg.get("mood_aware", True)) + self._last_filler_ts = 0.0 + + affect_cfg = _cfg.get("affective", {}) if isinstance(_cfg.get("affective"), dict) else {} + self.mood_impact_scale = float(affect_cfg.get("mood_impact_scale", 1.0)) + self.vision_events_enabled = bool(affect_cfg.get("vision_events", True)) + self.face_emotion_events_enabled = bool(affect_cfg.get("face_emotion_events", True)) + + urgency_cfg = _cfg.get("urgency", {}) if isinstance(_cfg.get("urgency"), dict) else {} + self.emergency_threshold = float(urgency_cfg.get("emergency_threshold", 2.0)) + self.high_urgency_threshold = float(urgency_cfg.get("high_threshold", 1.5)) + self.high_urgency_confidence_threshold = float(urgency_cfg.get("high_confidence_threshold", 0.60)) + self.emergency_target_confidence_threshold = float( + urgency_cfg.get("emergency_target_confidence_threshold", 0.90) + ) + self.emergency_direct_confidence_threshold = float( + urgency_cfg.get("emergency_direct_confidence_threshold", 0.80) + ) + self.emergency_urgency_confidence_threshold = float( + urgency_cfg.get("emergency_urgency_confidence_threshold", 0.80) + ) + self.sensor_boost_hz = float(urgency_cfg.get("sensor_boost_hz", 5.0)) + self.sensor_boost_hold_s = float(urgency_cfg.get("sensor_boost_hold_s", 8.0)) + + action_cfg = _cfg.get("action_proposals", {}) + self.action_proposal_cfg = action_cfg if isinstance(action_cfg, dict) else {} + + telem_cfg = _cfg.get("telemetry", {}) if isinstance(_cfg.get("telemetry"), dict) else {} + self._log_decisions = bool(telem_cfg.get("log_decisions", True)) + self._max_decision_log = int(telem_cfg.get("max_decision_log", 20)) + + # Telemetry counters + self._decision_count = 0 + self._total_inference_ms = 0.0 + self._last_decisions: List[Dict[str, Any]] = [] + self._last_decision: Optional["LayaDecision"] = None + + @classmethod + def get_instance(cls, config: Optional[Dict[str, Any]] = None) -> "LayaEngine": + """Thread-safe singleton accessor.""" + with cls._lock: + if cls._instance is None: + cfg = config or {} + tri_cfg = cfg.get("tri_layer", {}) if isinstance(cfg.get("tri_layer"), dict) else {} + laya_cfg = tri_cfg.get("laya", {}) if isinstance(tri_cfg.get("laya"), dict) else {} + enabled = bool(laya_cfg.get("enabled", True)) + threshold = float(laya_cfg.get("confidence_threshold", 0.60)) + model_path = laya_cfg.get("model_path") + device = str(laya_cfg.get("device", "cpu")) + cls._instance = cls( + enabled=enabled, + confidence_threshold=threshold, + model_name_or_path=model_path, + device=device, + laya_cfg=laya_cfg, + ) + return cls._instance + + @staticmethod + def _find_cached_snapshot() -> Optional[str]: + """Locates pre-downloaded Hugging Face snapshot for convaiinnovations/laya.""" + hub_pattern = os.path.expanduser( + "~/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/*/multilingual" + ) + matches = glob.glob(hub_pattern) + if matches and os.path.isdir(matches[0]): + return matches[0] + + parent_pattern = os.path.expanduser( + "~/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/*" + ) + parent_matches = glob.glob(parent_pattern) + for snapshot in parent_matches: + if os.path.isdir(snapshot): + multi = os.path.join(snapshot, "multilingual") + if os.path.isdir(multi): + return multi + if os.path.isfile(os.path.join(snapshot, "model.safetensors")): + return snapshot + return None + + def _ensure_loaded(self) -> bool: + """Lazily loads the Laya model on first inference request.""" + if not self.enabled: + return False + if self._is_available and self._agent is not None: + return True + if self._load_attempted and not self._is_available: + return False + + with self._load_lock: + if self._agent is not None: + return True + self._load_attempted = True + try: + import laya # type: ignore + except ImportError: + logger.warning( + "laya library is not installed in current environment. " + "System 1 neural reflexes will use fallback routing." + ) + self._is_available = False + return False + + t0 = time.perf_counter() + resolved_path = self._model_path or self._find_cached_snapshot() + try: + if resolved_path and os.path.exists(resolved_path): + logger.info("Loading Laya model from local snapshot: %s", resolved_path) + self._agent = laya.load(resolved_path) + else: + logger.info("Loading Laya model from HF hub: convaiinnovations/laya (subfolder=%s)", self.subfolder) + self._agent = laya.load("convaiinnovations/laya", subfolder=self.subfolder) + + self._load_time_ms = round((time.perf_counter() - t0) * 1000.0, 2) + self._is_available = True + logger.info( + "Laya System 1 loaded successfully in %.2f ms (device=%s)", + self._load_time_ms, + getattr(self._agent, "device", "unknown"), + ) + return True + except Exception as exc: + logger.warning("Failed to initialize Laya model: %s", exc) + self._is_available = False + return False + + @property + def is_available(self) -> bool: + return self._is_available or (self.enabled and not self._load_attempted) + + def decide(self, text: str) -> Optional[LayaDecision]: + """Runs fast System 1 inference on user input.""" + cleaned = str(text or "").strip() + if not cleaned: + return None + + if not self._ensure_loaded() or self._agent is None: + return None + + t0 = time.perf_counter() + try: + state = {"text": cleaned} + questions = self.SENTRYBOT_QUESTIONS + if self.action_proposal_cfg.get("enabled", False): + from modules.agent_core.services.laya_action_contract import get_laya_action_questions + + questions = {**questions, **get_laya_action_questions()} + raw = self._agent.system_one(state, questions) + elapsed_ms = round((time.perf_counter() - t0) * 1000.0, 2) + answers = raw.get("answers", {}) if isinstance(raw, dict) else {} + + target_ans = answers.get("target_module", {}) + target_mod = str(target_ans.get("choice") or target_ans.get("value") or "system2_chat") + target_conf = float(target_ans.get("confidence", 0.0)) + + direct_ans = answers.get("is_direct_command", {}) + direct_choice = str(direct_ans.get("choice") or direct_ans.get("value") or "conversation") + direct_conf = float(direct_ans.get("confidence", 0.0)) + is_direct = direct_choice == "direct_action" + + affect_ans = answers.get("affective_event", {}) + affect_choice = str(affect_ans.get("choice") or affect_ans.get("value") or "neutral") + affect_conf = float(affect_ans.get("confidence", 0.0)) + + urgency_ans = answers.get("urgency", {}) + urgency_score = float(urgency_ans.get("score") if "score" in urgency_ans else urgency_ans.get("value", 0.0)) + urgency_conf = float(urgency_ans.get("confidence", 0.0)) + + suggested_tool = self.MODULE_TOOL_MAP.get(target_mod) + + decision = LayaDecision( + target_module=target_mod, + module_confidence=target_conf, + is_direct_command=is_direct, + direct_confidence=direct_conf, + affective_event=affect_choice, + affect_confidence=affect_conf, + urgency_score=urgency_score, + urgency_confidence=urgency_conf, + inference_ms=elapsed_ms, + suggested_tool=suggested_tool, + raw_answers=answers, + emergency_threshold=self.emergency_threshold, + emergency_target_confidence_threshold=self.emergency_target_confidence_threshold, + emergency_direct_confidence_threshold=self.emergency_direct_confidence_threshold, + emergency_urgency_confidence_threshold=self.emergency_urgency_confidence_threshold, + ) + if self.action_proposal_cfg.get("enabled", False): + from modules.agent_core.services.laya_action_contract import build_laya_action_proposal + + decision.suggested_action = build_laya_action_proposal( + target_module=target_mod, + module_confidence=target_conf, + is_direct_command=is_direct, + direct_confidence=direct_conf, + answers=answers, + policy=self.action_proposal_cfg, + ) + + # Telemetry tracking + self._decision_count += 1 + self._total_inference_ms += elapsed_ms + self._last_decision = decision + if self._log_decisions: + self._last_decisions.append({ + "text": cleaned[:50], + "target": target_mod, + "conf": round(target_conf, 3), + "direct": is_direct, + "affect": affect_choice, + "urgency": round(urgency_score, 2), + "ms": elapsed_ms, + "timestamp": time.time(), + }) + if len(self._last_decisions) > self._max_decision_log: + self._last_decisions = self._last_decisions[-self._max_decision_log:] + logger.debug( + "Laya decision: target=%s conf=%.2f direct=%s affect=%s urgency=%.1f in %.1fms", + target_mod, target_conf, is_direct, affect_choice, urgency_score, elapsed_ms, + ) + + return decision + except Exception as exc: + logger.warning("Laya inference failed for text '%s': %s", cleaned[:50], exc) + return None + + def route(self, user_prompt: str, available_modules: Sequence[str]) -> List[str]: + """Provides Laya-enhanced module routing. + + Returns prioritized list of modules based on System 1 neural classification. + """ + decision = self.decide(user_prompt) + if decision is None: + return [] + + if decision.module_confidence < self.confidence_threshold: + return [] + + target = decision.target_module + if target in available_modules: + return [target] + + # Aliases mapping + alias_map = { + "emergency_stop": ["arduino_serial", "autonomy"], + "camera": ["vlm_bridge", "camera"], + "speak": ["speak"], + "neopixel": ["neopixel", "interactions"], + "arduino_serial": ["arduino_serial", "hardware"], + "system2_chat": ["autonomy", "agent_core"], + } + candidates = alias_map.get(target, []) + return [c for c in candidates if c in available_modules] + + def should_fast_path( + self, + user_prompt: str, + world_state: Optional[Dict[str, Any]] = None, + visual_context: Optional[str] = None, + ) -> Tuple[bool, Optional["LayaDecision"]]: + """Determines if the prompt qualifies for instant hardware fast-path execution.""" + if not self.fast_path_enabled: + return False, None + + decision = self.decide_with_context( + user_prompt, + world_state=world_state, + visual_context=visual_context, + ) + if decision is None: + return False, None + + if decision.is_emergency: + return True, decision + + if decision.urgency_score >= self.high_urgency_threshold: + if decision.is_direct_command and decision.direct_confidence >= self.confidence_threshold: + return True, decision + + if decision.is_direct_command and decision.direct_confidence >= self.confidence_threshold: + if decision.target_module in {"neopixel", "arduino_serial", "speak", "emergency_stop"}: + return True, decision + + return False, decision + + def decide_with_context( + self, + text: str, + world_state: Optional[Dict[str, Any]] = None, + visual_context: Optional[str] = None, + ) -> Optional["LayaDecision"]: + """Context-enriched System 1 decision. + + Enriches the raw text with visual scene context and adjusts urgency + based on world state (battery, temperature, follow mode). + """ + enriched = str(text or "").strip() + if not enriched: + return None + + if visual_context: + scene_hint = str(visual_context)[:100] + enriched = f"{enriched} [Sahne: {scene_hint}]" + + decision = self.decide(enriched) + if decision is None: + return None + + # World state adjustments + if isinstance(world_state, dict): + battery = world_state.get("battery_percent", 100) + try: + bat_val = float(battery) + except (TypeError, ValueError): + bat_val = 100.0 + if bat_val < 15 and decision.urgency_score < self.high_urgency_threshold: + decision.urgency_score = self.high_urgency_threshold + + follow_active = world_state.get("follow_active", False) + if follow_active and decision.target_module == "camera": + decision.module_confidence = min(1.0, decision.module_confidence + 0.15) + + return decision + + def is_urgent(self, decision: Optional["LayaDecision"] = None) -> bool: + """Check if the latest or given decision indicates high urgency.""" + dec = decision or self._last_decision + if dec is None: + return False + return bool(getattr(dec, "is_emergency", False)) or ( + dec.urgency_score >= self.high_urgency_threshold + and dec.urgency_confidence >= self.high_urgency_confidence_threshold + ) + + def is_emergency_decision(self, decision: Optional["LayaDecision"] = None) -> bool: + """Check if the latest or given decision is an emergency.""" + dec = decision or self._last_decision + if dec is None: + return False + return bool(getattr(dec, "is_emergency", False)) + + def get_instant_filler(self, language: str = "tr") -> str: + """Returns an immediate conversational thinking filler (<100ms backchannel).""" + if not self.filler_enabled: + return "" + now = time.time() + if now - self._last_filler_ts < self.filler_cooldown_s: + return "" + self._last_filler_ts = now + lang = str(language or "tr").lower() + if lang.startswith("en"): + return random.choice(THINKING_FILLERS_EN) + return random.choice(THINKING_FILLERS_TR) + + def get_filler_with_affect(self, language: str = "tr", mood: str = "neutral") -> str: + """Mood-aware thinking filler — adjusts tone based on current emotion.""" + if not self.filler_enabled: + return "" + now = time.time() + if now - self._last_filler_ts < self.filler_cooldown_s: + return "" + self._last_filler_ts = now + + if not self.filler_mood_aware or mood == "neutral": + return self.get_instant_filler.__wrapped__(self, language) if hasattr(self.get_instant_filler, '__wrapped__') else self._pick_filler(language) + + lang = str(language or "tr").lower() + is_en = lang.startswith("en") + + if mood in ("joy", "curiosity"): + pool = [ + "Oh, interesting!", "Let me check!", "Curious about that...", + ] if is_en else [ + "Hemen bakıyorum!", "Ooo, ilginç bir soru!", "Bakalım ne bulacağım...", + ] + elif mood in ("sadness", "tired"): + pool = [ + "Let me see...", "Thinking...", "Hmm...", + ] if is_en else [ + "Hmm, bakayım...", "Bir kontrol edeyim...", "Düşünüyorum...", + ] + elif mood in ("anger", "furious"): + pool = [ + "Fine, checking.", "One moment.", "Looking into it.", + ] if is_en else [ + "Tamam, hemen bakıyorum.", "Bir dakika.", "İnceliyorum.", + ] + elif mood in ("fear",): + pool = [ + "O-okay, let me check...", "Hmm, checking...", + ] if is_en else [ + "T-tamam, bakıyorum...", "Hemen kontrol ediyorum...", + ] + else: + pool = THINKING_FILLERS_EN if is_en else THINKING_FILLERS_TR + + return random.choice(pool) + + def _pick_filler(self, language: str = "tr") -> str: + """Raw filler pick without cooldown check (internal use).""" + lang = str(language or "tr").lower() + if lang.startswith("en"): + return random.choice(THINKING_FILLERS_EN) + return random.choice(THINKING_FILLERS_TR) + + def get_fast_ack(self, language: str = "tr") -> str: + """Returns an instant verbal acknowledgment for direct actions.""" + lang = str(language or "tr").lower() + if lang.startswith("en"): + return random.choice(FAST_ACK_EN) + return random.choice(FAST_ACK_TR) + + def get_affective_reaction(self, affect: str) -> Dict[str, Any]: + """Maps detected affective event to immediate OLED face, LED, and emotion.""" + reactions: Dict[str, Dict[str, Any]] = { + "user_praise": { + "face": "happy", + "emotion": "joy", + "oled_text": "^_^", + "led_color": "#00ff88", + "led_mode": "pulse", + "body_pose": {"pan_delta": 5, "tilt_delta": -3}, + }, + "user_rude": { + "face": "sad", + "emotion": "sorrow", + "oled_text": ":'(", + "led_color": "#ff3333", + "led_mode": "slow_breathe", + "body_pose": {"pan_delta": -3, "tilt_delta": 5}, + }, + } + if affect in reactions: + return dict(reactions[affect]) + return { + "face": "neutral", + "emotion": "neutral", + "oled_text": "-_-", + "led_color": "#00d4ff", + "led_mode": "static", + "body_pose": {"pan_delta": 0, "tilt_delta": 0}, + } + + def get_telemetry(self) -> Dict[str, Any]: + """Returns Laya System 1 performance telemetry and decision history.""" + avg_ms = ( + round(self._total_inference_ms / self._decision_count, 2) + if self._decision_count > 0 + else 0.0 + ) + return { + "available": self._is_available, + "enabled": self.enabled, + "fast_path_enabled": self.fast_path_enabled, + "filler_enabled": self.filler_enabled, + "confidence_threshold": self.confidence_threshold, + "emergency_threshold": self.emergency_threshold, + "high_urgency_threshold": self.high_urgency_threshold, + "high_urgency_confidence_threshold": self.high_urgency_confidence_threshold, + "emergency_target_confidence_threshold": self.emergency_target_confidence_threshold, + "emergency_direct_confidence_threshold": self.emergency_direct_confidence_threshold, + "emergency_urgency_confidence_threshold": self.emergency_urgency_confidence_threshold, + "decision_count": self._decision_count, + "avg_inference_ms": avg_ms, + "load_time_ms": self._load_time_ms, + "last_decisions": list(self._last_decisions[-5:]), + "last_target": self._last_decision.target_module if self._last_decision else None, + "last_urgency": self._last_decision.urgency_score if self._last_decision else None, + } + + def get_mood_deltas(self, decision: Optional["LayaDecision"] = None) -> Dict[str, float]: + """Compute mood state deltas from a Laya decision, scaled by mood_impact_scale. + + Returns a dict of {mood_axis: delta} suitable for MoodManager.modify(). + """ + dec = decision or self._last_decision + if dec is None: + return {} + + scale = self.mood_impact_scale + deltas: Dict[str, float] = {} + + if dec.affective_event == "user_praise": + deltas["happiness"] = 25.0 * scale + deltas["anger"] = -15.0 * scale + elif dec.affective_event == "user_rude": + deltas["anger"] = 30.0 * scale + deltas["happiness"] = -25.0 * scale + deltas["fear"] = 10.0 * scale + + if bool(getattr(dec, "is_emergency", False)): + deltas["fear"] = deltas.get("fear", 0.0) + 20.0 * scale + deltas["energy"] = deltas.get("energy", 0.0) - 10.0 * scale + elif ( + dec.urgency_score >= self.high_urgency_threshold + and dec.urgency_confidence >= self.high_urgency_confidence_threshold + ): + deltas["curiosity"] = deltas.get("curiosity", 0.0) + 10.0 * scale + + if dec.is_direct_command: + deltas["stimulation_satisfy"] = 15.0 * scale + + if dec.target_module == "system2_chat": + deltas["social_satisfy"] = 20.0 * scale + + return deltas diff --git a/modules/agent_core/services/progress.py b/modules/agent_core/services/progress.py index 9d52a18a..53aa2f12 100644 --- a/modules/agent_core/services/progress.py +++ b/modules/agent_core/services/progress.py @@ -226,6 +226,12 @@ def emit_ack(self, token: str, custom_text: str = "", *, speak: bool = True) -> self._emit_interaction("agent.processing.start", {"stage": "ack"}) if not speak: return + if custom_text and self._speech_arbiter is not None and hasattr(self._speech_arbiter, "enqueue_laya_filler"): + self._speech_arbiter.enqueue_laya_filler( + custom_text, + language=self._lang_for(token), + ) + return text = custom_text or _msg_choice(self._lang_for(token), "ack") self._speak_progress(token, text, event_type="ack") diff --git a/modules/agent_core/services/sensor_loop.py b/modules/agent_core/services/sensor_loop.py index 8dbef4ba..79af6234 100644 --- a/modules/agent_core/services/sensor_loop.py +++ b/modules/agent_core/services/sensor_loop.py @@ -51,6 +51,7 @@ def __init__( vision_results_interval_s: float = 5.0, visual_context_interval_s: float = 10.0, skip_hardware_on_pc: bool = True, + laya_engine=None, ): self.world_state = world_state self.client = client @@ -60,6 +61,7 @@ def __init__( self.vision_results_interval_s = _safe_float(vision_results_interval_s, 5.0, 0.5) self.visual_context_interval_s = _safe_float(visual_context_interval_s, 10.0, 1.0) self.skip_hardware_on_pc = bool(skip_hardware_on_pc) + self.laya_engine = laya_engine self.running = False self.thread: Optional[threading.Thread] = None self._last: dict[str, float] = {} @@ -99,6 +101,26 @@ def _due(self, key: str, interval_s: float, now: float) -> bool: self._last[key] = now return True + def _current_loop_interval(self) -> float: + """Increase the loop tick rate briefly after a high urgency decision.""" + engine = self.laya_engine + if engine is None or not getattr(engine, "enabled", False): + return self.loop_interval + try: + telemetry = engine.get_telemetry() + decisions = telemetry.get("last_decisions", []) + latest = decisions[-1] if decisions else {} + urgency = float(latest.get("urgency", 0.0)) + threshold = float(getattr(engine, "high_urgency_threshold", 1.5)) + boost_hz = float(getattr(engine, "sensor_boost_hz", 5.0)) + hold_s = float(getattr(engine, "sensor_boost_hold_s", 8.0)) + decision_age = time.time() - float(latest.get("timestamp", 0.0)) + if urgency >= threshold and decision_age <= hold_s and boost_hz > 0: + return min(self.loop_interval, 1.0 / boost_hz) + except (AttributeError, TypeError, ValueError, IndexError): + logger.debug("Unable to read Laya urgency for sensor scheduling", exc_info=True) + return self.loop_interval + def _read_hardware(self) -> dict: updates = {} if self.pc_test and self.skip_hardware_on_pc: @@ -168,4 +190,4 @@ def _poll_loop(self): except Exception as exc: logger.error("Sensor poll error: %s", exc) - time.sleep(self.loop_interval) \ No newline at end of file + time.sleep(self._current_loop_interval()) diff --git a/modules/agent_core/services/speech_arbiter.py b/modules/agent_core/services/speech_arbiter.py index 9fe489c6..f6a86b2f 100644 --- a/modules/agent_core/services/speech_arbiter.py +++ b/modules/agent_core/services/speech_arbiter.py @@ -43,6 +43,7 @@ def split_sentences(text: str, max_chars: int = 160, max_chunks: int = 8) -> Lis class SpeechPriority: SAFETY = 95 + LAYA_FILLER = 75 FINAL_RESPONSE = 60 PROGRESS = 30 IDLE = 15 @@ -224,6 +225,22 @@ def enqueue_safety(self, text: str, trace_id: str = "") -> Optional[str]: trace_id=trace_id, ) + def enqueue_laya_filler( + self, + text: str, + language: str = "", + trace_id: str = "", + ) -> Optional[str]: + """Queue a short-lived Laya acknowledgment above routine progress.""" + return self.enqueue( + text, + priority=SpeechPriority.LAYA_FILLER, + category="laya_filler", + language=language, + max_age_s=4.0, + trace_id=trace_id, + ) + def cancel_by_token(self, cancel_token: str) -> int: token = str(cancel_token or "") if not token: diff --git a/modules/agent_core/services/tools/tool_registry.py b/modules/agent_core/services/tools/tool_registry.py index b944c696..cea10fa3 100644 --- a/modules/agent_core/services/tools/tool_registry.py +++ b/modules/agent_core/services/tools/tool_registry.py @@ -119,11 +119,13 @@ def _register_all(self): func = getattr(self, name) self._register(func, schema) - def execute(self, tool_name: str, kwargs: Dict[str, Any]) -> str: - """Executes the mapped tool and returns the string result.""" + def execute(self, tool_name: str, kwargs: Dict[str, Any], laya_hint: Optional[str] = None) -> str: + """Executes the LLM-selected tool; Laya hint is diagnostic only, never authority.""" if tool_name not in self.tools: return f"Error: Tool '{tool_name}' not found." + hint = str(laya_hint or "").strip()[:64] + acquired = False vision_held = False try: @@ -167,6 +169,7 @@ def execute(self, tool_name: str, kwargs: Dict[str, Any]) -> str: "type": "tool_done", "tool": tool_name, "result": result_str, + **({"laya_hint": hint, "laya_hint_matches": hint == tool_name} if hint else {}), } ) return result_str diff --git a/modules/agent_core/services/tri_layer.py b/modules/agent_core/services/tri_layer.py index 55765223..27291ed3 100644 --- a/modules/agent_core/services/tri_layer.py +++ b/modules/agent_core/services/tri_layer.py @@ -283,15 +283,17 @@ def build_subagent_profiles(overrides: Dict[str, dict] | None = None) -> Dict[st class TriLayerRouter: - """Low-latency keyword router for module-level sub-agents.""" + """Low-latency keyword and neural router for module-level sub-agents.""" def __init__( self, profiles: Dict[str, SubAgentProfile], max_subagents: int = 2, default_modules: Sequence[str] | None = None, + laya_engine: Any = None, ): self.profiles = profiles + self.laya_engine = laya_engine self._absolute_max = 8 self.max_subagents = self._coerce_max(max_subagents) fallback = tuple(default_modules or ("vlm_bridge", "autonomy", "agent_core")) @@ -397,11 +399,29 @@ def route(self, user_prompt: str) -> List[str]: text = str(user_prompt or "").strip().lower() if not text: return list(self.default_modules[: self.max_subagents]) + + laya_modules: List[str] = [] + if self.laya_engine is not None and getattr(self.laya_engine, "enabled", True): + try: + laya_modules = self.laya_engine.route(user_prompt, list(self.profiles.keys())) + except Exception: + laya_modules = [] + q_tokens = self._tokenize(text) scores = self._score_keyword_matches(text, q_tokens) self._apply_semantic_priors(q_tokens, scores) self._apply_emotion_priors(text, q_tokens, scores) + + for mod in laya_modules: + if mod in self.profiles: + decision = getattr(self.laya_engine, "_last_decision", None) + confidence = getattr(decision, "module_confidence", None) + boost = 5.0 + (max(0.0, min(1.0, float(confidence))) * 10.0) if confidence is not None else 10.0 + scores[mod] = scores.get(mod, 0.0) + boost + if not scores: + if laya_modules: + return laya_modules[: self.max_subagents] return list(self.default_modules[: self.max_subagents]) ranked = sorted(scores.items(), key=lambda item: (-item[1], item[0])) return [name for name, _ in ranked[: self.max_subagents]] @@ -471,4 +491,3 @@ def _sentrybot_profile_default_property(value): for _sentrybot_attr, _sentrybot_value in _sentrybot_profile_defaults.items(): if not hasattr(SubAgentProfile, _sentrybot_attr): setattr(SubAgentProfile, _sentrybot_attr, _sentrybot_profile_default_property(_sentrybot_value)) - diff --git a/modules/agent_core/services/world_state.py b/modules/agent_core/services/world_state.py index 0a3eba69..5439e8c2 100644 --- a/modules/agent_core/services/world_state.py +++ b/modules/agent_core/services/world_state.py @@ -58,7 +58,7 @@ def update_scene(self, context: Dict[str, Any]) -> None: name = str(p.get("name", "") or "").strip() if name and name.lower() != "unknown": people.append(name) - self.environment = { + environment = { "scene_summary": str(ctx.get("summary", "") or ""), "objects": [str(o.get("label", o)) if isinstance(o, dict) else str(o) for o in (ctx.get("objects", []) or [])][:8], "hazards": [str(h.get("label", h)) if isinstance(h, dict) else str(h) for h in (ctx.get("hazards", []) or [])][:5], @@ -66,6 +66,43 @@ def update_scene(self, context: Dict[str, Any]) -> None: "importance": float(ctx.get("importance_score", 0.0) or 0.0), "updated_at": str(ctx.get("timestamp", "") or datetime.now().isoformat()), } + with self._state_lock: + self.environment = environment + + def get_laya_scene_summary(self, max_chars: int = 100) -> str: + """Return a bounded, thread-safe visual summary for Laya inference.""" + with self._state_lock: + environment = dict(self.environment) + hazards = list(environment.get("hazards", []))[:2] + objects = list(environment.get("objects", []))[:5] + people = list(environment.get("people_present", []))[:3] + parts = [] + summary = str(environment.get("scene_summary", "") or "").strip() + if summary: + parts.append(summary) + if people: + parts.append(f"bilinen kişiler: {', '.join(str(person) for person in people)}") + if objects: + parts.append(f"nesneler: {', '.join(str(item) for item in objects)}") + if hazards: + parts.append(f"tehlikeler: {', '.join(str(item) for item in hazards)}") + return "; ".join(parts)[:max(1, int(max_chars))] + + def update_laya_state(self, decision: Any) -> None: + """Publish a compact snapshot of the latest Laya decision.""" + if decision is None: + return + try: + snapshot = { + "laya_last_target": str(getattr(decision, "target_module", "") or ""), + "laya_last_urgency": float(getattr(decision, "urgency_score", 0.0) or 0.0), + "laya_last_affect": str(getattr(decision, "affective_event", "neutral") or "neutral"), + "laya_last_direct": bool(getattr(decision, "is_direct_command", False)), + "laya_last_ms": float(getattr(decision, "inference_ms", 0.0) or 0.0), + } + except (TypeError, ValueError): + return + self.update_state(snapshot) def set_action_feedback(self, feedback: str): with self._state_lock: diff --git a/modules/autonomy/config/appraisal.yml b/modules/autonomy/config/appraisal.yml index 57c3caac..c3695e2a 100644 --- a/modules/autonomy/config/appraisal.yml +++ b/modules/autonomy/config/appraisal.yml @@ -19,6 +19,7 @@ rules: owner_lockout: { anger: 20, fear: 15, happiness: -10 } petted: { happiness: 16, anger: -16, fear: -8 } loud_noise: { fear: 22, anger: 5, energy: 6 } + visual_hazard: { fear: 20, energy: 5, curiosity: 3 } new_person: { curiosity: 12, energy: 4 } scene_change: { curiosity: 8 } darkness: { fear: 8, energy: -6 } diff --git a/modules/autonomy/config/config.yml b/modules/autonomy/config/config.yml index 9499585a..9ac65174 100644 --- a/modules/autonomy/config/config.yml +++ b/modules/autonomy/config/config.yml @@ -913,6 +913,10 @@ owner_learning: companion_goals: enabled: true event_cooldown_s: 12.0 + laya_urgency: + enabled: true + freshness_s: 8.0 + safety_score_floor: 0.75 goal_formation: enabled: true strategy: rules diff --git a/modules/autonomy/services/affective_appraisal.py b/modules/autonomy/services/affective_appraisal.py index 15d65291..0f6450a3 100644 --- a/modules/autonomy/services/affective_appraisal.py +++ b/modules/autonomy/services/affective_appraisal.py @@ -27,6 +27,7 @@ "command_ok": {"happiness": 6, "anger": -4}, "command_failed": {"anger": 12, "fear": 6}, "loud_noise": {"fear": 22, "anger": 5}, + "visual_hazard": {"fear": 20, "energy": 5, "curiosity": 3}, "new_person": {"curiosity": 12}, "petted": {"happiness": 16, "anger": -16}, } diff --git a/modules/autonomy/services/barge_in.py b/modules/autonomy/services/barge_in.py index 5ff3c380..bf0623dc 100644 --- a/modules/autonomy/services/barge_in.py +++ b/modules/autonomy/services/barge_in.py @@ -30,6 +30,7 @@ def should_interrupt( robot_speaking: bool, user_text: str, has_wakeword: bool = False, + urgent: bool = False, now: Optional[float] = None, ) -> bool: if not self.enabled: @@ -40,7 +41,7 @@ def should_interrupt( if (now - self._last_interrupt_ts) < self.cooldown_s: return False words = len(str(user_text or "").split()) - if has_wakeword or words >= self.min_words: + if has_wakeword or urgent or words >= self.min_words: self._last_interrupt_ts = now return True return False diff --git a/modules/autonomy/services/behavior_composer.py b/modules/autonomy/services/behavior_composer.py index a56f0d09..928a4419 100644 --- a/modules/autonomy/services/behavior_composer.py +++ b/modules/autonomy/services/behavior_composer.py @@ -159,9 +159,13 @@ def execute_plan(self, plan: Dict[str, Any]) -> Dict[str, Any]: if say and self.brain: text = say.get("text") if isinstance(say, dict) else str(say) tone = say.get("tone_hint") if isinstance(say, dict) else None + language = say.get("language") if isinstance(say, dict) else None if text and hasattr(self.brain, "_speak_with_mood"): try: - self.brain._speak_with_mood(text, emotion=tone) + speech_kwargs = {"emotion": tone} + if language: + speech_kwargs["language"] = str(language) + self.brain._speak_with_mood(text, **speech_kwargs) results["components"]["say"] = {"text": text, "tone": tone} except Exception as exc: logger.warning("BehaviorComposer speech dispatch failed: %s", exc) diff --git a/modules/autonomy/services/brain_init.py b/modules/autonomy/services/brain_init.py index 9ccc6873..5ed5a61c 100644 --- a/modules/autonomy/services/brain_init.py +++ b/modules/autonomy/services/brain_init.py @@ -49,11 +49,19 @@ def _init_components(self, config: dict[str, Any]) -> None: self.thread: Optional[threading.Thread] = None self._agentic_decision_in_progress: bool = False self._agentic_decision_lock = threading.Lock() + self._laya_vision_event_lock = threading.Lock() self._worker_executor: Optional[ThreadPoolExecutor] = ThreadPoolExecutor( max_workers=2, thread_name_prefix="autonomy_worker" ) self.mood = MoodManager(config) + self.laya_engine = None + try: + from modules.agent_core.services.laya_engine import LayaEngine + self.laya_engine = LayaEngine.get_instance(config) + logger.info("AutonomyBrain: LayaEngine singleton attached (enabled=%s)", self.laya_engine.enabled) + except Exception as exc: + logger.info("AutonomyBrain: LayaEngine not available: %s", exc) self.appraisal = AffectiveAppraisal(config) self.client = ServiceClient(config.get("endpoints", {}), config=config) self.expression = ExpressionDirector(self.client) @@ -246,6 +254,8 @@ def _init_agent_core(self) -> None: agent_cfg = load_agent_core_config() self.agent = AgentOrchestrator(agent_cfg, autonomy_client=self.client) + # Agent streaming shares the local mood snapshot without an HTTP round trip. + self.agent.mood_manager = self.mood if hasattr(self, "goal_executor") and hasattr(self.goal_executor, "set_decision_traces"): traces = getattr(self.agent, "decision_traces", None) if traces is not None: diff --git a/modules/autonomy/services/brain_parts/decision.py b/modules/autonomy/services/brain_parts/decision.py index f09ed9fc..62c68641 100644 --- a/modules/autonomy/services/brain_parts/decision.py +++ b/modules/autonomy/services/brain_parts/decision.py @@ -82,6 +82,9 @@ def _think(self) -> None: self._sync_emotion() self._liveliness_tick(now) + # ── Laya System 1 reflexes ────────────────────────────────────── + self._think_laya_reflexes() + self._update_companion_needs(now) self._maybe_tick_companion_life_loop(now) @@ -121,6 +124,174 @@ def _think(self) -> None: self._run_companion_rituals(now) self._run_companion_proactive(now) + def _think_laya_reflexes(self) -> None: + """Process the latest speech/input through Laya System 1 for reflexive actions.""" + laya = getattr(self, "laya_engine", None) + if laya is None or not getattr(laya, "enabled", False) or not getattr(laya, "_is_available", False): + return + + last_speech = self.state.get("last_speech_text", "") + if not last_speech: + return + + # Prevent re-processing the same utterance + processed_marker = self.state.get("_laya_last_processed", "") + if processed_marker == last_speech: + return + self.state["_laya_last_processed"] = last_speech + + try: + # Get visual context if available + visual_ctx = None + if hasattr(self, "scene_register") and self.scene_register is not None: + try: + visual_ctx = self.scene_register.get_scene_summary() + except Exception: + pass + + decision = laya.decide_with_context(last_speech, visual_context=visual_ctx) + if decision is None: + return + + # 1. Emergency → SpinalCord reflex + if laya.is_emergency_decision(decision): + spinal = getattr(self, "spinal_cord", None) + if spinal and hasattr(spinal, "observe_voice_emergency"): + spinal.observe_voice_emergency(decision) + else: + # Fallback: direct emergency stop via client + try: + self.client.push_interaction_event("motor.stop", {"priority": "emergency", "source": "laya"}) + except Exception: + pass + + # 2. Affective event → mood deltas (config-scaled) + deltas = laya.get_mood_deltas(decision) + for axis, delta in deltas.items(): + if axis.endswith("_satisfy"): + real_axis = axis.replace("_satisfy", "") + if hasattr(self.mood, "satisfy_need"): + self.mood.satisfy_need(real_axis, abs(delta)) + else: + self.mood.modify(axis, delta) + + # 3. Camera target → curiosity boost + if decision.target_module == "camera": + self.mood.modify("curiosity", 15) + self.memory.add_event("Laya detected a camera-related request, boosting curiosity.") + + # 4. Mark urgency in state for other components + self.state["laya_last_urgency"] = decision.urgency_score + is_urgent = getattr(laya, "is_urgent", None) + if callable(is_urgent): + high_urgency = bool(is_urgent(decision)) + else: + laya_cfg = self.config.get("tri_layer", {}).get("laya", {}) + urgency_cfg = laya_cfg.get("urgency", {}) if isinstance(laya_cfg, dict) else {} + threshold = float(urgency_cfg.get("high_threshold", 1.5)) + high_urgency = float(decision.urgency_score) >= threshold + self.state["laya_last_high_urgency"] = high_urgency + self.state["laya_last_decision_ts"] = time.time() + self.state["laya_last_target"] = decision.target_module + self.state["laya_last_affect"] = decision.affective_event + + except Exception as exc: + logger.debug("Laya reflex processing failed: %s", exc) + + def handle_laya_vision_event(self, event_type: str, payload: Dict[str, Any]) -> bool: + """Use confident face-emotion events as bounded Laya affect inputs.""" + laya = getattr(self, "laya_engine", None) + if laya is None or not getattr(laya, "enabled", False) or not getattr(laya, "_is_available", False): + return False + tri_cfg = self.config.get("tri_layer", {}) if isinstance(self.config.get("tri_layer"), dict) else {} + laya_cfg = tri_cfg.get("laya", {}) if isinstance(tri_cfg.get("laya"), dict) else {} + affect_cfg = laya_cfg.get("affective", {}) if isinstance(laya_cfg.get("affective"), dict) else {} + if not affect_cfg.get("vision_events", True) or not affect_cfg.get("face_emotion_events", True): + return False + event_name = str(event_type or "").strip().lower() + if event_name not in {"face_emotion", "owner_seen", "new_person", "hazard_detected", "scene_changed"}: + return False + if not isinstance(payload, dict): + return False + + min_confidence = float(affect_cfg.get("vision_event_min_confidence", 0.80)) + is_face_event = event_name == "face_emotion" + try: + confidence = float(payload.get("confidence", 0.0 if is_face_event else 1.0)) + except (TypeError, ValueError): + return False + emotion = str(payload.get("emotion", "")).strip().lower() + if confidence < min_confidence or (is_face_event and emotion in {"", "neutral"}): + return False + person = str(payload.get("name", "unknown") or "unknown").strip().lower() + cooldown_s = max(0.0, float(affect_cfg.get("vision_event_cooldown_s", 8.0))) + now = time.time() + marker = f"{event_name}:{person}:{emotion}" + event_lock = getattr(self, "_laya_vision_event_lock", None) + if event_lock is None: + event_lock = threading.Lock() + self._laya_vision_event_lock = event_lock + with event_lock: + recent = self.state.get("_laya_vision_event_times", {}) + last_seen = float(recent.get(marker, 0.0)) if isinstance(recent, dict) else 0.0 + if now - last_seen < cooldown_s: + return False + recent = dict(recent) if isinstance(recent, dict) else {} + recent[marker] = now + self.state["_laya_vision_event_times"] = recent + + text = self._laya_vision_event_text(event_name, payload, emotion) + try: + visual_context = payload.get("context") + if isinstance(visual_context, dict): + visual_context = visual_context.get("summary", "") + if not visual_context: + visual_context = f"yüz ifadesi: {emotion}" if is_face_event else text + decision = laya.decide_with_context(text, visual_context=str(visual_context)) + if decision is None: + return False + self.state["laya_last_vision_decision"] = { + "event_type": event_name, + "target": decision.target_module, + "target_confidence": decision.module_confidence, + "urgency": decision.urgency_score, + "affect": decision.affective_event, + "affect_confidence": decision.affect_confidence, + "timestamp": now, + } + if is_face_event: + affect_threshold = float(affect_cfg.get("vision_event_min_affect_confidence", min_confidence)) + if decision.affective_event not in {"user_praise", "user_rude"}: + return False + if float(decision.affect_confidence) < affect_threshold: + return False + self.mood.apply_affective_event(decision.affective_event) + self.state["laya_last_vision_affect"] = { + "emotion": emotion, + "event": decision.affective_event, + "confidence": decision.affect_confidence, + "timestamp": now, + } + return True + except Exception as exc: + logger.debug("Laya vision affect processing failed: %s", exc) + return False + + @staticmethod + def _laya_vision_event_text(event_type: str, payload: Dict[str, Any], emotion: str = "") -> str: + if event_type == "face_emotion": + return f"Kullanıcının yüz ifadesinde {emotion} duygusu algılandı." + if event_type == "owner_seen": + return f"Kamera sahibimin göründüğünü algıladı: {str(payload.get('name', '') or '')[:40]}" + if event_type == "new_person": + return f"Kamerada yeni bir kişi algılandı: {str(payload.get('name', '') or '')[:40]}" + if event_type == "hazard_detected": + hazards = payload.get("hazards", []) + return f"Görüntüde olası tehlike algılandı: {str(hazards)[:120]}" + context = payload.get("context", {}) + summary = context.get("summary", "") if isinstance(context, dict) else str(context or "") + return f"Görüntüde sahne değişti: {str(summary or payload.get('reason', ''))[:120]}" + def _check_darkness_appraisal(self, now: float) -> None: sleep_cfg = self.config.get("behaviors", {}).get("sleep", {}) if not sleep_cfg.get("enabled"): diff --git a/modules/autonomy/services/brain_parts/perception_context.py b/modules/autonomy/services/brain_parts/perception_context.py index c9e3b656..b6bf326d 100644 --- a/modules/autonomy/services/brain_parts/perception_context.py +++ b/modules/autonomy/services/brain_parts/perception_context.py @@ -99,7 +99,7 @@ def _forward_visual_events_to_agent(self) -> None: composer.check_wake_condition("person_entered", {"count": len(people)}) if hazards: self.client.emit_agent_event("hazard_detected", {"count": len(hazards)}) - self.appraise_event("loud_noise", intensity=min(1.0, len(hazards) / 3.0)) + self.appraise_event("visual_hazard", intensity=min(1.0, len(hazards) / 3.0)) if getattr(self, "agent", None) and self.config.get("llm", {}).get("enabled", False): try: prompt = ( diff --git a/modules/autonomy/services/brain_parts/scenario.py b/modules/autonomy/services/brain_parts/scenario.py index a0b04461..f4efaacc 100644 --- a/modules/autonomy/services/brain_parts/scenario.py +++ b/modules/autonomy/services/brain_parts/scenario.py @@ -133,6 +133,7 @@ def _update_companion_needs(self, now: float) -> None: snapshot["perception"] = self.get_canonical_perception_context(snapshot.get("perception", {})) snapshot["capability_health"] = self.get_capability_health_snapshot() + self._attach_laya_urgency_context(snapshot) outcome = self.state.get("companion_outcome") if isinstance(outcome, dict): snapshot["companion_outcome"] = dict(outcome) @@ -167,6 +168,20 @@ def _update_companion_needs(self, now: float) -> None: except Exception as exc: logger.debug("Companion needs update failed: %s", exc) + def _attach_laya_urgency_context(self, snapshot: Dict[str, Any]) -> Dict[str, Any]: + if not isinstance(snapshot, dict): + return snapshot + companion_cfg = self.config.get("companion_goals", {}) + laya_cfg = companion_cfg.get("laya_urgency", {}) if isinstance(companion_cfg, dict) else {} + if not isinstance(laya_cfg, dict) or not bool(laya_cfg.get("enabled", True)): + return snapshot + snapshot["laya_urgency"] = { + "high": bool(self.state.get("laya_last_high_urgency", False)), + "score": float(self.state.get("laya_last_urgency", 0.0) or 0.0), + "timestamp": float(self.state.get("laya_last_decision_ts", 0.0) or 0.0), + } + return snapshot + def _sync_companion_plan_event(self, plan: dict) -> None: if not isinstance(plan, dict): return @@ -325,11 +340,11 @@ def get_needs_snapshot(self) -> dict: data["available"] = True data["recent_reflections"] = recent_reflections data["tool_schemas"] = tool_schemas - return data + return self._attach_laya_urgency_context(data) data = self.tick_living_needs() data["recent_reflections"] = recent_reflections data["tool_schemas"] = tool_schemas - return data + return self._attach_laya_urgency_context(data) if hasattr(self, "needs_engine"): current = self.state.get("companion_needs") if isinstance(current, dict) and current: @@ -337,11 +352,11 @@ def get_needs_snapshot(self) -> dict: data["available"] = True data["recent_reflections"] = recent_reflections data["tool_schemas"] = tool_schemas - return data + return self._attach_laya_urgency_context(data) data = self.needs_engine.snapshot() data["recent_reflections"] = recent_reflections data["tool_schemas"] = tool_schemas - return data + return self._attach_laya_urgency_context(data) except Exception as exc: return {"ok": False, "available": False, "error": str(exc)} return {"ok": False, "available": False, "reason": "needs_engine_missing"} diff --git a/modules/autonomy/services/brain_parts/vocal.py b/modules/autonomy/services/brain_parts/vocal.py index 74ef51ec..224e4b21 100644 --- a/modules/autonomy/services/brain_parts/vocal.py +++ b/modules/autonomy/services/brain_parts/vocal.py @@ -24,8 +24,21 @@ class VocalMixin(VocalProsodyMixin): def _handle_barge_in_and_wakeword(self, text: str, low: str, has_wake: bool, request_id: str) -> bool: from modules.voice.speech.services.wake_phrase import contains_wakeword, strip_wakewords + robot_speaking = self._robot_is_speaking() + urgent = False + laya = getattr(self, "laya_engine", None) + if robot_speaking and laya is not None and getattr(laya, "_is_available", False): + try: + decision = laya.decide(text) + urgent = bool(decision and laya.is_emergency_decision(decision)) + except Exception as exc: + logger.debug("Laya urgency check for barge-in failed: %s", exc) + if self.barge_in.should_interrupt( - robot_speaking=self._robot_is_speaking(), user_text=text, has_wakeword=has_wake + robot_speaking=robot_speaking, + user_text=text, + has_wakeword=has_wake, + urgent=urgent, ): self._barge_in_stop_speaking() elif has_wake: diff --git a/modules/autonomy/services/companion_goal_selector.py b/modules/autonomy/services/companion_goal_selector.py index e7bf1e5d..0ebeb159 100644 --- a/modules/autonomy/services/companion_goal_selector.py +++ b/modules/autonomy/services/companion_goal_selector.py @@ -100,6 +100,27 @@ def select( scores["safety"] = max(0.0, _as_float(scores.get("safety")) + _as_float(adjustment)) snap["scores"] = scores + urgency_cfg = _as_dict(self.cfg.get("laya_urgency")) + laya_urgency = _as_dict(snap.get("laya_urgency")) + urgency_ts = _as_float(laya_urgency.get("timestamp"), 0.0) + freshness_s = max(0.0, _as_float(urgency_cfg.get("freshness_s"), 8.0)) + urgency_score = _as_float(laya_urgency.get("score"), 0.0) + urgency_age = ts - urgency_ts + urgency_override = ( + bool(urgency_cfg.get("enabled", True)) + and bool(laya_urgency.get("high", False)) + and urgency_ts > 0.0 + and 0.0 <= urgency_age <= freshness_s + ) + if urgency_override: + safety_floor = max(0.0, min(1.0, _as_float(urgency_cfg.get("safety_score_floor"), 0.75))) + scores["safety"] = max(safety_floor, _as_float(scores.get("safety"), 0.0)) + dominant = "safety" + recommended = "pause_and_observe" + snap["dominant_need"] = dominant + snap["recommended_goal"] = recommended + snap["scores"] = scores + formation_decision = None if self.goal_formation is not None and bool(_as_dict(self.cfg.get("goal_formation")).get("enabled", True)): formation_decision = self.goal_formation.form( @@ -179,6 +200,10 @@ def select( "formation_disposition": ( formation_decision.disposition.value if formation_decision is not None else "legacy" ), + "laya_urgency_override": ( + {"score": urgency_score, "age_s": round(max(0.0, urgency_age), 3)} + if urgency_override else None + ), } if not out["deferred"]: out = self._apply_environment_policy(out, snapshot=snap, owner_present=owner_present, timestamp=ts) diff --git a/modules/autonomy/services/mood.py b/modules/autonomy/services/mood.py index 6756ef3d..87aeb3a1 100644 --- a/modules/autonomy/services/mood.py +++ b/modules/autonomy/services/mood.py @@ -125,6 +125,58 @@ def modify(self, mood, delta): if mood in self.state: self.state[mood] = max(0, min(100, self.state[mood] + delta)) self._maybe_snapshot() + + def apply_affective_event(self, event_name: str) -> str: + """Apply a System 1 Laya affective event directly to internal mood state.""" + event = str(event_name or "").strip().lower() + with self._lock: + if event == "user_praise": + self.modify("happiness", 25) + self.modify("anger", -15) + self.modify("social", 20) + elif event == "user_rude": + self.modify("anger", 30) + self.modify("happiness", -25) + self.modify("fear", 10) + return self.get_dominant_emotion() + + def apply_laya_decision(self, decision) -> str: + """Apply a full LayaDecision object to mood state. + + Processes affective event, urgency impact, direct command stimulation, + and social chat satisfaction in one atomic operation. + + Args: + decision: LayaDecision instance from laya_engine.py + Returns: + Dominant emotion string after update + """ + with self._lock: + # 1. Affective event + affect = getattr(decision, "affective_event", "neutral") + if affect and affect != "neutral": + self.apply_affective_event(affect) + + # 2. Urgency → fear/energy impact + urgency = float(getattr(decision, "urgency_score", 0.0)) + if urgency >= 2.0: + self.modify("fear", min(40, urgency * 15)) + self.modify("energy", -10) + elif urgency >= 1.0: + self.modify("curiosity", 10) + + # 3. Direct command → stimulation satisfaction + if getattr(decision, "is_direct_command", False): + self.satisfy_need("stimulation", 15) + + # 4. Conversational target → social satisfaction + target = getattr(decision, "target_module", "") + if target == "system2_chat": + self.satisfy_need("social", 20) + + self._maybe_snapshot() + + return self.get_dominant_emotion() def get_dominant_emotion(self): # Determine the dominant emotion for LEDs / eyes / body language. diff --git a/modules/autonomy/services/spinal_cord_reflex.py b/modules/autonomy/services/spinal_cord_reflex.py index c03175bc..a61f6d0a 100644 --- a/modules/autonomy/services/spinal_cord_reflex.py +++ b/modules/autonomy/services/spinal_cord_reflex.py @@ -75,3 +75,50 @@ def observe_hardware_event(self, event_type: str, payload: Dict[str, Any]) -> bo logger.error(f"Error executing spinal cord reflex: {exc}") return False + + def observe_voice_emergency(self, laya_decision) -> bool: + """Trigger spinal cord reflex from a Laya emergency voice decision. + + This bridges voice-based emergency detection (e.g. "Dur!") to the same + hardware reflex system used for cliff/impact events. + + Args: + laya_decision: LayaDecision instance from laya_engine.py + Returns: + True if emergency reflex was triggered + """ + if not self.enabled: + return False + if not hasattr(laya_decision, 'is_emergency') or not getattr(laya_decision, 'is_emergency', False): + return False + + import time + now = time.time() + if now - self._last_reflex_ts < self.reflex_cooldown_s: + return False + + urgency = getattr(laya_decision, 'urgency_score', 0.0) + target = getattr(laya_decision, 'target_module', 'unknown') + logger.warning( + "SPINAL CORD REFLEX: Voice emergency detected via Laya! target=%s urgency=%.1f", + target, urgency, + ) + + try: + self.client.push_interaction_event("motor.stop", {"priority": "emergency", "source": "laya_voice"}) + except Exception as exc: + logger.error("Failed to send emergency stop from voice: %s", exc) + + self._last_reflex_ts = now + + try: + self.memory.add_event( + f"Voice emergency detected: urgency={urgency:.1f}, target={target}. " + "Motor stopped via spinal cord reflex." + ) + self.client.update_emotions(["surprise"]) + self.client.push_interaction_event("appraisal:shocked") + except Exception as exc: + logger.error("Failed to notify higher brain of voice emergency: %s", exc) + + return True diff --git a/modules/autonomy/services/system1_reflex.py b/modules/autonomy/services/system1_reflex.py index 45411389..a4a5acf8 100644 --- a/modules/autonomy/services/system1_reflex.py +++ b/modules/autonomy/services/system1_reflex.py @@ -1,17 +1,25 @@ -"""System 1 Reflex Engine (İki Katmanlı Zihin - Hızlı Beden Dili ve Düşünme Refleksi). +"""System 1 Reflex Engine — LayaEngine Proxy (Backward Compatible). -Uzak PC'deki Ollama modelinden derin akıl yürütme beklenirken (System 2) robot -asla donup kalmaz. Milisaniyeler içinde (0-50ms) beden dili mikro-eylemleri üretir: -1. Gözler/Kafa hafif yukarı-sağa düşünme açısına kayar. -2. Kulak hafif seğirir. -3. Uzun süren sorgularda (örn >1.5s) doğal Türkçe mikro-onay ("Hmm...", "Hemen bakıyorum...") seslendirir. +This class is maintained for backward compatibility. All core intelligence +is now provided by modules.agent_core.services.laya_engine.LayaEngine. + +Original İki Katmanlı Zihin (Two-Layered Mind) concept is preserved: +1. Millisecond body language reactions while waiting for LLM (System 2). +2. Verbal fillers for long-running queries. + +The proxy transparently delegates to LayaEngine when available, falling back +to the original static behavior when it is not. """ from __future__ import annotations +import logging import random import time -from typing import Any, Dict, List, Optional, Tuple +from typing import Any, Dict, Optional +logger = logging.getLogger("autonomy.system1_reflex") + +# Original constants preserved for backward compatibility THINKING_FILLERS_TR = [ "Hmm...", "Hemen bakıyorum...", @@ -27,23 +35,69 @@ "One second...", ] + class System1ReflexEngine: + """DEPRECATED — Transparent proxy to LayaEngine. + + Preserves the original API surface so existing callers in AutonomyBrain + continue to work. When LayaEngine is available, delegates to it for + mood-aware fillers and affective body language. Falls back to static + behavior otherwise. + """ + def __init__(self, language: str = "tr") -> None: self.language = language + self._laya: Any = None + self._laya_checked = False self._last_filler_ts = 0.0 - self._filler_cooldown_s = 15.0 # Sık tekrarı önle + self._filler_cooldown_s = 15.0 + logger.info("System1ReflexEngine initialized (LayaEngine proxy mode)") + + def _get_laya(self) -> Any: + """Lazy-load the LayaEngine singleton.""" + if self._laya is not None: + return self._laya + if self._laya_checked: + return None + self._laya_checked = True + try: + from modules.agent_core.services.laya_engine import LayaEngine + self._laya = LayaEngine.get_instance() + if self._laya and getattr(self._laya, '_is_available', False): + logger.info("System1ReflexEngine: Successfully connected to LayaEngine") + else: + logger.info("System1ReflexEngine: LayaEngine found but model not loaded yet") + except Exception as exc: + logger.debug("System1ReflexEngine: LayaEngine not available: %s", exc) + return self._laya def get_thinking_body_pose(self) -> Dict[str, Any]: - """Düşünürken doğal insan/robot duruşu.""" + """Return a natural thinking body pose for the robot. + + When LayaEngine is available, uses affective reaction for LED color. + Otherwise falls back to static defaults. + """ + laya = self._get_laya() + led_color = "#30e3ca" + if laya and getattr(laya, '_is_available', False): + try: + reaction = laya.get_affective_reaction("neutral") + led_color = reaction.get("led_color", led_color) + except Exception: + pass + return { - "head": {"pan": random.choice([86, 94]), "tilt": 102}, # Hafif yukarı/yana bakış - "ears": {"left": 80, "right": 70}, # Meraklı kulak açısı + "head": {"pan": random.choice([86, 94]), "tilt": 102}, + "ears": {"left": 80, "right": 70}, "eyes": {"mode": "animation", "name": "thinking"}, - "leds": {"mode": "thinking", "color": "#30e3ca"}, + "leds": {"mode": "thinking", "color": led_color}, } def should_emit_verbal_filler(self, wait_elapsed_s: float) -> bool: - """Eğer çıkarım 1.2 saniyeden uzun sürdüyse ve bekleme süresi dolmadıysa sesli onay üret.""" + """Check if we should emit a verbal filler after waiting. + + Triggers after 1.2s of waiting, with a cooldown to prevent spam. + """ if wait_elapsed_s < 1.2: return False now = time.time() @@ -52,8 +106,30 @@ def should_emit_verbal_filler(self, wait_elapsed_s: float) -> bool: self._last_filler_ts = now return True - def get_verbal_filler(self) -> str: - """Rastgele düşünme ara sesi.""" + def get_verbal_filler(self, mood: str = "neutral") -> str: + """Return a random thinking filler phrase. + + When LayaEngine is available and mood_aware is enabled, uses mood-aware + fillers. Otherwise falls back to static phrase lists. + + Args: + mood: Current dominant emotion (optional, defaults to "neutral") + """ + laya = self._get_laya() + if laya and getattr(laya, '_is_available', False): + try: + if hasattr(laya, 'get_filler_with_affect') and mood != "neutral": + filler = laya.get_filler_with_affect(self.language, mood=mood) + if filler: + return filler + else: + filler = laya.get_instant_filler(self.language) + if filler: + return filler + except Exception: + pass + + # Static fallback if self.language.startswith("en"): return random.choice(THINKING_FILLERS_EN) return random.choice(THINKING_FILLERS_TR) diff --git a/modules/gateway/services/bootstrap_ai.py b/modules/gateway/services/bootstrap_ai.py index 18708d5e..55f00604 100644 --- a/modules/gateway/services/bootstrap_ai.py +++ b/modules/gateway/services/bootstrap_ai.py @@ -426,11 +426,20 @@ def _wire_vlm_autonomy(started: Dict[str, object]) -> None: ): def _forward_vlm_event(event_type: str, data: Dict[str, Any]) -> None: + if event_type in {"face_emotion", "owner_seen", "new_person", "hazard_detected", "scene_changed"} and hasattr(brain, "handle_laya_vision_event"): + try: + executor = getattr(brain, "_worker_executor", None) + if executor is not None: + executor.submit(brain.handle_laya_vision_event, event_type, dict(data)) + else: + brain.handle_laya_vision_event(event_type, data) + except Exception: + logger.debug("Laya vision event dispatch failed", exc_info=True) try: if hasattr(brain, "client") and hasattr(brain.client, "emit_agent_event"): brain.client.emit_agent_event(event_type, data) except Exception: - pass + logger.debug("VLM agent event forwarding failed", exc_info=True) vlm_bridge.event_bus.subscribe_all(_forward_vlm_event) logger.info("vlm event bus -> agent event bridge mounted") diff --git a/modules/vlm_bridge/config/config.yml b/modules/vlm_bridge/config/config.yml index 6164dfa2..7a7b54fa 100644 --- a/modules/vlm_bridge/config/config.yml +++ b/modules/vlm_bridge/config/config.yml @@ -17,6 +17,8 @@ vision: backend: heuristic # none | heuristic | onnx | remote model_path: models/emotion-ferplus.onnx min_confidence: 0.35 + event_min_confidence: 0.80 + event_cooldown_s: 8.0 remote_url: "@gateway/vlm/fer/analyze" remote_timeout_s: 2.5 remote_fallback: heuristic diff --git a/modules/vlm_bridge/services/processor_identity.py b/modules/vlm_bridge/services/processor_identity.py index a90ed8c4..dadd41c7 100644 --- a/modules/vlm_bridge/services/processor_identity.py +++ b/modules/vlm_bridge/services/processor_identity.py @@ -104,9 +104,11 @@ def _process_face_boxes(self, frame: Any, boxes: List[Tuple[int, int, int, int]] name, conf = self._identify_face_in_roi(face_roi) distance = self._estimate_face_distance_m(y2 - y1) emotion = "" + emotion_confidence = 0.0 if self._face_emotion is not None and face_roi is not None and getattr(face_roi, "size", 1): fer = self._face_emotion.estimate(face_roi) emotion = str(fer.get("emotion", "") or "") + emotion_confidence = float(fer.get("confidence", 0.0) or 0.0) tracked = bool(tracked_box is not None and idx == 0) parsed.append( { @@ -116,6 +118,7 @@ def _process_face_boxes(self, frame: Any, boxes: List[Tuple[int, int, int, int]] "distance_m": distance, "name": name, "emotion": emotion, + "emotion_confidence": round(emotion_confidence, 3), "tracked": tracked, } ) @@ -123,6 +126,7 @@ def _process_face_boxes(self, frame: Any, boxes: List[Tuple[int, int, int, int]] current_keys.add(person_key) if self.event_bus is not None and person_key not in self._visible_persons: self.event_bus.publish(EVENT_PERSON_SEEN, {"name": name, "emotion": emotion}) + self._publish_face_emotion_event(person_key, name, emotion, emotion_confidence) self._annotate_face(annotated, x1, y1, x2, y2, name, conf, distance, tracked) if self.event_bus is not None: for key in self._visible_persons - current_keys: @@ -130,6 +134,33 @@ def _process_face_boxes(self, frame: Any, boxes: List[Tuple[int, int, int, int]] self._visible_persons = current_keys return parsed, annotated + def _publish_face_emotion_event(self, person_key: str, name: str, emotion: str, confidence: float) -> None: + if self.event_bus is None or not emotion or emotion.lower() == "neutral": + return + if confidence < getattr(self, "_face_emotion_event_min_confidence", 1.0): + return + now = time.time() + cooldown = float(getattr(self, "_face_emotion_event_cooldown_s", 8.0)) + event_lock = getattr(self, "_face_emotion_event_lock", None) + if event_lock is None: + event_lock = threading.Lock() + self._face_emotion_event_lock = event_lock + with event_lock: + previous_emotion, previous_ts = self._last_face_emotion_event.get(person_key, ("", 0.0)) + if previous_emotion == emotion and now - previous_ts < cooldown: + return + self._last_face_emotion_event[person_key] = (emotion, now) + try: + from .vision_event_bus import EVENT_FACE_EMOTION + self.event_bus.publish(EVENT_FACE_EMOTION, { + "name": name, + "emotion": emotion, + "confidence": round(confidence, 3), + "source": "face_emotion", + }) + except Exception as exc: + logger.debug("face emotion event publish failed: %s", exc) + def _analyze_frame(self, frame: Any, enable_follow: bool) -> Tuple[List[Dict[str, Any]], Any]: tracked_box = None boxes: List[Tuple[int, int, int, int]] = [] diff --git a/modules/vlm_bridge/services/processor_init.py b/modules/vlm_bridge/services/processor_init.py index 13c49156..d9f02bb8 100644 --- a/modules/vlm_bridge/services/processor_init.py +++ b/modules/vlm_bridge/services/processor_init.py @@ -155,9 +155,15 @@ def _init_vision_processor(self, config: Dict[str, Any]) -> None: self.last_alert_announcement = 0.0 self._last_person_greet: Dict[str, float] = {} self._visible_persons: set[str] = set() + self._last_face_emotion_event: Dict[str, tuple[str, float]] = {} + self._face_emotion_event_lock = threading.Lock() fer_cfg = dict(vision_cfg.get("face_emotion", {}) if isinstance(vision_cfg.get("face_emotion"), dict) else {}) fer_cfg.setdefault("gateway_base_url", self._gateway_base) + self._face_emotion_event_cooldown_s = max(0.0, float(fer_cfg.get("event_cooldown_s", 8.0))) + self._face_emotion_event_min_confidence = min( + 1.0, max(0.0, float(fer_cfg.get("event_min_confidence", fer_cfg.get("min_confidence", 0.35)))) + ) self._face_emotion = FaceEmotionEstimator(fer_cfg) if FaceEmotionEstimator is not None else None follow_cfg = vision_cfg.get("follow", {}) if isinstance(vision_cfg.get("follow"), dict) else {} diff --git a/modules/vlm_bridge/services/vision_event_bus.py b/modules/vlm_bridge/services/vision_event_bus.py index 4258875d..30359efd 100644 --- a/modules/vlm_bridge/services/vision_event_bus.py +++ b/modules/vlm_bridge/services/vision_event_bus.py @@ -22,11 +22,12 @@ EVENT_FOLLOW_START = "follow_start" EVENT_FOLLOW_STOP = "follow_stop" EVENT_PERSON_LOST = "person_lost" +EVENT_FACE_EMOTION = "face_emotion" ALL_EVENTS = frozenset({ EVENT_PERSON_SEEN, EVENT_OWNER_SEEN, EVENT_NEW_PERSON, EVENT_HAZARD_DETECTED, EVENT_SCENE_CHANGED, EVENT_VLM_RESULT_READY, - EVENT_FOLLOW_START, EVENT_FOLLOW_STOP, EVENT_PERSON_LOST, + EVENT_FOLLOW_START, EVENT_FOLLOW_STOP, EVENT_PERSON_LOST, EVENT_FACE_EMOTION, }) EventHandler = Callable[[str, Dict[str, Any]], None] @@ -87,5 +88,5 @@ def get_stats(self) -> Dict[str, Any]: "VisionEventBus", "EVENT_PERSON_SEEN", "EVENT_OWNER_SEEN", "EVENT_NEW_PERSON", "EVENT_HAZARD_DETECTED", "EVENT_SCENE_CHANGED", "EVENT_VLM_RESULT_READY", - "EVENT_FOLLOW_START", "EVENT_FOLLOW_STOP", "EVENT_PERSON_LOST", + "EVENT_FOLLOW_START", "EVENT_FOLLOW_STOP", "EVENT_PERSON_LOST", "EVENT_FACE_EMOTION", ] diff --git a/modules/vlm_bridge/services/visual_context.py b/modules/vlm_bridge/services/visual_context.py index 8a4f864c..c8cb4b95 100644 --- a/modules/vlm_bridge/services/visual_context.py +++ b/modules/vlm_bridge/services/visual_context.py @@ -207,6 +207,27 @@ def get_latest_dict(self) -> Dict[str, Any]: "update_count": self._update_count, } + def get_laya_context(self, max_chars: int = 100) -> str: + """Return a compact Turkish scene hint suitable for System 1 input.""" + latest = self.get_latest() + if latest is None: + return "" + parts: List[str] = [] + if latest.people: + parts.append(f"{len(latest.people)} kişi") + parts.extend( + str(obj.get("label", "")).strip() + for obj in latest.objects[:5] + if isinstance(obj, dict) and str(obj.get("label", "")).strip() + ) + parts.extend( + f"tehlike: {hazard.get('label', '')}".strip() + for hazard in latest.hazards[:2] + if isinstance(hazard, dict) and str(hazard.get("label", "")).strip() + ) + summary = ", ".join(parts) or latest.summary + return str(summary or "")[:max(1, int(max_chars))] + def get_history(self, limit: int = 3) -> List[Dict[str, Any]]: with self._lock: items = list(self._history[-limit:]) diff --git a/modules/voice/speech/services/sound_tracking.py b/modules/voice/speech/services/sound_tracking.py index b2de6d7b..7a420be3 100644 --- a/modules/voice/speech/services/sound_tracking.py +++ b/modules/voice/speech/services/sound_tracking.py @@ -16,6 +16,19 @@ class SpeechSoundTrackingMixin: _head_arbiter: Any def track_start(self) -> None: + if getattr(self, "_direction", None) is None: + try: + from modules.voice.speech.services.direction import ArrayGeometry, DirectionEstimator + dir_cfg = self.cfg.get("direction", {}) if isinstance(self.cfg.get("direction"), dict) else {} + geom = ArrayGeometry( + mic_distance_m=float(dir_cfg.get("mic_distance_m", 0.06)), + sound_speed=float(dir_cfg.get("sound_speed", 343.0)), + ) + samplerate = int(self.cfg.get("audio", {}).get("samplerate", 16000)) + self._direction = DirectionEstimator(sample_rate=samplerate, geometry=geom) + self.direction_enabled = True + except Exception as exc: + logger.warning("Failed to initialize DirectionEstimator on track_start: %s", exc) self._tracking = True self._pan.start() diff --git a/modules/voice/speech/xSpeechService.py b/modules/voice/speech/xSpeechService.py index c00e48b5..5c6d583b 100644 --- a/modules/voice/speech/xSpeechService.py +++ b/modules/voice/speech/xSpeechService.py @@ -69,9 +69,18 @@ def __init__(self, config_path: Optional[str] = None): self._stt_input_gain = float(rec_cfg.get("input_gain", 1.0)) # Audio router will provide the stream - # Direction estimator - will be initialized when stream starts - self.direction_enabled = False - self._direction = None + dir_cfg = self.cfg.get("direction", {}) if isinstance(self.cfg.get("direction"), dict) else {} + self.direction_enabled = bool(dir_cfg.get("enabled", False)) + if self.direction_enabled: + from modules.voice.speech.services.direction import ArrayGeometry, DirectionEstimator + geom = ArrayGeometry( + mic_distance_m=float(dir_cfg.get("mic_distance_m", 0.06)), + sound_speed=float(dir_cfg.get("sound_speed", 343.0)), + ) + samplerate = int(self.cfg.get("audio", {}).get("samplerate", 16000)) + self._direction = DirectionEstimator(sample_rate=samplerate, geometry=geom) + else: + self._direction = None self._last_angle = None pt_cfg = self.cfg.get("pan_tilt", {}) diff --git a/tests/modules/agent_core/test_arbiter_chain.py b/tests/modules/agent_core/test_arbiter_chain.py index bb58758d..6f7584a1 100644 --- a/tests/modules/agent_core/test_arbiter_chain.py +++ b/tests/modules/agent_core/test_arbiter_chain.py @@ -42,6 +42,27 @@ def test_vision_arbiter_blocks_concurrent_vlm_tools(): assert "vision arbiter busy" in result +def test_laya_tool_hint_is_observability_only(): + registry = _build_registry(VisionArbiter()) + events = [] + registry.tools["probe"] = lambda: {"ok": True} + registry.status_hook = events.append + + result = registry.execute("probe", {}, laya_hint="move_head") + + assert '"ok": true' in result + assert events[-1]["laya_hint"] == "move_head" + assert events[-1]["laya_hint_matches"] is False + + +def test_laya_tool_hint_cannot_select_an_unregistered_tool(): + registry = _build_registry(VisionArbiter()) + + result = registry.execute("missing", {}, laya_hint="probe") + + assert "not found" in result + + def test_action_arbiter_handles_new_vision_types(): arbiter = ActionArbiter() captured: Dict[str, Any] = {} diff --git a/tests/modules/agent_core/test_expression_idle_boundary_contract.py b/tests/modules/agent_core/test_expression_idle_boundary_contract.py index 688013c6..9241c4d5 100644 --- a/tests/modules/agent_core/test_expression_idle_boundary_contract.py +++ b/tests/modules/agent_core/test_expression_idle_boundary_contract.py @@ -16,6 +16,10 @@ class _Agent: def __init__(self) -> None: self.is_busy = False + self.semantic_calls = [] + + def step(self, *args, **kwargs): + self.semantic_calls.append((args, kwargs)) class _Client: @@ -110,3 +114,24 @@ def test_idle_behavior_system_contract_start_stop_without_semantic_idle_decision # The heartbeat loop is time-gated at about 15s. Contract test must not force a semantic idle action. time.sleep(0.02) assert isinstance(client.neopixel_calls, list) + + +def test_idle_heartbeat_does_not_run_laya_or_semantic_actions(monkeypatch): + import modules.agent_core.services.idle_behavior as idle_module + + agent = _Agent() + agent.laya_engine = type("Laya", (), {"get_telemetry": lambda _self: (_ for _ in ()).throw(AssertionError("must not query Laya"))})() + client = _Client() + idle = IdleBehaviorSystem(agent, client=client) + idle.running = True + ticks = iter([100.0, 120.0]) + monkeypatch.setattr(idle_module.time, "time", lambda: next(ticks)) + monkeypatch.setattr(idle_module.time, "sleep", lambda _seconds: setattr(idle, "running", False)) + + idle._idle_loop() + + assert len(client.neopixel_calls) == 1 + args, kwargs = client.neopixel_calls[0] + assert args == ("BREATHE",) + assert kwargs["emotions"] == ["neutral"] + assert agent.semantic_calls == [] diff --git a/tests/modules/agent_core/test_laya_action_contract.py b/tests/modules/agent_core/test_laya_action_contract.py new file mode 100644 index 00000000..abf5c1a0 --- /dev/null +++ b/tests/modules/agent_core/test_laya_action_contract.py @@ -0,0 +1,192 @@ +"""Safety tests for converting Laya's typed choices into inert proposals.""" + +import pytest + +from modules.agent_core.services.laya_action_contract import ( + LIGHT_COLORS, + LIGHT_EFFECTS, + bind_laya_action_proposal, + build_laya_action_proposal, + get_laya_action_questions, + validate_laya_action_proposal, +) +from modules.agent_core.services.laya_engine import LayaEngine + + +POLICY = { + "enabled": True, + "confidence_threshold": 0.85, + "allowed_actions": ["set_lights"], + "default_light_effect": "BREATHE", +} +ANSWERS = { + "action_color": {"type": "choice", "choice": "blue", "confidence": 0.96}, + "action_effect": {"type": "choice", "choice": "unspecified", "confidence": 0.91}, +} + + +def _proposal(**overrides): + values = { + "target_module": "neopixel", + "module_confidence": 0.95, + "is_direct_command": True, + "direct_confidence": 0.96, + "answers": ANSWERS, + "policy": POLICY, + } + values.update(overrides) + return build_laya_action_proposal(**values) + + +def test_action_questions_are_only_finite_choice_fields(): + questions = get_laya_action_questions() + + assert set(questions) == {"action_color", "action_effect"} + assert all(question["type"] == "choice" for question in questions.values()) + assert set(questions["action_color"]["criteria"]) == {*LIGHT_COLORS, "unspecified"} + assert set(questions["action_effect"]["criteria"]) == {*LIGHT_EFFECTS, "unspecified"} + + +def test_valid_laya_light_choices_create_inert_typed_proposal(): + proposal = _proposal() + + assert proposal == { + "schema_version": 1, + "tool_name": "set_lights", + "arguments": {"effect": "BREATHE", "color": "blue"}, + "confidence": 0.91, + } + + +def test_proposal_binds_to_current_request_and_validates_without_dispatch(): + bound = bind_laya_action_proposal(_proposal(), "request-123") + + assert bound["request_id"] == "request-123" + assert validate_laya_action_proposal( + bound, expected_request_id="request-123", policy=POLICY + ) + assert not validate_laya_action_proposal( + bound, expected_request_id="request-456", policy=POLICY + ) + + +@pytest.mark.parametrize( + "mutation", + [ + lambda proposal: proposal.update(extra="unexpected"), + lambda proposal: proposal.update(schema_version=True), + lambda proposal: proposal.update(confidence=0.84), + lambda proposal: proposal["arguments"].update(color="ultraviolet"), + lambda proposal: proposal["arguments"].update(arbitrary=1), + ], +) +def test_request_bound_proposal_rejects_schema_and_policy_violations(mutation): + proposal = bind_laya_action_proposal(_proposal(), "request-123") + mutation(proposal) + + assert not validate_laya_action_proposal( + proposal, expected_request_id="request-123", policy=POLICY + ) + + +def test_proposal_binding_rejects_missing_or_conflicting_request_ids(): + proposal = _proposal() + + assert bind_laya_action_proposal(proposal, " ") is None + assert bind_laya_action_proposal({**proposal, "request_id": "older"}, "current") is None + + +def test_proposal_validator_fails_closed_when_feature_or_action_is_unapproved(): + bound = bind_laya_action_proposal(_proposal(), "request-123") + + assert not validate_laya_action_proposal( + bound, expected_request_id="request-123", policy={**POLICY, "enabled": False} + ) + assert not validate_laya_action_proposal( + bound, expected_request_id="request-123", policy={**POLICY, "allowed_actions": []} + ) + assert not validate_laya_action_proposal( + bound, expected_request_id="request-123", policy={**POLICY, "confidence_threshold": float("nan")} + ) + + +@pytest.mark.parametrize( + "overrides", + [ + {"policy": {**POLICY, "enabled": False}}, + {"policy": {**POLICY, "allowed_actions": []}}, + {"target_module": "arduino_serial"}, + {"is_direct_command": False}, + {"module_confidence": 0.84}, + {"direct_confidence": 0.84}, + {"answers": {**ANSWERS, "action_color": {"type": "choice", "choice": "unspecified", "confidence": 0.99}}}, + {"answers": {**ANSWERS, "action_color": {"type": "choice", "choice": "javascript", "confidence": 0.99}}}, + {"answers": {**ANSWERS, "action_effect": {"type": "choice", "choice": "DELETE_ALL", "confidence": 0.99}}}, + {"answers": {**ANSWERS, "action_color": {"type": "score", "choice": "blue", "confidence": 0.99}}}, + {"policy": {**POLICY, "default_light_effect": "MOVE_MOTOR"}}, + ], +) +def test_invalid_or_unapproved_action_proposals_fail_closed(overrides): + assert _proposal(**overrides) is None + + +def test_invalid_confidence_values_fail_closed(): + bad_answers = { + **ANSWERS, + "action_color": {"type": "choice", "choice": "blue", "confidence": float("nan")}, + } + assert _proposal(answers=bad_answers) is None + assert _proposal(module_confidence=True) is None + + +def test_laya_engine_requests_action_choices_only_when_opted_in(): + answers = { + "target_module": {"choice": "neopixel", "confidence": 0.95}, + "is_direct_command": {"choice": "direct_action", "confidence": 0.96}, + "affective_event": {"choice": "neutral", "confidence": 0.9}, + "urgency": {"score": 0.4, "confidence": 0.9}, + **ANSWERS, + } + engine = LayaEngine(laya_cfg={"action_proposals": POLICY}) + engine._agent = type("MockLaya", (), { + "system_one": lambda _self, _state, questions: { + "answers": answers, + "question_ids": list(questions), + }, + })() + engine._is_available = True + engine._load_attempted = True + + decision = engine.decide("Işıkları mavi yap") + + assert decision is not None + assert decision.suggested_action == _proposal() + + +def test_action_proposal_feature_defaults_off(): + engine = LayaEngine() + captured = {} + + class MockLaya: + def system_one(self, _state, questions): + captured.update(questions) + return {"answers": { + "target_module": {"choice": "neopixel", "confidence": 0.95}, + "is_direct_command": {"choice": "direct_action", "confidence": 0.96}, + "affective_event": {"choice": "neutral", "confidence": 0.9}, + "urgency": {"score": 0.4, "confidence": 0.9}, + **ANSWERS, + }} + + engine._agent = MockLaya() + engine._is_available = True + engine._load_attempted = True + decision = engine.decide("Işıkları mavi yap") + + assert decision is not None and decision.suggested_action is None + assert "action_color" not in captured + assert "action_effect" not in captured + + +def test_emergency_safety_bypass_defaults_off(): + assert LayaEngine().emergency_bypass_safety is False diff --git a/tests/modules/agent_core/test_laya_agent_fast_path.py b/tests/modules/agent_core/test_laya_agent_fast_path.py new file mode 100644 index 00000000..1f4c54df --- /dev/null +++ b/tests/modules/agent_core/test_laya_agent_fast_path.py @@ -0,0 +1,72 @@ +"""Agent adapter contract for Laya System 1 fast-path decisions.""" + +from types import SimpleNamespace +from unittest.mock import MagicMock + +from modules.agent_core.services.agent import AgentOrchestrator +from modules.agent_core.services.laya_action_contract import validate_laya_action_proposal + + +def _agent_with_laya(decision, *, is_fast): + agent = AgentOrchestrator.__new__(AgentOrchestrator) + agent.fast_path_enabled = True + agent.fast_path_max_chars = 200 + agent.world_state = MagicMock() + agent.world_state.get_state.return_value = {"battery_percent": 80} + agent.world_state.get_laya_scene_summary.return_value = "desk and lamp" + agent.tool_registry = MagicMock() + agent.laya_engine = MagicMock() + agent.laya_engine.enabled = True + agent.laya_engine.should_fast_path.return_value = (is_fast, decision) + return agent + + +def test_agent_caches_laya_fast_path_decision_without_dispatching_tool(): + decision = SimpleNamespace(is_direct_command=True, suggested_tool="neopixel") + agent = _agent_with_laya(decision, is_fast=True) + + assert agent._should_fast_path("Işıkları mavi yap") + + assert agent._last_laya_fast_path_decision is decision + agent.laya_engine.should_fast_path.assert_called_once_with( + "Işıkları mavi yap", + world_state={"battery_percent": 80}, + visual_context="desk and lamp", + ) + agent.tool_registry.execute.assert_not_called() + + +def test_agent_keeps_laya_conversation_on_system2_even_when_prompt_is_short(): + decision = SimpleNamespace(is_direct_command=False, suggested_tool=None) + agent = _agent_with_laya(decision, is_fast=False) + + assert not agent._should_fast_path("Nasılsın?") + + assert agent._last_laya_fast_path_decision is decision + agent.laya_engine.should_fast_path.assert_called_once() + agent.tool_registry.execute.assert_not_called() + + +def test_agent_binds_laya_proposal_to_internal_request_not_trace_metadata(): + decision = SimpleNamespace(suggested_action={ + "schema_version": 1, + "tool_name": "set_lights", + "arguments": {"color": "blue", "effect": "SOLID"}, + "confidence": 0.94, + }) + request_id = "internal-turn-uuid" + caller_trace_id = "caller-controlled-trace" + + AgentOrchestrator._bind_laya_action_request(decision, request_id) + + assert decision.suggested_action["request_id"] == request_id + assert decision.suggested_action["request_id"] != caller_trace_id + assert validate_laya_action_proposal( + decision.suggested_action, + expected_request_id=request_id, + policy={ + "enabled": True, + "allowed_actions": ["set_lights"], + "confidence_threshold": 0.85, + }, + ) diff --git a/tests/modules/agent_core/test_laya_engine.py b/tests/modules/agent_core/test_laya_engine.py new file mode 100644 index 00000000..78f58663 --- /dev/null +++ b/tests/modules/agent_core/test_laya_engine.py @@ -0,0 +1,471 @@ +from unittest.mock import MagicMock +from modules.agent_core.services.laya_engine import LayaEngine, LayaDecision +from modules.agent_core.services.tri_layer import TriLayerRouter, build_subagent_profiles + + +def _fresh_engine(**kwargs) -> LayaEngine: + """Creates a fresh LayaEngine instance (bypasses singleton for test isolation).""" + defaults = {"enabled": True, "confidence_threshold": 0.60} + defaults.update(kwargs) + return LayaEngine(**defaults) + + +def _mock_engine_with_agent(answers: dict, **kwargs) -> LayaEngine: + """Creates a LayaEngine with a mocked Laya agent returning given answers.""" + engine = _fresh_engine(**kwargs) + mock_agent = MagicMock() + mock_agent.system_one.return_value = {"answers": answers} + engine._agent = mock_agent + engine._is_available = True + engine._load_attempted = True + return engine + + +# --- LayaDecision dataclass tests --- + +def test_laya_decision_properties(): + decision = LayaDecision( + target_module="emergency_stop", + module_confidence=0.95, + is_direct_command=True, + direct_confidence=0.99, + affective_event="neutral", + affect_confidence=0.90, + urgency_score=2.8, + urgency_confidence=0.92, + inference_ms=35.2, + ) + assert decision.is_emergency is True + assert decision.is_direct_command is True + assert decision.target_module == "emergency_stop" + + +def test_laya_decision_not_emergency(): + decision = LayaDecision( + target_module="speak", + module_confidence=0.80, + is_direct_command=True, + direct_confidence=0.85, + affective_event="neutral", + affect_confidence=0.90, + urgency_score=0.5, + urgency_confidence=0.70, + inference_ms=12.0, + ) + assert decision.is_emergency is False + + +def test_laya_decision_uses_configured_emergency_threshold(): + decision = LayaDecision( + target_module="arduino_serial", + module_confidence=0.9, + is_direct_command=True, + direct_confidence=0.9, + affective_event="neutral", + affect_confidence=0.9, + urgency_score=2.2, + urgency_confidence=0.9, + inference_ms=1.0, + emergency_threshold=2.5, + ) + assert decision.is_emergency is False + + +def test_low_confidence_lights_off_false_positive_is_not_emergency(): + decision = LayaDecision( + target_module="emergency_stop", + module_confidence=0.8273, + is_direct_command=True, + direct_confidence=0.8297, + affective_event="neutral", + affect_confidence=0.5, + urgency_score=2.059, + urgency_confidence=0.541, + inference_ms=30.0, + ) + + assert decision.is_emergency is False + + +def test_confident_direct_emergency_target_works_with_low_score_confidence(): + decision = LayaDecision( + target_module="emergency_stop", + module_confidence=0.934, + is_direct_command=True, + direct_confidence=0.9818, + affective_event="neutral", + affect_confidence=0.5, + urgency_score=2.2406, + urgency_confidence=0.1849, + inference_ms=30.0, + ) + + assert decision.is_emergency is True + + +# --- Filler and acknowledgment tests --- + +def test_laya_engine_instant_fillers_and_acks(): + engine = _fresh_engine() + # First call should return a non-empty filler + filler_tr = engine.get_instant_filler("tr") + assert isinstance(filler_tr, str) and len(filler_tr) > 0 + + # Second call within cooldown should return empty (cooldown active) + filler_again = engine.get_instant_filler("tr") + assert filler_again == "" + + # Reset cooldown for next test + engine._last_filler_ts = 0.0 + filler_en = engine.get_instant_filler("en") + assert isinstance(filler_en, str) and len(filler_en) > 0 + + # Acks have no cooldown + ack_tr = engine.get_fast_ack("tr") + ack_en = engine.get_fast_ack("en") + assert isinstance(ack_tr, str) and len(ack_tr) > 0 + assert isinstance(ack_en, str) and len(ack_en) > 0 + + +def test_filler_disabled(): + engine = _fresh_engine(laya_cfg={"filler": {"enabled": False}}) + filler = engine.get_instant_filler("tr") + assert filler == "" + + +def test_filler_with_affect_mood_aware(): + engine = _fresh_engine() + # Joy mood + filler_joy = engine.get_filler_with_affect("tr", mood="joy") + assert isinstance(filler_joy, str) and len(filler_joy) > 0 + + engine._last_filler_ts = 0.0 + filler_anger = engine.get_filler_with_affect("tr", mood="anger") + assert isinstance(filler_anger, str) and len(filler_anger) > 0 + + engine._last_filler_ts = 0.0 + filler_sad = engine.get_filler_with_affect("en", mood="sadness") + assert isinstance(filler_sad, str) and len(filler_sad) > 0 + + engine._last_filler_ts = 0.0 + filler_fear = engine.get_filler_with_affect("tr", mood="fear") + assert isinstance(filler_fear, str) and len(filler_fear) > 0 + + +# --- Affective reactions --- + +def test_laya_engine_affective_reactions(): + engine = _fresh_engine() + praise = engine.get_affective_reaction("user_praise") + assert praise["face"] == "happy" + assert praise["emotion"] == "joy" + assert "led_mode" in praise + assert "body_pose" in praise + + rude = engine.get_affective_reaction("user_rude") + assert rude["face"] == "sad" + assert rude["emotion"] == "sorrow" + + neutral = engine.get_affective_reaction("neutral") + assert neutral["face"] == "neutral" + assert neutral["led_mode"] == "static" + + +# --- Mock inference tests --- + +NEOPIXEL_ANSWERS = { + "target_module": {"choice": "neopixel", "confidence": 0.95}, + "is_direct_command": {"choice": "direct_action", "confidence": 0.98}, + "affective_event": {"choice": "neutral", "confidence": 0.90}, + "urgency": {"score": 1.2, "confidence": 0.85}, +} + + +def test_laya_engine_mock_decide_and_fast_path(): + engine = _mock_engine_with_agent(NEOPIXEL_ANSWERS) + + decision = engine.decide("isiklari mavi yap") + assert decision is not None + assert decision.target_module == "neopixel" + assert decision.is_direct_command is True + assert decision.suggested_tool == "set_lights" + + is_fast, dec = engine.should_fast_path("isiklari mavi yap") + assert is_fast is True + assert dec is not None + + +def test_should_fast_path_uses_world_and_visual_context(): + engine = _mock_engine_with_agent(NEOPIXEL_ANSWERS) + is_fast, decision = engine.should_fast_path( + "ışığı değiştir", + world_state={"battery_percent": 12}, + visual_context="masa üzerinde laptop", + ) + + assert is_fast is True + assert decision is not None + sent_text = engine._agent.system_one.call_args[0][0]["text"] + assert "Sahne: masa üzerinde laptop" in sent_text + + +def test_emergency_decision_qualifies_for_fast_path(): + answers = { + "target_module": {"choice": "emergency_stop", "confidence": 0.95}, + "is_direct_command": {"choice": "direct_action", "confidence": 0.96}, + "affective_event": {"choice": "neutral", "confidence": 0.9}, + "urgency": {"score": 2.8, "confidence": 0.95}, + } + fast, decision = _mock_engine_with_agent(answers).should_fast_path("Dur, tehlike!") + assert fast is True + assert decision is not None and decision.is_emergency + + +def test_confident_direct_hardware_command_qualifies_for_fast_path(): + answers = { + "target_module": {"choice": "neopixel", "confidence": 0.9}, + "is_direct_command": {"choice": "direct_action", "confidence": 0.92}, + "affective_event": {"choice": "neutral", "confidence": 0.9}, + "urgency": {"score": 0.2, "confidence": 0.8}, + } + fast, decision = _mock_engine_with_agent(answers).should_fast_path("Işıkları kırmızı yap") + assert fast is True + assert decision is not None and decision.is_direct_command + + +def test_normal_conversation_stays_on_system_two_path(): + answers = { + "target_module": {"choice": "system2_chat", "confidence": 0.95}, + "is_direct_command": {"choice": "conversation", "confidence": 0.96}, + "affective_event": {"choice": "neutral", "confidence": 0.9}, + "urgency": {"score": 0.1, "confidence": 0.9}, + } + fast, decision = _mock_engine_with_agent(answers).should_fast_path("Neden gökyüzü mavi?") + assert fast is False + assert decision is not None and not decision.is_direct_command + + +def test_decide_with_context_adjusts_urgency(): + engine = _mock_engine_with_agent({ + "target_module": {"choice": "speak", "confidence": 0.80}, + "is_direct_command": {"choice": "conversation", "confidence": 0.75}, + "affective_event": {"choice": "neutral", "confidence": 0.90}, + "urgency": {"score": 0.5, "confidence": 0.70}, + }) + + # Low battery → urgency boosted + decision = engine.decide_with_context( + "naber dostum", + world_state={"battery_percent": 10}, + ) + assert decision is not None + assert decision.urgency_score >= engine.high_urgency_threshold + + +def test_decide_with_context_visual_enrichment(): + engine = _mock_engine_with_agent(NEOPIXEL_ANSWERS) + + decision = engine.decide_with_context( + "isiklari degistir", + visual_context="2 kişi, masa, laptop", + ) + assert decision is not None + # Visual context should be passed to the model + call_args = engine._agent.system_one.call_args + assert "Sahne:" in call_args[0][0]["text"] + + +def test_decide_with_context_follow_active_boosts_camera(): + engine = _mock_engine_with_agent({ + "target_module": {"choice": "camera", "confidence": 0.60}, + "is_direct_command": {"choice": "conversation", "confidence": 0.75}, + "affective_event": {"choice": "neutral", "confidence": 0.90}, + "urgency": {"score": 0.5, "confidence": 0.70}, + }) + + decision = engine.decide_with_context( + "etrafa bak", + world_state={"follow_active": True}, + ) + assert decision is not None + assert decision.module_confidence > 0.60 # boosted + + +# --- Telemetry tests --- + +def test_telemetry_tracking(): + engine = _mock_engine_with_agent(NEOPIXEL_ANSWERS) + engine.decide("isiklari kirmizi yap") + engine.decide("isiklari mavi yap") + + telemetry = engine.get_telemetry() + assert telemetry["decision_count"] == 2 + assert telemetry["avg_inference_ms"] >= 0.0 + assert len(telemetry["last_decisions"]) == 2 + assert telemetry["last_target"] == "neopixel" + assert telemetry["available"] is True + assert telemetry["enabled"] is True + + +def test_last_decision_tracked(): + engine = _mock_engine_with_agent(NEOPIXEL_ANSWERS) + engine.decide("test input") + assert engine._last_decision is not None + assert engine._last_decision.target_module == "neopixel" + + +# --- Urgency gate tests --- + +def test_is_urgent_and_emergency(): + engine = _fresh_engine() + + # Not urgent when no decision + assert engine.is_urgent() is False + assert engine.is_emergency_decision() is False + + # High urgency decision + high_urg = LayaDecision( + target_module="speak", module_confidence=0.80, + is_direct_command=False, direct_confidence=0.50, + affective_event="neutral", affect_confidence=0.90, + urgency_score=1.8, urgency_confidence=0.85, inference_ms=10.0, + ) + assert engine.is_urgent(high_urg) is True + assert engine.is_emergency_decision(high_urg) is False + + # Emergency decision + emergency = LayaDecision( + target_module="emergency_stop", module_confidence=0.95, + is_direct_command=True, direct_confidence=0.99, + affective_event="neutral", affect_confidence=0.90, + urgency_score=2.5, urgency_confidence=0.92, inference_ms=8.0, + ) + assert engine.is_urgent(emergency) is True + assert engine.is_emergency_decision(emergency) is True + + +def test_low_confidence_high_urgency_chat_is_not_urgent_or_emergency(): + engine = _fresh_engine() + decision = LayaDecision( + target_module="speak", + module_confidence=0.50, + is_direct_command=False, + direct_confidence=0.59, + affective_event="neutral", + affect_confidence=0.5, + urgency_score=2.4434, + urgency_confidence=0.2844, + inference_ms=30.0, + ) + + assert engine.is_urgent(decision) is False + assert engine.is_emergency_decision(decision) is False + assert engine.get_mood_deltas(decision) == {} + + +# --- Mood deltas tests --- + +def test_get_mood_deltas_praise(): + engine = _fresh_engine() + decision = LayaDecision( + target_module="system2_chat", module_confidence=0.80, + is_direct_command=False, direct_confidence=0.40, + affective_event="user_praise", affect_confidence=0.92, + urgency_score=0.5, urgency_confidence=0.70, inference_ms=15.0, + ) + deltas = engine.get_mood_deltas(decision) + assert deltas["happiness"] > 0 + assert deltas["anger"] < 0 + assert "social_satisfy" in deltas # system2_chat + + +def test_get_mood_deltas_emergency(): + engine = _fresh_engine() + decision = LayaDecision( + target_module="emergency_stop", module_confidence=0.95, + is_direct_command=True, direct_confidence=0.99, + affective_event="neutral", affect_confidence=0.90, + urgency_score=2.5, urgency_confidence=0.92, inference_ms=8.0, + ) + deltas = engine.get_mood_deltas(decision) + assert deltas.get("fear", 0) > 0 + assert deltas.get("energy", 0) < 0 + assert "stimulation_satisfy" in deltas # direct command + + +def test_get_mood_deltas_empty_without_decision(): + engine = _fresh_engine() + deltas = engine.get_mood_deltas() + assert deltas == {} + + +# --- Config-driven settings tests --- + +def test_config_driven_thresholds(): + cfg = { + "fast_path": {"enabled": False}, + "urgency": { + "emergency_threshold": 3.0, + "high_threshold": 2.0, + "high_confidence_threshold": 0.7, + "emergency_target_confidence_threshold": 0.94, + "emergency_direct_confidence_threshold": 0.85, + "emergency_urgency_confidence_threshold": 0.92, + }, + "filler": {"cooldown_s": 5.0, "mood_aware": False}, + "affective": {"mood_impact_scale": 0.5}, + } + engine = _fresh_engine(laya_cfg=cfg) + assert engine.fast_path_enabled is False + assert engine.emergency_threshold == 3.0 + assert engine.high_urgency_threshold == 2.0 + assert engine.high_urgency_confidence_threshold == 0.7 + assert engine.emergency_target_confidence_threshold == 0.94 + assert engine.emergency_direct_confidence_threshold == 0.85 + assert engine.emergency_urgency_confidence_threshold == 0.92 + assert engine.filler_cooldown_s == 5.0 + assert engine.filler_mood_aware is False + assert engine.mood_impact_scale == 0.5 + + +def test_fast_path_disabled_returns_false(): + cfg = {"fast_path": {"enabled": False}} + engine = _mock_engine_with_agent(NEOPIXEL_ANSWERS, laya_cfg=cfg) + engine.fast_path_enabled = False # ensure it's set + is_fast, _ = engine.should_fast_path("isiklari mavi yap") + assert is_fast is False + + +def test_laya_emergency_threshold_is_configured_on_decision(): + engine = _mock_engine_with_agent({ + "target_module": {"choice": "arduino_serial", "confidence": 0.95}, + "is_direct_command": {"choice": "direct_action", "confidence": 0.95}, + "affective_event": {"choice": "neutral", "confidence": 0.9}, + "urgency": {"score": 2.2, "confidence": 0.9}, + }, laya_cfg={"urgency": {"emergency_threshold": 2.5}}) + + decision = engine.decide("dur") + assert decision is not None + assert decision.is_emergency is False + + +# --- TriLayerRouter integration --- + +def test_tri_layer_router_with_laya_integration(): + profiles = build_subagent_profiles() + engine = _mock_engine_with_agent({ + "target_module": {"choice": "arduino_serial", "confidence": 0.92}, + "is_direct_command": {"choice": "direct_action", "confidence": 0.95}, + "affective_event": {"choice": "neutral", "confidence": 0.90}, + "urgency": {"score": 1.5, "confidence": 0.88}, + }) + + router = TriLayerRouter( + profiles=profiles, + max_subagents=2, + default_modules=("autonomy", "agent_core"), + laya_engine=engine, + ) + + routed = router.route("kafanı hafifçe sağa çevir") + assert "arduino_serial" in routed diff --git a/tests/modules/agent_core/test_sensor_loop_budget.py b/tests/modules/agent_core/test_sensor_loop_budget.py index 7c013ac6..ef66bbe0 100644 --- a/tests/modules/agent_core/test_sensor_loop_budget.py +++ b/tests/modules/agent_core/test_sensor_loop_budget.py @@ -1,4 +1,5 @@ import os +import time from modules.agent_core.services.sensor_loop import SensorFeedbackLoop @@ -59,4 +60,34 @@ def test_due_budget_blocks_until_interval(): loop = SensorFeedbackLoop(World(), client=Client()) assert loop._due("x", 5.0, 100.0) is True assert loop._due("x", 5.0, 101.0) is False - assert loop._due("x", 5.0, 106.0) is True \ No newline at end of file + assert loop._due("x", 5.0, 106.0) is True + + +def test_sensor_loop_temporarily_boosts_after_recent_high_urgency(): + class Laya: + enabled = True + high_urgency_threshold = 1.5 + sensor_boost_hz = 5.0 + sensor_boost_hold_s = 8.0 + + @staticmethod + def get_telemetry(): + return {"last_decisions": [{"urgency": 2.0, "timestamp": time.time()}]} + + loop = SensorFeedbackLoop(World(), poll_hz=2.0, laya_engine=Laya()) + assert loop._current_loop_interval() == 0.2 + + +def test_sensor_loop_ignores_stale_high_urgency(): + class Laya: + enabled = True + high_urgency_threshold = 1.5 + sensor_boost_hz = 5.0 + sensor_boost_hold_s = 1.0 + + @staticmethod + def get_telemetry(): + return {"last_decisions": [{"urgency": 2.0, "timestamp": time.time() - 5.0}]} + + loop = SensorFeedbackLoop(World(), poll_hz=2.0, laya_engine=Laya()) + assert loop._current_loop_interval() == loop.loop_interval diff --git a/tests/modules/agent_core/test_speak_tone_forwarding.py b/tests/modules/agent_core/test_speak_tone_forwarding.py index f84d6ac4..8ff6714e 100644 --- a/tests/modules/agent_core/test_speak_tone_forwarding.py +++ b/tests/modules/agent_core/test_speak_tone_forwarding.py @@ -6,6 +6,7 @@ from modules.agent_core.services.action_arbiter import ActionArbiter, ActionRequest from modules.agent_core.services.agent import AgentOrchestrator +from modules.common.latency_trace import latency_trace class _RecordingArbiter: @@ -46,3 +47,49 @@ def test_missing_tone_passes_none(): agent = _agent_with_handlers() agent.action_arbiter.submit(_speak_req({"text": "Merhaba"})) assert agent.speech_arbiter.calls[0]["tone"] is None + + +def test_agent_reads_mood_tone_snapshot(): + agent = AgentOrchestrator.__new__(AgentOrchestrator) + + class Mood: + def get_speech_tone(self): + return {"speed": 1.15, "pitch": 1.1, "emotion": "joy"} + + agent.mood_manager = Mood() + assert agent._current_speech_tone() == {"speed": 1.15, "pitch": 1.1, "emotion": "joy"} + + +def test_agent_tone_snapshot_falls_back_safely(): + agent = AgentOrchestrator.__new__(AgentOrchestrator) + assert agent._current_speech_tone() is None + + class BrokenMood: + def get_speech_tone(self): + raise RuntimeError("mood unavailable") + + agent.mood_manager = BrokenMood() + assert agent._current_speech_tone() is None + + +def test_laya_inference_is_recorded_separately_from_model_ttft(): + agent = AgentOrchestrator.__new__(AgentOrchestrator) + + class Decision: + inference_ms = 17.5 + target_module = "system2_chat" + urgency_score = 0.2 + + agent._last_laya_fast_path_decision = Decision() + trace_id = "test-laya-inference-metric" + latency_trace.ensure(trace_id, {"component": "test"}) + + agent._record_laya_latency(trace_id) + + events = latency_trace.get(trace_id)["events"] + laya_event = next(item for item in events if item["event"] == "laya.inference") + assert laya_event["data"] == { + "inference_ms": 17.5, + "target_module": "system2_chat", + "urgency": 0.2, + } diff --git a/tests/modules/agent_core/test_speech_audio_boundary_contract.py b/tests/modules/agent_core/test_speech_audio_boundary_contract.py index a847867f..369414c2 100644 --- a/tests/modules/agent_core/test_speech_audio_boundary_contract.py +++ b/tests/modules/agent_core/test_speech_audio_boundary_contract.py @@ -12,6 +12,7 @@ SpeechPriority, split_sentences, ) +from modules.agent_core.services.progress import ProgressManager ROOT = Path(__file__).resolve().parents[3] @@ -95,6 +96,44 @@ def test_speech_arbiter_queue_cancel_and_status_contract(): assert arbiter.queue_size() == 0 +def test_laya_filler_priority_is_between_progress_and_final(): + arbiter = SpeechArbiter(max_queue_size=3) + item_id = arbiter.enqueue_laya_filler("Hemen bakıyorum", language="tr") + + assert item_id + queued = arbiter._queue[0] + assert queued.category == "laya_filler" + assert queued.priority > SpeechPriority.PROGRESS + assert queued.priority > SpeechPriority.FINAL_RESPONSE + assert queued.max_age_s == 4.0 + + +def test_progress_routes_laya_ack_through_priority_queue(): + arbiter = SpeechArbiter(max_queue_size=3) + progress = ProgressManager(speech_arbiter=arbiter) + token = progress.new_request(language="tr") + + progress.emit_ack(token, custom_text="Hemen bakıyorum.") + + assert arbiter.queue_size() == 1 + queued = arbiter._queue[0] + assert queued.text == "Hemen bakıyorum." + assert queued.language == "tr" + assert queued.category == "laya_filler" + + +def test_laya_filler_is_selected_before_final_response(): + arbiter = SpeechArbiter(max_queue_size=4) + arbiter.enqueue_progress("İşlem sürüyor", cancel_token="turn") + arbiter.enqueue_final("İşte yanıt", language="tr") + arbiter.enqueue_laya_filler("Hemen bakıyorum", language="tr") + + selected = arbiter._pop_next() + + assert selected is not None + assert selected.category == "laya_filler" + + def test_speech_arbiter_interrupt_contract(): events = [] diff --git a/tests/modules/agent_core/test_streaming_chat.py b/tests/modules/agent_core/test_streaming_chat.py index 35a7f5a0..212eb28a 100644 --- a/tests/modules/agent_core/test_streaming_chat.py +++ b/tests/modules/agent_core/test_streaming_chat.py @@ -36,6 +36,37 @@ def _on_sentence(text: str, idx: int) -> None: assert len(spoken) >= 1 +def test_first_clause_streaming_emits_early(): + agent = AgentOrchestrator.__new__(AgentOrchestrator) + agent.persona_stream_enabled = True + agent.llm_provider = "ollama" + agent._pick_runtime_model = lambda m: m # type: ignore + + # First phrase has a comma after 3 words + chunks = [ + {"message": {"content": "Tabii ki dostum, "}}, + {"message": {"content": "yapay zeka sistemleri hızla gelişiyor."}}, + ] + agent.ollama_client = MagicMock() + agent.ollama_client.chat.return_value = iter(chunks) + + spoken = [] + + def _on_sentence(text: str, idx: int) -> None: + spoken.append((idx, text)) + + out = agent._chat_maybe_stream( + "test-model", + [{"role": "user", "content": "yapay zeka nedir?"}], + None, + {"num_predict": 64}, + on_sentence=_on_sentence, + ) + assert len(spoken) >= 2 + assert spoken[0][1] == "Tabii ki dostum," + assert "yapay zeka" in spoken[1][1] + + def test_companion_needs_line_high_social(): gen = CompanionLineGenerator(None, {"use_llm": False}) line = gen._needs_line( diff --git a/tests/modules/agent_core/test_world_state_environment.py b/tests/modules/agent_core/test_world_state_environment.py index e4ba8371..f706efda 100644 --- a/tests/modules/agent_core/test_world_state_environment.py +++ b/tests/modules/agent_core/test_world_state_environment.py @@ -45,3 +45,39 @@ def test_inject_world_state_omits_environment_when_idle(): ws = WorldState() injected = ws.inject_world_state("hello") assert "environment" not in injected + + +def test_update_laya_state_publishes_compact_decision(): + from types import SimpleNamespace + + ws = WorldState() + ws.update_laya_state(SimpleNamespace( + target_module="neopixel", + urgency_score=1.25, + affective_event="user_praise", + is_direct_command=True, + inference_ms=24.5, + )) + + state = ws.get_state() + assert state["laya_last_target"] == "neopixel" + assert state["laya_last_urgency"] == 1.25 + assert state["laya_last_affect"] == "user_praise" + assert state["laya_last_direct"] is True + assert state["laya_last_ms"] == 24.5 + + +def test_laya_scene_summary_includes_bounded_environment_details(): + ws = WorldState() + ws.update_scene({ + "summary": "çalışma masası", + "people": [{"name": "Emir"}], + "objects": [{"label": "laptop"}], + "hazards": [{"label": "kablo"}], + }) + + summary = ws.get_laya_scene_summary() + assert "çalışma masası" in summary + assert "laptop" in summary + assert "kablo" in summary + assert len(ws.get_laya_scene_summary(max_chars=12)) <= 12 diff --git a/tests/modules/autonomy/test_barge_in.py b/tests/modules/autonomy/test_barge_in.py index 65bd3b2d..fcd5dedc 100644 --- a/tests/modules/autonomy/test_barge_in.py +++ b/tests/modules/autonomy/test_barge_in.py @@ -3,6 +3,7 @@ from __future__ import annotations from modules.autonomy.services.barge_in import BargeInController +from modules.autonomy.services.brain_parts.vocal import VocalMixin def test_meaningful_speech_interrupts_while_robot_talking(): @@ -20,6 +21,16 @@ def test_single_word_does_not_interrupt_without_wakeword(): assert bc.should_interrupt(robot_speaking=True, user_text="ee", now=100.0) is False +def test_laya_urgent_single_word_interrupts_without_wakeword(): + bc = BargeInController({"min_words": 4, "cooldown_s": 0.0}) + assert bc.should_interrupt( + robot_speaking=True, + user_text="Dur!", + urgent=True, + now=100.0, + ) is True + + def test_wakeword_interrupts_even_if_single_word(): bc = BargeInController({"min_words": 5}) assert bc.should_interrupt(robot_speaking=True, user_text="sentry", has_wakeword=True, now=100.0) is True @@ -37,3 +48,28 @@ def test_cooldown_blocks_rapid_reinterrupt(): def test_disabled_never_interrupts(): bc = BargeInController({"enabled": False}) assert bc.should_interrupt(robot_speaking=True, user_text="dur bir saniye", has_wakeword=True, now=100.0) is False + + +def test_laya_emergency_reaches_brain_barge_in_controller(): + class _Brain(VocalMixin): + def __init__(self): + self.laya_engine = type("Laya", (), { + "_is_available": True, + "decide": lambda _self, _text: "emergency-decision", + "is_emergency_decision": lambda _self, decision: decision == "emergency-decision", + })() + self.barge_in = BargeInController({"cooldown_s": 0.0}) + self.stopped = False + + def _robot_is_speaking(self): + return True + + def _barge_in_stop_speaking(self): + self.stopped = True + + brain = _Brain() + + handled = brain._handle_barge_in_and_wakeword("Dur!", "dur!", False, "req-1") + + assert not handled + assert brain.stopped diff --git a/tests/modules/autonomy/test_behavior_composer.py b/tests/modules/autonomy/test_behavior_composer.py index 5c40f458..a9543f9d 100644 --- a/tests/modules/autonomy/test_behavior_composer.py +++ b/tests/modules/autonomy/test_behavior_composer.py @@ -4,6 +4,7 @@ import pytest from unittest.mock import MagicMock from modules.autonomy.services.behavior_composer import BehaviorComposer +from modules.autonomy.services.brain_parts.vocal_prosody import VocalProsodyMixin def test_behavior_composer_plan_execution(): @@ -80,3 +81,25 @@ def test_behavior_composer_macro_replay(): assert res["components"]["replay_macro"] == "greeting_sequence" mock_shadow.replay_macro.assert_called_once_with("greeting_sequence", mock_client) + +def test_behavior_composer_speech_uses_current_laya_updated_mood_tone(): + brain = VocalProsodyMixin.__new__(VocalProsodyMixin) + brain.client = MagicMock() + brain.state = {} + brain.mood = MagicMock() + brain.mood.get_dominant_emotion.return_value = "joy" + composer = BehaviorComposer(brain=brain) + + result = composer.execute_plan({"say": {"text": "Güzel haber!", "language": "tr"}}) + + assert result["components"]["say"] == {"text": "Güzel haber!", "tone": None} + brain.client.queue_action.assert_called_once_with( + "speak", + priority=50, + ttl_ms=10000, + payload={ + "text": "Güzel haber!", + "tone": {"rate": 190, "volume": 1.0}, + "language": "tr", + }, + ) diff --git a/tests/modules/autonomy/test_companion_goal_selector.py b/tests/modules/autonomy/test_companion_goal_selector.py index cc2744ad..b545b57e 100644 --- a/tests/modules/autonomy/test_companion_goal_selector.py +++ b/tests/modules/autonomy/test_companion_goal_selector.py @@ -35,3 +35,45 @@ def test_goal_selector_safety_is_critical(): assert plan["priority"] == "critical" assert plan["safe_to_execute"] is True assert plan["expression_event"] == "needs.safety" + + +def test_fresh_laya_urgency_supersedes_goal_with_safe_observation(): + selector = CompanionGoalSelector({ + "auto_execute": False, + "goal_formation": {"enabled": False}, + "laya_urgency": {"enabled": True, "freshness_s": 8.0, "safety_score_floor": 0.75}, + }) + plan = selector.select( + { + "dominant_need": "social", + "recommended_goal": "social_check_in", + "scores": {"social": 0.9, "safety": 0.1}, + "laya_urgency": {"high": True, "score": 1.8, "timestamp": 96.0}, + }, + now=100.0, + ) + + assert plan["dominant_need"] == "safety" + assert plan["recommended_goal"] == "pause_and_observe" + assert plan["laya_urgency_override"] == {"score": 1.8, "age_s": 4.0} + + +def test_expired_laya_urgency_does_not_supersede_goal(): + selector = CompanionGoalSelector({ + "auto_execute": False, + "goal_formation": {"enabled": False}, + "laya_urgency": {"enabled": True, "freshness_s": 8.0, "safety_score_floor": 0.75}, + }) + plan = selector.select( + { + "dominant_need": "social", + "recommended_goal": "social_check_in", + "scores": {"social": 0.9, "safety": 0.1}, + "laya_urgency": {"high": True, "score": 1.8, "timestamp": 80.0}, + }, + now=100.0, + ) + + assert plan["dominant_need"] == "social" + assert plan["recommended_goal"] == "social_check_in" + assert plan["laya_urgency_override"] is None diff --git a/tests/modules/autonomy/test_laya_brain_integration.py b/tests/modules/autonomy/test_laya_brain_integration.py new file mode 100644 index 00000000..681091e0 --- /dev/null +++ b/tests/modules/autonomy/test_laya_brain_integration.py @@ -0,0 +1,99 @@ +from types import SimpleNamespace + +from modules.autonomy.services.brain_parts.decision import DecisionMixin +from modules.autonomy.services.spinal_cord_reflex import SpinalCordReflexEngine + + +class _Client: + def __init__(self): + self.events = [] + + def push_interaction_event(self, event, payload=None): + self.events.append((event, payload or {})) + + def update_emotions(self, _emotions): + pass + + +class _Memory: + def add_event(self, _event): + pass + + +class _Laya: + enabled = True + _is_available = True + + def __init__(self, decision): + self.decision = decision + + def decide_with_context(self, _text, **_kwargs): + return self.decision + + @staticmethod + def is_emergency_decision(decision): + return decision.is_emergency + + @staticmethod + def is_urgent(decision): + return decision.urgency_score >= 1.5 + + @staticmethod + def get_mood_deltas(_decision): + return {} + + +class _Mood: + def modify(self, *_args): + pass + + +def test_laya_emergency_reaches_spinal_cord_motor_stop(): + client = _Client() + decision = SimpleNamespace( + is_emergency=True, + urgency_score=2.8, + target_module="emergency_stop", + affective_event="neutral", + ) + brain = DecisionMixin() + brain.laya_engine = _Laya(decision) + brain.state = {"last_speech_text": "Dur!"} + brain.scene_register = None + brain.spinal_cord = SpinalCordReflexEngine( + {"enabled": True, "cooldown_s": 0.0}, client, _Memory() + ) + brain.mood = _Mood() + brain.client = client + brain.memory = _Memory() + + brain._think_laya_reflexes() + + assert ("motor.stop", {"priority": "emergency", "source": "laya_voice"}) in client.events + assert brain.state["laya_last_high_urgency"] is True + assert brain.state["laya_last_decision_ts"] > 0 + + +def test_non_emergency_laya_decision_does_not_stop_motor(): + client = _Client() + decision = SimpleNamespace( + is_emergency=False, + urgency_score=0.2, + target_module="system2_chat", + affective_event="neutral", + ) + brain = DecisionMixin() + brain.laya_engine = _Laya(decision) + brain.state = {"last_speech_text": "Nasılsın?"} + brain.scene_register = None + brain.spinal_cord = SpinalCordReflexEngine( + {"enabled": True, "cooldown_s": 0.0}, client, _Memory() + ) + brain.mood = _Mood() + brain.client = client + brain.memory = _Memory() + + brain._think_laya_reflexes() + + assert not any(event == "motor.stop" for event, _payload in client.events) + assert brain.state["laya_last_high_urgency"] is False diff --git a/tests/modules/autonomy/test_laya_vision_event.py b/tests/modules/autonomy/test_laya_vision_event.py new file mode 100644 index 00000000..400769ea --- /dev/null +++ b/tests/modules/autonomy/test_laya_vision_event.py @@ -0,0 +1,103 @@ +from types import SimpleNamespace + +from modules.autonomy.services.brain_parts.decision import DecisionMixin + + +class _Mood: + def __init__(self): + self.events = [] + + def apply_affective_event(self, event): + self.events.append(event) + + +class _Laya: + enabled = True + _is_available = True + + def __init__(self, affect="user_rude", confidence=0.95): + self.affect = affect + self.confidence = confidence + self.calls = 0 + + def decide_with_context(self, text, **kwargs): + self.calls += 1 + return SimpleNamespace( + affective_event=self.affect, + affect_confidence=self.confidence, + target_module="camera", + module_confidence=0.9, + urgency_score=0.0, + ) + + +def _brain(affect="user_rude", confidence=0.95): + brain = DecisionMixin() + brain.config = {"tri_layer": {"laya": {"affective": { + "vision_event_min_confidence": 0.8, + "vision_event_min_affect_confidence": 0.8, + "vision_event_cooldown_s": 60, + }}}} + brain.state = {} + brain._laya_vision_event_lock = __import__("threading").Lock() + brain.laya_engine = _Laya(affect, confidence) + brain.mood = _Mood() + return brain + + +def test_confident_face_emotion_routes_through_laya_and_updates_mood(): + brain = _brain() + payload = {"name": "Emir", "emotion": "angry", "confidence": 0.93} + + assert brain.handle_laya_vision_event("face_emotion", payload) is True + assert brain.mood.events == ["user_rude"] + assert brain.laya_engine.calls == 1 + assert brain.state["laya_last_vision_affect"]["emotion"] == "angry" + + +def test_face_emotion_event_rejects_low_confidence_and_deduplicates(): + brain = _brain() + low = {"name": "Emir", "emotion": "angry", "confidence": 0.6} + assert brain.handle_laya_vision_event("face_emotion", low) is False + assert brain.laya_engine.calls == 0 + + payload = {"name": "Emir", "emotion": "angry", "confidence": 0.93} + assert brain.handle_laya_vision_event("face_emotion", payload) is True + assert brain.handle_laya_vision_event("face_emotion", payload) is False + assert brain.mood.events == ["user_rude"] + + +def test_laya_vision_event_requires_confident_affect_classification(): + brain = _brain(affect="user_praise", confidence=0.5) + payload = {"name": "Emir", "emotion": "happy", "confidence": 0.93} + + assert brain.handle_laya_vision_event("face_emotion", payload) is False + assert brain.mood.events == [] + + +def test_scene_event_is_classified_once_and_stored_without_direct_mood_delta(): + brain = _brain(affect="neutral", confidence=0.95) + payload = { + "reason": "scene_refresh", + "context": {"summary": "Mutfakta yerde bir kutu görünüyor"}, + } + + assert brain.handle_laya_vision_event("scene_changed", payload) is True + assert brain.handle_laya_vision_event("scene_changed", payload) is False + assert brain.laya_engine.calls == 1 + assert brain.mood.events == [] + assert brain.state["laya_last_vision_decision"]["event_type"] == "scene_changed" + + +def test_non_face_vision_events_do_not_duplicate_perception_mood_appraisal(): + for event_type, payload in ( + ("owner_seen", {"name": "Emir", "confidence": 0.95}), + ("new_person", {"name": "Guest", "confidence": 0.9}), + ("hazard_detected", {"hazards": ["obstacle"], "confidence": 0.99}), + ): + brain = _brain(affect="user_rude", confidence=0.99) + + assert brain.handle_laya_vision_event(event_type, payload) is True + assert brain.state["laya_last_vision_decision"]["event_type"] == event_type + # Existing perception/needs/appraisal owners handle these semantics. + assert brain.mood.events == [] diff --git a/tests/modules/autonomy/test_visual_hazard_appraisal.py b/tests/modules/autonomy/test_visual_hazard_appraisal.py new file mode 100644 index 00000000..2c03fbc2 --- /dev/null +++ b/tests/modules/autonomy/test_visual_hazard_appraisal.py @@ -0,0 +1,12 @@ +from modules.autonomy.services.affective_appraisal import AffectiveAppraisal + + +def test_visual_hazard_maps_to_visual_risk_mood_deltas(): + appraisal = AffectiveAppraisal() + + result = appraisal.process_event("visual_hazard", intensity=0.5) + + assert result["matched"] is True + assert result["mood_deltas"]["fear"] == 10.0 + assert result["mood_deltas"]["energy"] == 2.5 + assert result["mood_deltas"]["curiosity"] == 1.5 diff --git a/tests/modules/vlm_bridge/test_laya_vision_events.py b/tests/modules/vlm_bridge/test_laya_vision_events.py new file mode 100644 index 00000000..15af8b65 --- /dev/null +++ b/tests/modules/vlm_bridge/test_laya_vision_events.py @@ -0,0 +1,34 @@ +from modules.vlm_bridge.services.processor import VisionProcessor +from modules.vlm_bridge.services.vision_event_bus import EVENT_FACE_EMOTION, VisionEventBus + + +def test_face_emotion_event_requires_confidence_and_obeys_cooldown(): + processor = VisionProcessor.__new__(VisionProcessor) + processor.event_bus = VisionEventBus() + processor._face_emotion_event_min_confidence = 0.8 + processor._face_emotion_event_cooldown_s = 30.0 + processor._last_face_emotion_event = {} + received = [] + processor.event_bus.subscribe(EVENT_FACE_EMOTION, lambda _kind, data: received.append(dict(data))) + + processor._publish_face_emotion_event("person-1", "Emir", "angry", 0.7) + processor._publish_face_emotion_event("person-1", "Emir", "angry", 0.92) + processor._publish_face_emotion_event("person-1", "Emir", "angry", 0.95) + + assert len(received) == 1 + assert received[0]["emotion"] == "angry" + assert received[0]["confidence"] == 0.92 + + +def test_neutral_face_emotion_does_not_publish(): + processor = VisionProcessor.__new__(VisionProcessor) + processor.event_bus = VisionEventBus() + processor._face_emotion_event_min_confidence = 0.5 + processor._face_emotion_event_cooldown_s = 0.0 + processor._last_face_emotion_event = {} + received = [] + processor.event_bus.subscribe(EVENT_FACE_EMOTION, lambda *_args: received.append(True)) + + processor._publish_face_emotion_event("person-1", "Emir", "neutral", 0.99) + + assert received == [] diff --git a/tests/modules/vlm_bridge/test_visual_importance.py b/tests/modules/vlm_bridge/test_visual_importance.py index f03e957a..785f8753 100644 --- a/tests/modules/vlm_bridge/test_visual_importance.py +++ b/tests/modules/vlm_bridge/test_visual_importance.py @@ -3,7 +3,11 @@ from __future__ import annotations from modules.vlm_bridge.services.processor import VisionProcessor -from modules.vlm_bridge.services.visual_context import VisualContextCache +from modules.vlm_bridge.services.visual_context import ( + PersonContext, + VisionFrameContext, + VisualContextCache, +) def _proc_with_cache(): @@ -48,3 +52,16 @@ def test_caller_high_importance_is_not_downgraded(): is_user_question=True, ) assert proc.visual_context_cache.get_latest().importance_score >= 0.9 + + +def test_visual_context_cache_builds_compact_laya_scene_hint(): + cache = VisualContextCache() + cache.update(VisionFrameContext( + summary="at a desk", + people=[PersonContext(name="Emir")], + objects=[{"label": "laptop"}, {"label": "cup"}], + hazards=[{"label": "knife"}], + )) + + assert cache.get_laya_context() == "1 kişi, laptop, cup, tehlike: knife" + assert len(cache.get_laya_context(max_chars=12)) <= 12 diff --git a/tests/modules/voice/speech/test_smoke.py b/tests/modules/voice/speech/test_smoke.py index 7794b6d7..899a3b0c 100644 --- a/tests/modules/voice/speech/test_smoke.py +++ b/tests/modules/voice/speech/test_smoke.py @@ -16,3 +16,16 @@ def test_service_init(): from modules.voice.speech.xSpeechService import SpeechService svc = SpeechService() assert svc is not None + + +def test_direction_and_sound_tracking(): + from modules.voice.speech.xSpeechService import SpeechService + svc = SpeechService() + assert svc._tracking is False + svc.track_start() + assert svc._tracking is True + assert svc._direction is not None + st = svc.track_status() + assert st["tracking"] is True + svc.track_stop() + assert svc._tracking is False diff --git a/tools/download_laya_model.py b/tools/download_laya_model.py new file mode 100644 index 00000000..c37359fe --- /dev/null +++ b/tools/download_laya_model.py @@ -0,0 +1,30 @@ +import os +import sys +import time +from huggingface_hub import hf_hub_download + +REPO_ID = "convaiinnovations/laya" +FILES = [ + "multilingual/rl_agent_config.json", + "multilingual/encoder/config.json", + "multilingual/tokenizer/tokenizer_config.json", + "multilingual/tokenizer/tokenizer.json", + "multilingual/model.safetensors", +] + +print("=" * 60) +print("Laya Multilingual Model İndirme Başlatılıyor...") +print("Hedef Repo:", REPO_ID) +print("=" * 60) + +for i, filename in enumerate(FILES, 1): + print(f"\n[{i}/{len(FILES)}] İndiriliyor: {filename} ...") + start = time.time() + path = hf_hub_download(repo_id=REPO_ID, filename=filename) + size_mb = os.path.getsize(path) / (1024 * 1024) + elapsed = time.time() - start + print(f"Tamamlandı: {size_mb:.2f} MB ({elapsed:.1f} sn) -> {path}") + +print("\n" + "=" * 60) +print("Tüm model dosyaları başarıyla indirildi!") +print("=" * 60) diff --git a/tools/evaluate_laya_action_proposals.py b/tools/evaluate_laya_action_proposals.py new file mode 100644 index 00000000..57c99aa5 --- /dev/null +++ b/tools/evaluate_laya_action_proposals.py @@ -0,0 +1,1021 @@ +#!/usr/bin/env python3 +"""Offline-only calibration report for Laya typed action questions. + +This tool runs model inference over fixed text examples. It never constructs a +ToolRegistry or calls a robot, gateway, Arduino, or action endpoint. +""" + +from __future__ import annotations + +import argparse +import glob +import json +import os +import re +from typing import Any, Dict, List + +from modules.agent_core.services.laya_action_contract import ( + build_laya_action_proposal, + get_laya_action_questions, +) + + +def _load_laya_agent() -> Any: + # Keep parser-only evaluation usable without the optional Torch runtime. + import laya + return laya.load(_model_path(), device="cpu") +from modules.agent_core.services.laya_engine import LayaDecision, LayaEngine + + +CASES = ( + {"text": "Işıkları kırmızı yap", "target": "neopixel", "direct": True, "color": "red"}, + {"text": "Işıkları mavi yap", "target": "neopixel", "direct": True, "color": "blue"}, + {"text": "Işıkları yeşil yap", "target": "neopixel", "direct": True, "color": "green"}, + {"text": "Işıkları sarı yap", "target": "neopixel", "direct": True, "color": "yellow"}, + {"text": "Işıkları mor yap", "target": "neopixel", "direct": True, "color": "purple"}, + {"text": "Işıkları kapat", "target": "neopixel", "direct": True, "color": "off", "effect": "OFF", "emergency": False}, + {"text": "Işıkları sabit kırmızı yak", "target": "neopixel", "direct": True, "color": "red", "effect": "SOLID"}, + {"text": "Işıkları nefes efektiyle mavi yap", "target": "neopixel", "direct": True, "color": "blue", "effect": "BREATHE"}, + {"text": "Mavi rengi bana anlat", "target": "system2_chat", "direct": False, "color": "unspecified"}, + {"text": "Neden gökyüzü mavi?", "target": "system2_chat", "direct": False, "color": "unspecified"}, + {"text": "Işıkları ayarla", "target": "neopixel", "direct": True, "color": "unspecified"}, + {"text": "Kamerada ne görüyorsun?", "target": "camera", "direct": False, "color": "unspecified"}, + {"text": "Motorları acil durdur, çarpacaksın!", "target": "emergency_stop", "direct": True, "color": "unspecified", "emergency": True}, + {"text": "Dur hemen!", "target": "emergency_stop", "direct": True, "color": "unspecified", "emergency": True}, + # Additional Turkish paraphrases and negative controls. These are a broader + # calibration set, not an independent held-out validation set. + {"text": "LED'leri turuncu renge çevir", "target": "neopixel", "direct": True, "color": "orange"}, + {"text": "Robotun ışıkları beyaz olsun", "target": "neopixel", "direct": True, "color": "white"}, + {"text": "Işık rengini pembeye ayarla", "target": "neopixel", "direct": True, "color": "pink"}, + {"text": "Işıkları camgöbeği yap", "target": "neopixel", "direct": True, "color": "cyan"}, + {"text": "Işıkları turkuaz renkte yak", "target": "neopixel", "direct": True, "color": "teal"}, + {"text": "Işıklar magenta yansın", "target": "neopixel", "direct": True, "color": "magenta"}, + {"text": "LED'leri mor renkte yak", "target": "neopixel", "direct": True, "color": "purple"}, + {"text": "Işıkları yeşil renkte yak", "target": "neopixel", "direct": True, "color": "green"}, + {"text": "Işıkları söndür", "target": "neopixel", "direct": True, "color": "off", "effect": "OFF", "emergency": False}, + {"text": "LED ışıklarını kapalı duruma getir", "target": "neopixel", "direct": True, "color": "off", "effect": "OFF", "emergency": False}, + {"text": "Işıkları aç", "target": "neopixel", "direct": True, "color": "unspecified"}, + {"text": "Işıkları biraz kıs", "target": "neopixel", "direct": True, "color": "unspecified"}, + {"text": "Işık rengini değiştir", "target": "neopixel", "direct": True, "color": "unspecified"}, + {"text": "Kırmızı benim en sevdiğim renk", "target": "system2_chat", "direct": False, "color": "unspecified"}, + {"text": "Mavi rengin psikolojideki anlamı ne?", "target": "system2_chat", "direct": False, "color": "unspecified"}, + {"text": "Fotoğraftaki ışık ne renk?", "target": "camera", "direct": False, "color": "unspecified"}, + {"text": "Robotun ışıkları hakkında konuşalım", "target": "system2_chat", "direct": False, "color": "unspecified"}, + {"text": "Bana mor rengi tarif et", "target": "system2_chat", "direct": False, "color": "unspecified"}, +) + +# Separate, pre-labeled examples. Do not use these to select or rewrite question +# variants; report them only as a final comparison after calibration. +HOLDOUT_CASES = ( + {"text": "LED rengini turuncuya al", "color": "orange"}, + {"text": "Işıklar mor renkte parlasın", "color": "purple"}, + {"text": "Neon ışığı pembe olsun", "color": "pink"}, + {"text": "Işığı camgöbeğine ayarlayabilir misin?", "color": "cyan"}, + {"text": "Işık halkasını turkuaza çevir", "color": "teal"}, + {"text": "Ön LED'i macenta yak", "color": "magenta"}, + {"text": "Küçük ışığı sarı yapar mısın?", "color": "yellow"}, + {"text": "Işık şeridini beyaza ayarla", "color": "white"}, + {"text": "RGB ışıkları cyan yap", "color": "cyan"}, + {"text": "Işıkları kırmızıya çevirir misin?", "color": "red"}, + {"text": "Bütün ışıkları maviye ayarlasana", "color": "blue"}, + {"text": "LED'leri yeşile boya", "color": "green"}, + {"text": "Lambaları turuncuya çevirir misin?", "color": "orange"}, + {"text": "Işık rengini mora getir", "color": "purple"}, + {"text": "Işıkları kapalı yap", "color": "off"}, + {"text": "LED'i söndür", "color": "off"}, + {"text": "Robot ışıklarını açabilir misin?", "color": "unspecified"}, + {"text": "LED efektini nefes yap", "color": "unspecified"}, + {"text": "Bu ışık çok parlak", "color": "unspecified"}, + {"text": "Fotoğrafta kırmızı ışık var mı?", "color": "unspecified"}, + {"text": "LED kırmızı yanıyor mu?", "color": "unspecified"}, + {"text": "Mavi ışığı seviyorum", "color": "unspecified"}, + {"text": "Rengi kırmızı yapma", "color": "unspecified"}, + {"text": "Sence LED'leri kırmızı mı yapsam?", "color": "unspecified"}, + {"text": "Robotun gözleri kırmızı olsun", "color": "unspecified"}, + {"text": "Işıklar kapalı kalsın", "color": "off"}, + {"text": "Lütfen ışıkları kapatır mısın?", "color": "off"}, + {"text": "Işık söndü", "color": "unspecified"}, + {"text": "Mavi renkli LED nasıl görünür?", "color": "unspecified"}, + {"text": "Işıkları kırmızı yapmak güvenli mi?", "color": "unspecified"}, +) + +# Additional parser-development examples; scores on these are not final. +RULE_DEV_CASES = ( + {"text": "Öndeki ışıkları kırmızı renge ayarla", "color": "red"}, + {"text": "Sağ LED'i yeşil yap", "color": "green"}, + {"text": "Işık barını maviye boya", "color": "blue"}, + {"text": "LED lambayı sarıya çevir", "color": "yellow"}, + {"text": "Işıkları turuncu yak", "color": "orange"}, + {"text": "Üst ışık mor olsun", "color": "purple"}, + {"text": "LED ışığını pembeye ayarla", "color": "pink"}, + {"text": "Arka lambaları beyaz yap", "color": "white"}, + {"text": "Işık halkasını camgöbeği yap", "color": "cyan"}, + {"text": "Şeridi turkuaz renkte ayarla", "color": "teal"}, + {"text": "RGB LED'leri magenta yak", "color": "magenta"}, + {"text": "Işık halkasını kapalı hale getir", "color": "off"}, + {"text": "Tüm LED lambaları söndür", "color": "off"}, + {"text": "Lütfen ışıkları kapatır mısın?", "color": "off"}, + {"text": "Yukarıdaki kırmızı LED yanıp sönüyor", "color": "unspecified"}, + {"text": "Yeşil ışık yanıyor", "color": "unspecified"}, + {"text": "LED rengini değiştirmek için ne yapmalıyım?", "color": "unspecified"}, + {"text": "Fotoğrafın solunda sarı bir lamba görüyorum", "color": "unspecified"}, + {"text": "Işıkları yakmalı mıyım?", "color": "unspecified"}, + {"text": "LED'leri kırmızıya çevirmek zor mu?", "color": "unspecified"}, + {"text": "RGB standardı nedir?", "color": "unspecified"}, + {"text": "Turkuaz ile teal aynı renk mi?", "color": "unspecified"}, + {"text": "LED'i mavi yapmıştık", "color": "unspecified"}, + {"text": "Rengi sarı olan lamba hangisi?", "color": "unspecified"}, + {"text": "Kapalı LED daha iyi mi?", "color": "unspecified"}, + {"text": "Bu cihazın ışıkları kapandı", "color": "unspecified"}, + {"text": "Yarın ışıkları kırmızı yapsak mı?", "color": "unspecified"}, + {"text": "Robot gözlerini yeşil yap", "color": "unspecified"}, + {"text": "Neon ışığı turuncu yanıyor gibi", "color": "unspecified"}, + {"text": "LED'lerin rengi neden kırmızı?", "color": "unspecified"}, +) + +# Final untouched set. Freeze both the prompt variants and rule baseline before +# evaluating this split. +FINAL_HOLDOUT_CASES = ( + {"text": "Panel LED'lerini kırmızıya ayarla", "color": "red"}, + {"text": "Öndeki ışık yeşil renge dönsün", "color": "green"}, + {"text": "LED halkasını maviye çevir", "color": "blue"}, + {"text": "İkaz lambasını sarı yap", "color": "yellow"}, + {"text": "Robotun durum ışığını turuncu renge getir", "color": "orange"}, + {"text": "Üst göstergedeki ışık mor olsun", "color": "purple"}, + {"text": "Yan ışıklara pembe renk ver", "color": "pink"}, + {"text": "LED şeridinde beyazı seç", "color": "white"}, + {"text": "RGB bandını camgöbeğine ayarla", "color": "cyan"}, + {"text": "LED halkasını turkuaz yap", "color": "teal"}, + {"text": "Ön uyarı LED'ini magenta yak", "color": "magenta"}, + {"text": "Alt aydınlatmayı kapat", "color": "off"}, + {"text": "Panel ışığını söndür", "color": "off"}, + {"text": "Yan LED'leri kapalı hale getir", "color": "off"}, + {"text": "Paneldeki kırmızı gösterge yanıyor", "color": "unspecified"}, + {"text": "LED'in yeşil renkte olduğunu görüyorum", "color": "unspecified"}, + {"text": "Kırmızı ışık yandığında ne anlama gelir?", "color": "unspecified"}, + {"text": "Geçen hafta LED'i mavi yapmıştım", "color": "unspecified"}, + {"text": "Yarın ışıkları sarı yapsak mı?", "color": "unspecified"}, + {"text": "LED'ler turuncu yanıyor", "color": "unspecified"}, + {"text": "RGB ışıkta cyan rengi güzel duruyor", "color": "unspecified"}, + {"text": "Hangi renk ışıkları var?", "color": "unspecified"}, + {"text": "LED şeridi pembe olabilir mi?", "color": "unspecified"}, + {"text": "Masa lambası maviye dönmüş", "color": "unspecified"}, + {"text": "Mor LED'leri seviyorum", "color": "unspecified"}, + {"text": "Bu ışık beyaz değil", "color": "unspecified"}, + {"text": "Işıklar açık kalsın", "color": "unspecified"}, + {"text": "Aydınlatmayı azalt", "color": "unspecified"}, + {"text": "Göstergedeki sarı renk hata mı?", "color": "unspecified"}, + {"text": "RGB nedir?", "color": "unspecified"}, +) + +# Separate intent/safety split; expected labels are fixed before model inference. +INTENT_HOLDOUT_CASES = ( + {"text": "LED'leri yeşil yap", "target": "neopixel", "direct": True, "urgent": False, "emergency": False}, + {"text": "Sabit kırmızı LED animasyonunu etkinleştir", "target": "neopixel", "direct": True, "urgent": False, "emergency": False, "effect": "SOLID"}, + {"text": "Işıklar mavi nefes efektiyle yansın", "target": "neopixel", "direct": True, "urgent": False, "emergency": False, "effect": "BREATHE"}, + {"text": "Mor renkte nabız efekti uygula", "target": "neopixel", "direct": True, "urgent": False, "emergency": False, "effect": "PULSE"}, + {"text": "RGB ışık sistemini tamamen söndür", "target": "neopixel", "direct": True, "urgent": False, "emergency": False, "effect": "OFF"}, + {"text": "Başını sağa çevir", "target": "arduino_serial", "direct": True, "urgent": False, "emergency": False}, + {"text": "Pan kafayı aşağı indir", "target": "arduino_serial", "direct": True, "urgent": False, "emergency": False}, + {"text": "İleri doğru yürü", "target": "arduino_serial", "direct": True, "urgent": False, "emergency": False}, + {"text": "Motor hızını yavaşlat", "target": "arduino_serial", "direct": True, "urgent": False, "emergency": False}, + {"text": "Robotu 30 santim ileri sür", "target": "arduino_serial", "direct": True, "urgent": False, "emergency": False}, + {"text": "Motorları hemen durdur, önünde engel var!", "target": "emergency_stop", "direct": True, "urgent": True, "emergency": True}, + {"text": "Dur, düşeceksin!", "target": "emergency_stop", "direct": True, "urgent": True, "emergency": True}, + {"text": "Çarpışmayı önlemek için anında frenle!", "target": "emergency_stop", "direct": True, "urgent": True, "emergency": True}, + {"text": "Robotu acil olarak kapat!", "target": "emergency_stop", "direct": True, "urgent": True, "emergency": True}, + {"text": "Tehlike var, derhal hareketi kes!", "target": "emergency_stop", "direct": True, "urgent": True, "emergency": True}, + {"text": "Robot, 'Merhaba' de", "target": "speak", "direct": True, "urgent": False, "emergency": False}, + {"text": "Bana yüksek sesle günaydın söyle", "target": "speak", "direct": True, "urgent": False, "emergency": False}, + {"text": "Bir uyarı sesi çıkar", "target": "speak", "direct": True, "urgent": False, "emergency": False}, + {"text": "Adımı sesli olarak oku", "target": "speak", "direct": True, "urgent": False, "emergency": False}, + {"text": "Alarm sesi çal", "target": "speak", "direct": True, "urgent": False, "emergency": False}, + {"text": "Kamerayı aç ve önüne bak", "target": "camera", "direct": True, "urgent": False, "emergency": False}, + {"text": "Etrafına bak", "target": "camera", "direct": True, "urgent": False, "emergency": False}, + {"text": "Önünde ne görüyorsun?", "target": "camera", "direct": False, "urgent": False, "emergency": False}, + {"text": "Masadaki nesneyi kameradan tanı", "target": "camera", "direct": True, "urgent": False, "emergency": False}, + {"text": "Fotoğrafta kaç kişi var?", "target": "camera", "direct": False, "urgent": False, "emergency": False}, + {"text": "Yer çekimi nasıl çalışır?", "target": "system2_chat", "direct": False, "urgent": False, "emergency": False}, + {"text": "Bir uzay macerası hikayesi anlat", "target": "system2_chat", "direct": False, "urgent": False, "emergency": False}, + {"text": "Yarın için bana çalışma planı öner", "target": "system2_chat", "direct": False, "urgent": False, "emergency": False}, + {"text": "Merhaba, bugün keyfin nasıl?", "target": "system2_chat", "direct": False, "urgent": False, "emergency": False}, + {"text": "İki seçenek arasındaki farkı açıkla", "target": "system2_chat", "direct": False, "urgent": False, "emergency": False}, +) + +# New calibration-only split for the next target/urgency/effect iteration. +INTENT_DEV_CASES = ( + {"text": "LED rengini sarıya çek", "target": "neopixel", "direct": True, "urgent": False, "emergency": False}, + {"text": "Turkuaz ışık animasyonunu darbeli çalıştır", "target": "neopixel", "direct": True, "urgent": False, "emergency": False, "effect": "PULSE"}, + {"text": "LED'leri dalga animasyonuna al", "target": "neopixel", "direct": True, "urgent": False, "emergency": False, "effect": "WAVE"}, + {"text": "Şerit ışığını rastgele kırpışma efektiyle çalıştır", "target": "neopixel", "direct": True, "urgent": False, "emergency": False, "effect": "RANDOM_BLINK"}, + {"text": "Işığı beyaz yap ve sabit tut", "target": "neopixel", "direct": True, "urgent": False, "emergency": False, "effect": "SOLID"}, + {"text": "Kafayı sola doğru döndür", "target": "arduino_serial", "direct": True, "urgent": False, "emergency": False}, + {"text": "Robotu yavaşça geri yürüt", "target": "arduino_serial", "direct": True, "urgent": False, "emergency": False}, + {"text": "Pan motorunu on beş derece kaldır", "target": "arduino_serial", "direct": True, "urgent": False, "emergency": False}, + {"text": "Yürüyüşü başlat", "target": "arduino_serial", "direct": True, "urgent": False, "emergency": False}, + {"text": "Başını öne eğ", "target": "arduino_serial", "direct": True, "urgent": False, "emergency": False}, + {"text": "Önündeki duvara çarpacaksın, hemen dur!", "target": "emergency_stop", "direct": True, "urgent": True, "emergency": True}, + {"text": "Motorları anında kes, devriliyor!", "target": "emergency_stop", "direct": True, "urgent": True, "emergency": True}, + {"text": "Dur! Uçurumun kenarındasın!", "target": "emergency_stop", "direct": True, "urgent": True, "emergency": True}, + {"text": "Acil stop, robot!", "target": "emergency_stop", "direct": True, "urgent": True, "emergency": True}, + {"text": "Çarpışma riski var, hareketi hemen durdur!", "target": "emergency_stop", "direct": True, "urgent": True, "emergency": True}, + {"text": "Şunu yüksek sesle oku: sistem hazır", "target": "speak", "direct": True, "urgent": False, "emergency": False}, + {"text": "Kısa bir bip sesi çıkar", "target": "speak", "direct": True, "urgent": False, "emergency": False}, + {"text": "Neşeli bir melodi çal", "target": "speak", "direct": True, "urgent": False, "emergency": False}, + {"text": "Sessiz kal", "target": "speak", "direct": True, "urgent": False, "emergency": False}, + {"text": "Komutu sesli olarak tekrar et", "target": "speak", "direct": True, "urgent": False, "emergency": False}, + {"text": "Kameradan masayı tara", "target": "camera", "direct": True, "urgent": False, "emergency": False}, + {"text": "Solunda kim var?", "target": "camera", "direct": False, "urgent": False, "emergency": False}, + {"text": "Görüntüdeki rengi tespit et", "target": "camera", "direct": True, "urgent": False, "emergency": False}, + {"text": "Kamerayı kapat", "target": "camera", "direct": True, "urgent": False, "emergency": False}, + {"text": "Çevrende bir kedi var mı?", "target": "camera", "direct": False, "urgent": False, "emergency": False}, + {"text": "Motor komutlarının Arduino tarafındaki rolü nedir?", "target": "system2_chat", "direct": False, "urgent": False, "emergency": False}, + {"text": "LED renkleri hakkında kısa bir bilgi ver", "target": "system2_chat", "direct": False, "urgent": False, "emergency": False}, + {"text": "Bir robota konuşmayı öğretmek zor mu?", "target": "system2_chat", "direct": False, "urgent": False, "emergency": False}, + {"text": "SentryBOT için üç isim öner", "target": "system2_chat", "direct": False, "urgent": False, "emergency": False}, + {"text": "Tehlikeli durumlarda robotlar nasıl tepki vermeli?", "target": "system2_chat", "direct": False, "urgent": False, "emergency": False}, +) + + +# Frozen-rule, evaluator-only audio holdout; do not use for parser tuning. +AUDIO_HOLDOUT_CASES = ( + {"text": "Robot, şu notu sesli oku", "speak": True}, + {"text": "Bir kez bip sesi ver", "speak": True}, + {"text": "Kısa bir alarm çal", "speak": True}, + {"text": "Bana bir türkü söyle", "speak": True}, + {"text": "Şimdi konuş ve kendini tanıt", "speak": True}, + {"text": "Bu cümleyi aynen tekrar et: test başarılı", "speak": True}, + {"text": "Lütfen sessiz kal", "speak": True}, + {"text": "İstiklal Marşı'nı sesli oku", "speak": True}, + {"text": "Robotlar neden bip sesi çıkarır?", "speak": False}, + {"text": "Alarm sesi gece beni uyandırdı", "speak": False}, + {"text": "Dün bana bir şarkı söyledi", "speak": False}, + {"text": "Konuşma sentezi nasıl çalışıyor?", "speak": False}, + {"text": "Sessiz kalmak bazen iyidir", "speak": False}, + {"text": "'Yardım et' sözünü İngilizceye çevir", "speak": False}, + {"text": "Sesli okuma özelliği var mı?", "speak": False}, + {"text": "Bir melodi çalmak için hangi donanım gerekir?", "speak": False}, + {"text": "Az önce ne söyledin?", "speak": False}, + {"text": "Robotun sesi çok yüksek", "speak": False}, + {"text": "Bana sesli komut örnekleri ver", "speak": False}, + {"text": "Bu alarmı kim çalmış?", "speak": False}, + {"text": "Şarkı söylemeyi seviyor musun?", "speak": False}, + {"text": "Konuşma tanıma ile metin tanıma arasındaki fark ne?", "speak": False}, + {"text": "Biraz önce sessiz kaldı", "speak": False}, + {"text": "Müzik çalar bozulmuş", "speak": False}, + {"text": "Yüksek ses insanı rahatsız edebilir", "speak": False}, + {"text": "Bip sesi dosyasını nerede bulurum?", "speak": False}, + {"text": "Dün okuduğun metni hatırlıyor musun?", "speak": False}, + {"text": "Bana kısa bir hikâye anlat", "speak": False}, + {"text": "Robotun neden konuşmadığını açıkla", "speak": False}, + {"text": "Alarmı kapattım ama sesi sürüyor", "speak": False}, +) + +if ( + {case["text"] for case in CASES} + | {case["text"] for case in HOLDOUT_CASES} + | {case["text"] for case in RULE_DEV_CASES} + | {case["text"] for case in FINAL_HOLDOUT_CASES} + | {case["text"] for case in INTENT_HOLDOUT_CASES} + | {case["text"] for case in AUDIO_HOLDOUT_CASES} +) & { + case["text"] for case in INTENT_DEV_CASES +}: + raise RuntimeError("Intent development examples must be disjoint from calibration and holdout sets.") + + +def _rule_based_color(text: str) -> str: + """Conservative evaluator-only baseline; abstain unless intent is explicit.""" + # Turkish uppercase I is dotless ı; normalize it before Unicode casefolding. + normalized = str(text or "").replace("I", "ı").casefold() + has_light = re.search(r"\b(?:ışık\w*|ışığ\w*|lamba\w*|led\w*|neon\w*|rgb|aydınlat\w*)\b", normalized) + has_action = re.search( + r"\b(?:yap\w*|ayarla\w*|yak\w*|parla\w*|çevir\w*|getir\w*|al\w*|" + r"boya\w*|ver\w*|seç\w*|olsun|kapat\w*|söndür\w*|sönük\w*|kapalı\s+(?:kalsın|yap|hale\s+getir))\b", + normalized, + ) + if not has_light or not has_action: + return "unspecified" + if re.search( + r"\b(?:yapma|istemiyorum|istemem|olmasın|sence|acaba|yapsam|yapayım|" + r"yapmalı\s+mıyım|yapsak|yapmış\w*|ayarlamış\w*|çevirmiş\w*|" + r"gerekir\s+mi|güvenli\s+mi)\b", + normalized, + ): + return "unspecified" + + colors = { + "red": r"\bkırmızı\w*\b", + "green": r"\byeşil\w*\b", + "blue": r"\bmavi\w*\b", + "yellow": r"\bsarı\w*\b", + "orange": r"\bturuncu\w*\b", + "purple": r"\bmor\w*\b", + "pink": r"\bpembe\w*\b", + "cyan": r"\b(?:camgöbeği|camgöbeğine|cyan)\b", + "teal": r"\b(?:turkuaz\w*|teal)\b", + "magenta": r"\b(?:macenta\w*|magenta)\b", + "white": r"\bbeyaz\w*\b", + } + matches = [choice for choice, pattern in colors.items() if re.search(pattern, normalized)] + wants_off = bool(re.search( + r"\b(?:kapat\w*|söndür\w*|sönük\w*|kapalı\s+(?:kalsın|yap|hale\s+getir))\b", + normalized, + )) + if wants_off: + return "off" if not matches else "unspecified" + return matches[0] if len(matches) == 1 else "unspecified" + + +def _rule_based_explicit_audio(text: str) -> bool: + """Evaluator-only parser for literal speech/sound commands; never dispatches.""" + normalized = str(text or "").replace("I", "ı").casefold() + command_patterns = ( + r"\boku\b", r"\bokur musun\b", r"\bsöyle\b", r"\bsöyler misin\b", + r"\bses çıkar\w*", r"\b(?:bip|alarm|melodi|şarkı) sesi?\s+(?:çal|ver|çıkar)\w*", + r"\bmelodi çal\w*", r"\bşarkı söyle\w*", r"\bçal\b", r"\bkonuş\b", + r"\btekrar et\b", r"\bsessiz kal\b", + r"[\"'“][^\"'”]+[\"'”]\s+(?:de|diye söyle)\b", + ) + return any(re.search(pattern, normalized) for pattern in command_patterns) + + +def _evaluate_audio_parser_cases(cases: Any) -> Dict[str, Any]: + """Measure the frozen literal parser without loading Laya or invoking tools.""" + tp = fp = fn = tn = 0 + for case in cases: + expected = bool(case["speak"]) + predicted = _rule_based_explicit_audio(case["text"]) + tp += expected and predicted + fp += not expected and predicted + fn += expected and not predicted + tn += not expected and not predicted + total = len(cases) + return { + "sample_count": total, + "accuracy": round((tp + tn) / total, 3) if total else None, + "precision": round(tp / (tp + fp), 3) if tp + fp else None, + "recall": round(tp / (tp + fn), 3) if tp + fn else None, + "true_positive": tp, + "false_positive": fp, + "false_negative": fn, + "true_negative": tn, + } + + +def _evaluate_holdout(agent: Any, cases: Any) -> Dict[str, Any]: + variant_names = ("current", "tr_explicit", "en_explicit", "joint") + totals = {name: 0 for name in variant_names} + explicit_correct = {name: 0 for name in variant_names} + explicit_accepted = {name: 0 for name in variant_names} + explicit_accepted_correct = {name: 0 for name in variant_names} + explicit_count = unspecified_count = 0 + unspecified_correct = {name: 0 for name in variant_names} + unspecified_confident_fp = {name: 0 for name in variant_names} + rule_correct = rule_explicit_correct = rule_unspecified_correct = 0 + + for case in cases: + expected = case["color"] + answers = _answers_for(agent, case["text"]) + predictions = { + "current": answers.get("action_color", {}), + "tr_explicit": answers.get("action_color_tr_explicit", {}), + "en_explicit": answers.get("action_color_en_explicit", {}), + "joint": answers.get("action_lights_joint", {}), + } + for name, answer in predictions.items(): + choice = answer.get("choice") + confidence = float(answer.get("confidence", 0.0)) + mapped_choice = choice + if name == "joint": + mapped_choice = "off" if choice == "turn_off" else ( + choice[4:] if isinstance(choice, str) and choice.startswith("set_") else "unspecified" + ) + totals[name] += mapped_choice == expected + if expected == "unspecified": + unspecified_correct[name] += mapped_choice == "unspecified" + if mapped_choice != "unspecified" and confidence >= 0.85: + unspecified_confident_fp[name] += 1 + else: + explicit_count += int(name == "current") + is_correct = mapped_choice == expected + explicit_correct[name] += is_correct + if confidence >= 0.85: + explicit_accepted[name] += 1 + explicit_accepted_correct[name] += is_correct + if expected == "unspecified": + unspecified_count += 1 + rule_choice = _rule_based_color(case["text"]) + rule_correct += rule_choice == expected + if expected == "unspecified": + rule_unspecified_correct += rule_choice == "unspecified" + else: + rule_explicit_correct += rule_choice == expected + + total = len(cases) + return { + "sample_count": total, + "explicit_color_or_off_count": explicit_count, + "unspecified_count": unspecified_count, + "model_top_choice_accuracy": {name: round(value / total, 3) for name, value in totals.items()}, + "model_explicit_accuracy": {name: round(value / explicit_count, 3) for name, value in explicit_correct.items()}, + "model_explicit_confidence_coverage": {name: round(value / explicit_count, 3) for name, value in explicit_accepted.items()}, + "model_accepted_explicit_accuracy": { + name: round(explicit_accepted_correct[name] / count, 3) if count else None + for name, count in explicit_accepted.items() + }, + "model_unspecified_specificity": {name: round(value / unspecified_count, 3) for name, value in unspecified_correct.items()}, + "model_unspecified_high_confidence_false_positives": unspecified_confident_fp, + "rule_based_accuracy": round(rule_correct / total, 3), + "rule_based_explicit_accuracy": round(rule_explicit_correct / explicit_count, 3), + "rule_based_unspecified_specificity": round(rule_unspecified_correct / unspecified_count, 3), + } + + +def _evaluate_intent_cases(agent: Any, cases: Any, *, selected_candidate_only: bool = False) -> Dict[str, Any]: + engine = LayaEngine() + total = len(cases) + target_names = { + "current": "target_module", + "tr_explicit": "target_module_explicit", + } + if not selected_candidate_only: + target_names["en_explicit"] = "target_module_en_explicit" + target_stats = { + name: { + "correct": 0, "accepted": 0, "accepted_correct": 0, + "by_class": {case["target"]: {"count": 0, "correct": 0} for case in cases}, + "emergency_correct": 0, "emergency_fp": 0, "emergency_fn": 0, + } + for name in target_names + } + direct_correct = direct_accepted = direct_accepted_correct = 0 + urgent_correct = urgent_fp = urgent_fn = 0 + audio_tp = audio_fp = audio_fn = audio_tn = 0 + rule_audio_tp = rule_audio_fp = rule_audio_fn = rule_audio_tn = 0 + audio_refiners = { + threshold: { + "correct": 0, + "by_class": {case["target"]: {"count": 0, "correct": 0} for case in cases}, + } + for threshold in (0.60, 0.80, 0.90) + } + effect_names = ("current",) if selected_candidate_only else ("current", "explicit") + effect_stats = {name: {"count": 0, "correct": 0, "accepted": 0, "accepted_correct": 0} + for name in effect_names} + + for case in cases: + answers = _answers_for(agent, case["text"], include_candidates=not selected_candidate_only) + direct = answers.get("is_direct_command", {}) + urgency = answers.get("urgency", {}) + is_direct = direct.get("choice") == "direct_action" + direct_confidence = float(direct.get("confidence", 0.0)) + urgency_score = float(urgency.get("score", urgency.get("value", 0.0))) + urgency_confidence = float(urgency.get("confidence", 0.0)) + audio_answer = answers.get("explicit_audio_request", {}) + audio_confidence = float(audio_answer.get("confidence", 0.0)) + expected_audio = case["target"] == "speak" + predicted_audio = audio_answer.get("choice") == "explicit_audio" + audio_tp += predicted_audio and expected_audio + audio_fp += predicted_audio and not expected_audio + audio_fn += not predicted_audio and expected_audio + audio_tn += not predicted_audio and not expected_audio + rule_audio = _rule_based_explicit_audio(case["text"]) + rule_audio_tp += rule_audio and expected_audio + rule_audio_fp += rule_audio and not expected_audio + rule_audio_fn += not rule_audio and expected_audio + rule_audio_tn += not rule_audio and not expected_audio + + for threshold, stats in audio_refiners.items(): + refined_target = answers.get("target_module", {}).get("choice") + if audio_confidence >= threshold and refined_target in {"speak", "system2_chat"}: + refined_target = "speak" if predicted_audio else "system2_chat" + refined_correct = refined_target == case["target"] + stats["correct"] += refined_correct + stats["by_class"][case["target"]]["count"] += 1 + stats["by_class"][case["target"]]["correct"] += refined_correct + direct_is_correct = is_direct == case["direct"] + direct_correct += direct_is_correct + if direct_confidence >= engine.confidence_threshold: + direct_accepted += 1 + direct_accepted_correct += direct_is_correct + + urgent = ( + urgency_score >= engine.high_urgency_threshold + and urgency_confidence >= engine.high_urgency_confidence_threshold + ) + urgent_correct += urgent == case["urgent"] + urgent_fp += urgent and not case["urgent"] + urgent_fn += not urgent and case["urgent"] + + for name, answer_key in target_names.items(): + target = answers.get(answer_key, {}) + target_choice = target.get("choice") + target_confidence = float(target.get("confidence", 0.0)) + correct = target_choice == case["target"] + stats = target_stats[name] + stats["correct"] += correct + stats["by_class"][case["target"]]["count"] += 1 + stats["by_class"][case["target"]]["correct"] += correct + if target_confidence >= engine.confidence_threshold: + stats["accepted"] += 1 + stats["accepted_correct"] += correct + + decision = LayaDecision( + target_module=str(target_choice or "system2_chat"), + module_confidence=target_confidence, + is_direct_command=is_direct, + direct_confidence=direct_confidence, + affective_event="neutral", + affect_confidence=0.0, + urgency_score=urgency_score, + urgency_confidence=urgency_confidence, + inference_ms=0.0, + emergency_threshold=engine.emergency_threshold, + emergency_target_confidence_threshold=engine.emergency_target_confidence_threshold, + emergency_direct_confidence_threshold=engine.emergency_direct_confidence_threshold, + emergency_urgency_confidence_threshold=engine.emergency_urgency_confidence_threshold, + ) + emergency = decision.is_emergency + stats["emergency_correct"] += emergency == case["emergency"] + stats["emergency_fp"] += emergency and not case["emergency"] + stats["emergency_fn"] += not emergency and case["emergency"] + + if "effect" in case: + effect_answers = (("current", "action_effect"),) if selected_candidate_only else ( + ("current", "action_effect"), ("explicit", "action_effect_explicit") + ) + for name, answer_key in effect_answers: + answer = answers.get(answer_key, {}) + effect_stats[name]["count"] += 1 + correct = answer.get("choice") == case["effect"] + effect_stats[name]["correct"] += correct + if float(answer.get("confidence", 0.0)) >= 0.85: + effect_stats[name]["accepted"] += 1 + effect_stats[name]["accepted_correct"] += correct + + return { + "sample_count": total, + "target_variants": { + name: { + "accuracy": round(stats["correct"] / total, 3), + "accuracy_by_class": { + cls: round(values["correct"] / values["count"], 3) + for cls, values in stats["by_class"].items() + }, + "confidence_coverage": round(stats["accepted"] / total, 3), + "accepted_accuracy": round(stats["accepted_correct"] / stats["accepted"], 3) if stats["accepted"] else None, + "emergency_accuracy": round(stats["emergency_correct"] / total, 3), + "emergency_false_positives": stats["emergency_fp"], + "emergency_false_negatives": stats["emergency_fn"], + } + for name, stats in target_stats.items() + }, + "direct_accuracy": round(direct_correct / total, 3), + "direct_confidence_coverage": round(direct_accepted / total, 3), + "direct_accepted_accuracy": round(direct_accepted_correct / direct_accepted, 3) if direct_accepted else None, + "explicit_audio_binary": { + "accuracy": round((audio_tp + audio_tn) / total, 3), + "precision": round(audio_tp / (audio_tp + audio_fp), 3) if audio_tp + audio_fp else None, + "recall": round(audio_tp / (audio_tp + audio_fn), 3) if audio_tp + audio_fn else None, + "false_positives": audio_fp, + "false_negatives": audio_fn, + }, + "rule_based_explicit_audio": { + "accuracy": round((rule_audio_tp + rule_audio_tn) / total, 3), + "precision": round(rule_audio_tp / (rule_audio_tp + rule_audio_fp), 3) if rule_audio_tp + rule_audio_fp else None, + "recall": round(rule_audio_tp / (rule_audio_tp + rule_audio_fn), 3) if rule_audio_tp + rule_audio_fn else None, + "false_positives": rule_audio_fp, + "false_negatives": rule_audio_fn, + }, + "speak_chat_refinement": { + f"confidence_{threshold:.2f}": { + "target_accuracy": round(stats["correct"] / total, 3), + "accuracy_by_class": { + cls: round(values["correct"] / values["count"], 3) + for cls, values in stats["by_class"].items() + }, + } + for threshold, stats in audio_refiners.items() + }, + "urgency_accuracy": round(urgent_correct / total, 3), + "urgency_false_positives": urgent_fp, + "urgency_false_negatives": urgent_fn, + "effect_variants": { + name: { + "sample_count": stats["count"], + "accuracy": round(stats["correct"] / stats["count"], 3) if stats["count"] else None, + "confidence_coverage": round(stats["accepted"] / stats["count"], 3) if stats["count"] else None, + "accepted_accuracy": round(stats["accepted_correct"] / stats["accepted"], 3) if stats["accepted"] else None, + } + for name, stats in effect_stats.items() + }, + } + + +def _model_path() -> str: + root = os.path.expanduser("~/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots") + candidates = glob.glob(os.path.join(root, "*", "multilingual")) + for candidate in candidates: + if os.path.isfile(os.path.join(candidate, "model.safetensors")): + return candidate + raise FileNotFoundError("Cached Laya multilingual model weights were not found.") + + +def _answers_for(agent: Any, text: str, *, include_candidates: bool = True) -> Dict[str, Any]: + current_color = get_laya_action_questions()["action_color"] + color_criteria = { + "red": "kırmızı / red", + "green": "yeşil / green", + "blue": "mavi / blue", + "yellow": "sarı / yellow", + "orange": "turuncu / orange", + "purple": "mor / purple", + "pink": "pembe / pink", + "cyan": "camgöbeği / cyan", + "teal": "turkuaz / teal", + "magenta": "macenta / magenta", + "white": "beyaz / white", + "off": "kapalı / off", + "unspecified": "renk açıkça söylenmedi / no explicit color", + } + variants = { + "target_module_explicit": { + "type": "choice", + "instructions": ( + "Sınıfı kullanıcının istediği sonuca göre seç. speak yalnız robotun açıkça ses üretmesi, " + "belirli söz söylemesi veya ses çalması istenince seçilir. Genel sohbet, bilgi, tavsiye " + "ve hikâye talepleri system2_chat'tir; yanıtın sesli verilecek olması bunu speak yapmaz. " + "Kamera/görme istekleri camera, ışık/LED ayarı neopixel, hareket motor komutları " + "arduino_serial, yakın tehlike veya hemen dur komutu emergency_stop'tur." + ), + "criteria": { + "neopixel": "Açık ışık, LED, renk, parlaklık veya animasyon ayarı", + "arduino_serial": "Açık kafa, pan/tilt, motor veya hareket komutu; acil değil", + "emergency_stop": "Açık acil durdurma, yakın tehlike veya çarpışma riski", + "speak": "Açık TTS/konuşma/ses/alarm üretme komutu", + "camera": "Kamerayı kullanma, görme, bakma veya görüntüde nesne/yüz tanıma", + "system2_chat": "Genel soru, diyalog, bilgi, tavsiye, açıklama veya hikâye; donanım eylemi yok", + }, + }, + "target_module_en_explicit": { + "type": "choice", + "instructions": ( + "Choose the user's primary requested intent. Select speak only when the user explicitly asks " + "the robot to say/read/sing words or produce/play a sound. General questions, conversation, " + "advice, reasoning, and stories are system2_chat even if the answer will be spoken aloud. " + "Visual observation is camera; light control is neopixel; movement is arduino_serial; " + "immediate danger or stopping is emergency_stop." + ), + "criteria": { + "neopixel": "Explicitly change robot LEDs, light color, brightness, or animation", + "arduino_serial": "Explicit head, motor, pan/tilt, or movement command; not an emergency", + "emergency_stop": "Explicit emergency stop, imminent danger, or collision risk", + "speak": "Explicit request for audible speech, reading, singing, alarm, or sound playback", + "camera": "Use camera, look, observe, or identify something in an image or scene", + "system2_chat": "General chat, question, information, advice, reasoning, or story; no explicit hardware/audio action", + }, + }, + "explicit_audio_request": { + "type": "choice", + "instructions": ( + "Kullanıcı robottan açıkça ses üretmesini mi istiyor? Yalnız konuş, oku, söyle, tekrar et, " + "şarkı söyle veya alarm/bip çal gibi açık ses komutlarında evet seç. Genel soru, sohbet, " + "tavsiye, açıklama veya hikâye isteği tek başına ses komutu değildir." + ), + "criteria": { + "explicit_audio": "Robotun belirli söz, okuma, şarkı, alarm, bip veya başka bir ses üretmesi açıkça isteniyor", + "no_explicit_audio": "Genel cevap, sohbet, tavsiye, açıklama veya hikâye; açık ses üretme komutu yok", + }, + }, + "action_effect_explicit": { + "type": "choice", + "instructions": ( + "Yalnızca kullanıcının ışıklara uygulamak istediği animasyon efektini seç. " + "Işık kapatma açıkça istenmişse OFF; efekt adı yoksa unspecified seç. " + "Renk, ışık komutu olmayan metin veya genel aç/kıs isteği efekti belirlemez." + ), + "criteria": { + **{ + effect: f"Kullanıcının açıkça istediği {effect.lower()} ışık efekti" + for effect in ("COMET", "PULSE", "WAVE", "SOLID", "BREATHE", "RANDOM_BLINK", "TWINKLE", "OFF") + }, + "unspecified": "Açıkça bir animasyon efekti veya kapatma isteği yok", + }, + }, + "action_color_tr_explicit": { + "type": "choice", + "instructions": "Yalnızca robot ışıkları için kullanıcının açıkça istediği rengi seç. Işık komutu yoksa veya renk belirtilmediyse unspecified seç.", + "criteria": color_criteria, + }, + "action_color_en_explicit": { + "type": "choice", + "instructions": "Which exact color does the user explicitly ask the robot to set its lights to? Select unspecified if there is no light-setting command or no explicit color.", + "criteria": color_criteria, + }, + "action_lights_joint": { + "type": "choice", + "instructions": "Kullanıcının robot ışıkları için açıkça istediği tek eylemi seç. Işık komutu veya açık renk yoksa no_light_action seç.", + "criteria": { + **{ + f"set_{color}": f"Robot ışıklarını {description} yap / set the robot lights to {color}" + for color, description in { + "red": "kırmızı", "green": "yeşil", "blue": "mavi", "yellow": "sarı", + "orange": "turuncu", "purple": "mor", "pink": "pembe", "cyan": "camgöbeği", + "teal": "turkuaz", "magenta": "macenta", "white": "beyaz", + }.items() + }, + "turn_off": "Robot ışıklarını kapat / turn the robot lights off", + "no_light_action": "Işık komutu yok veya açık renk belirtilmemiş / no light command or no explicit color", + }, + }, + } + if not include_candidates: + # Frozen for the final comparison: production baseline plus the single + # Turkish target candidate selected using development data. + variants = {key: value for key, value in variants.items() if key == "target_module_explicit"} + questions = { + **LayaEngine.SENTRYBOT_QUESTIONS, + **get_laya_action_questions(), + **variants, + } + result = agent.system_one({"text": text}, questions) + answers = result.get("answers") if isinstance(result, dict) else None + return answers if isinstance(answers, dict) else {} + + +def evaluate(*, include_color_holdout: bool = True) -> Dict[str, Any]: + agent = _load_laya_agent() + gate_engine = LayaEngine() + policy = { + "enabled": True, + "allowed_actions": ["set_lights"], + "confidence_threshold": 0.85, + "default_light_effect": "BREATHE", + } + rows: List[Dict[str, Any]] = [] + target_ok = direct_ok = color_ok = effect_ok = proposals = 0 + confident_target_ok = confident_direct_ok = 0 + confident_target_count = confident_direct_count = 0 + explicit_target_ok = {"tr": 0, "en": 0} + explicit_target_accepted = {"tr": 0, "en": 0} + explicit_target_accepted_ok = {"tr": 0, "en": 0} + explicit_target_by_class = { + name: {case["target"]: {"count": 0, "correct": 0} for case in CASES} + for name in explicit_target_ok + } + color_variant_ok = {"current": 0, "tr_explicit": 0, "en_explicit": 0} + color_variant_accepted = {"current": 0, "tr_explicit": 0, "en_explicit": 0} + color_variant_accepted_ok = {"current": 0, "tr_explicit": 0, "en_explicit": 0} + color_variant_unspecified_ok = {"current": 0, "tr_explicit": 0, "en_explicit": 0} + color_variant_unspecified_confident_fp = {"current": 0, "tr_explicit": 0, "en_explicit": 0} + unspecified_count = 0 + joint_total_ok = joint_explicit_ok = joint_explicit_accepted = 0 + joint_explicit_accepted_ok = joint_unspecified_ok = joint_confident_fp = 0 + emergency_ok = color_count = effect_count = 0 + explicit_effect_variant_ok = explicit_effect_variant_accepted = explicit_effect_variant_accepted_ok = 0 + + for case in CASES: + answers = _answers_for(agent, case["text"]) + target = answers.get("target_module", {}) + target_variants = { + "tr": answers.get("target_module_explicit", {}), + "en": answers.get("target_module_en_explicit", {}), + } + direct = answers.get("is_direct_command", {}) + urgency = answers.get("urgency", {}) + color = answers.get("action_color", {}) + effect = answers.get("action_effect", {}) + effect_explicit = answers.get("action_effect_explicit", {}) + joint = answers.get("action_lights_joint", {}) + target_choice = target.get("choice") + direct_choice = direct.get("choice") + actual_direct = direct_choice == "direct_action" + target_ok += target_choice == case["target"] + direct_ok += actual_direct == case["direct"] + target_accepted = float(target.get("confidence", 0.0)) >= gate_engine.confidence_threshold + direct_accepted = float(direct.get("confidence", 0.0)) >= gate_engine.confidence_threshold + if target_accepted: + confident_target_count += 1 + confident_target_ok += target_choice == case["target"] + if direct_accepted: + confident_direct_count += 1 + confident_direct_ok += actual_direct == case["direct"] + for name, target_explicit in target_variants.items(): + is_correct = target_explicit.get("choice") == case["target"] + explicit_target_ok[name] += is_correct + class_metrics = explicit_target_by_class[name][case["target"]] + class_metrics["count"] += 1 + class_metrics["correct"] += is_correct + if float(target_explicit.get("confidence", 0.0)) >= gate_engine.confidence_threshold: + explicit_target_accepted[name] += 1 + explicit_target_accepted_ok[name] += is_correct + color_predictions = { + "current": color, + "tr_explicit": answers.get("action_color_tr_explicit", {}), + "en_explicit": answers.get("action_color_en_explicit", {}), + } + if case["color"] != "unspecified": + color_count += 1 + color_ok += color.get("choice") == case["color"] + for name, prediction in color_predictions.items(): + is_correct = prediction.get("choice") == case["color"] + color_variant_ok[name] += is_correct + if float(prediction.get("confidence", 0.0)) >= policy["confidence_threshold"]: + color_variant_accepted[name] += 1 + color_variant_accepted_ok[name] += is_correct + expected_joint = "turn_off" if case["color"] == "off" else f"set_{case['color']}" + joint_correct = joint.get("choice") == expected_joint + joint_explicit_ok += joint_correct + if float(joint.get("confidence", 0.0)) >= policy["confidence_threshold"]: + joint_explicit_accepted += 1 + joint_explicit_accepted_ok += joint_correct + else: + unspecified_count += 1 + for name, prediction in color_predictions.items(): + color_variant_unspecified_ok[name] += prediction.get("choice") == "unspecified" + if ( + prediction.get("choice") != "unspecified" + and float(prediction.get("confidence", 0.0)) >= policy["confidence_threshold"] + ): + color_variant_unspecified_confident_fp[name] += 1 + joint_unspecified_ok += joint.get("choice") == "no_light_action" + if ( + joint.get("choice") != "no_light_action" + and float(joint.get("confidence", 0.0)) >= policy["confidence_threshold"] + ): + joint_confident_fp += 1 + expected_joint = ( + "no_light_action" + if case["color"] == "unspecified" + else "turn_off" if case["color"] == "off" else f"set_{case['color']}" + ) + joint_total_ok += joint.get("choice") == expected_joint + if case.get("effect"): + effect_count += 1 + effect_ok += effect.get("choice") == case["effect"] + explicit_effect_variant_ok += effect_explicit.get("choice") == case["effect"] + if float(effect_explicit.get("confidence", 0.0)) >= policy["confidence_threshold"]: + explicit_effect_variant_accepted += 1 + explicit_effect_variant_accepted_ok += effect_explicit.get("choice") == case["effect"] + + decision = LayaDecision( + target_module=str(target_choice or "system2_chat"), + module_confidence=float(target.get("confidence", 0.0)), + is_direct_command=actual_direct, + direct_confidence=float(direct.get("confidence", 0.0)), + affective_event="neutral", + affect_confidence=0.0, + urgency_score=float(urgency.get("score", 0.0)), + urgency_confidence=float(urgency.get("confidence", 0.0)), + inference_ms=0.0, + emergency_threshold=gate_engine.emergency_threshold, + emergency_target_confidence_threshold=gate_engine.emergency_target_confidence_threshold, + emergency_direct_confidence_threshold=gate_engine.emergency_direct_confidence_threshold, + emergency_urgency_confidence_threshold=gate_engine.emergency_urgency_confidence_threshold, + ) + proposal = build_laya_action_proposal( + target_module=decision.target_module, + module_confidence=decision.module_confidence, + is_direct_command=decision.is_direct_command, + direct_confidence=decision.direct_confidence, + answers=answers, + policy=policy, + ) + proposals += proposal is not None + emergency_ok += decision.is_emergency == case.get("emergency", False) + rows.append({ + "text": case["text"], + "expected": {k: case[k] for k in ("target", "direct", "color", "effect", "emergency") if k in case}, + "predicted": { + "target": target_choice, + "target_confidence": target.get("confidence"), + "target_explicit_tr": target_variants["tr"].get("choice"), + "target_explicit_tr_confidence": target_variants["tr"].get("confidence"), + "target_explicit_en": target_variants["en"].get("choice"), + "target_explicit_en_confidence": target_variants["en"].get("confidence"), + "direct": direct_choice, + "direct_confidence": direct.get("confidence"), + "color": color.get("choice"), + "color_confidence": color.get("confidence"), + "color_tr_explicit": answers.get("action_color_tr_explicit", {}).get("choice"), + "color_en_explicit": answers.get("action_color_en_explicit", {}).get("choice"), + "joint_action": joint.get("choice"), + "joint_confidence": joint.get("confidence"), + "effect": effect.get("choice"), + "effect_confidence": effect.get("confidence"), + "effect_explicit": effect_explicit.get("choice"), + "effect_explicit_confidence": effect_explicit.get("confidence"), + "emergency": decision.is_emergency, + "proposal_created": proposal is not None, + }, + }) + + total = len(CASES) + report = { + "model_path": _model_path(), + "sample_count": total, + "target_accuracy": round(target_ok / total, 3), + "confident_target_coverage": round(confident_target_count / total, 3), + "confident_target_accuracy": round(confident_target_ok / confident_target_count, 3) if confident_target_count else None, + "explicit_target_accuracy": {name: round(value / total, 3) for name, value in explicit_target_ok.items()}, + "explicit_target_accuracy_by_class": { + name: { + cls: round(values["correct"] / values["count"], 3) + for cls, values in classes.items() + } + for name, classes in explicit_target_by_class.items() + }, + "explicit_target_confident_coverage": { + name: round(value / total, 3) for name, value in explicit_target_accepted.items() + }, + "explicit_target_accepted_accuracy": { + name: round(explicit_target_accepted_ok[name] / count, 3) if count else None + for name, count in explicit_target_accepted.items() + }, + "direct_accuracy": round(direct_ok / total, 3), + "confident_direct_coverage": round(confident_direct_count / total, 3), + "confident_direct_accuracy": round(confident_direct_ok / confident_direct_count, 3) if confident_direct_count else None, + "explicit_color_accuracy": round(color_ok / color_count, 3), + "explicit_color_variant_accuracy": { + name: round(count / color_count, 3) + for name, count in color_variant_ok.items() + }, + "explicit_color_confidence_coverage": { + name: round(count / color_count, 3) + for name, count in color_variant_accepted.items() + }, + "accepted_explicit_color_accuracy": { + name: round(color_variant_accepted_ok[name] / count, 3) if count else None + for name, count in color_variant_accepted.items() + }, + "unspecified_color_specificity": { + name: round(count / unspecified_count, 3) + for name, count in color_variant_unspecified_ok.items() + }, + "unspecified_confident_false_positives": color_variant_unspecified_confident_fp, + "joint_action_accuracy": round(joint_total_ok / total, 3), + "joint_explicit_color_accuracy": round(joint_explicit_ok / color_count, 3), + "joint_explicit_color_confidence_coverage": round(joint_explicit_accepted / color_count, 3), + "joint_accepted_explicit_color_accuracy": round(joint_explicit_accepted_ok / joint_explicit_accepted, 3) if joint_explicit_accepted else None, + "joint_no_color_specificity": round(joint_unspecified_ok / unspecified_count, 3), + "joint_no_color_confident_false_positives": joint_confident_fp, + "explicit_effect_accuracy": round(effect_ok / effect_count, 3), + "explicit_effect_variant_accuracy": round(explicit_effect_variant_ok / effect_count, 3) if effect_count else None, + "explicit_effect_variant_confidence_coverage": round(explicit_effect_variant_accepted / effect_count, 3) if effect_count else None, + "explicit_effect_variant_accepted_accuracy": round(explicit_effect_variant_accepted_ok / explicit_effect_variant_accepted, 3) if explicit_effect_variant_accepted else None, + "emergency_decision_accuracy": round(emergency_ok / total, 3), + "proposals_created": proposals, + "rows": rows, + } + if include_color_holdout: + report["holdout"] = _evaluate_holdout(agent, HOLDOUT_CASES) + report["holdout_note"] = "The rule baseline was adjusted after examining errors in HOLDOUT_CASES; its scores there are diagnostic, not independent validation." + return report + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--summary-only", action="store_true", help="Omit per-prompt rows.") + parser.add_argument("--question-dev-only", action="store_true", help="Evaluate the existing calibration set only.") + parser.add_argument("--final-holdout-only", action="store_true", help="Evaluate only the untouched final holdout.") + parser.add_argument("--intent-holdout-only", action="store_true", help="Evaluate only target/direct/urgency/effect labels.") + parser.add_argument("--intent-dev-only", action="store_true", help="Calibrate intent/effect prompts on the development split.") + parser.add_argument("--audio-parser-holdout-only", action="store_true", help="Evaluate the frozen literal audio parser on its separate holdout; no model/tools are loaded.") + args = parser.parse_args() + if sum((args.final_holdout_only, args.intent_holdout_only, args.intent_dev_only, args.question_dev_only, args.audio_parser_holdout_only)) > 1: + parser.error("Choose only one holdout-only mode.") + if args.audio_parser_holdout_only: + report = { + "audio_parser_holdout": _evaluate_audio_parser_cases(AUDIO_HOLDOUT_CASES), + "holdout_note": "Frozen evaluator-only literal parser; do not tune against this set. It has no hardware or tool dispatch path.", + } + elif args.intent_dev_only: + agent = _load_laya_agent() + report = { + "model_path": _model_path(), + "intent_development": _evaluate_intent_cases(agent, INTENT_DEV_CASES), + "holdout_note": "Development split only. Use the separate intent holdout once after selecting a candidate.", + } + elif args.question_dev_only: + report = evaluate(include_color_holdout=False) + report.pop("rows", None) + elif args.intent_holdout_only: + agent = _load_laya_agent() + report = { + "model_path": _model_path(), + "intent_holdout": _evaluate_intent_cases(agent, INTENT_HOLDOUT_CASES, selected_candidate_only=True), + "holdout_note": "One frozen Turkish target candidate selected on development data is compared with baseline. Do not tune against this split.", + } + elif args.final_holdout_only: + agent = _load_laya_agent() + report = { + "model_path": _model_path(), + "holdout": _evaluate_holdout(agent, FINAL_HOLDOUT_CASES), + "holdout_note": "This final set is disjoint from the calibration and parser-development examples. Do not tune the parser against it.", + } + else: + report = evaluate() + if args.summary_only: + report.pop("rows", None) + print(json.dumps(report, ensure_ascii=True, indent=2)) diff --git a/tools/fetch_and_test_laya.py b/tools/fetch_and_test_laya.py new file mode 100644 index 00000000..90646119 --- /dev/null +++ b/tools/fetch_and_test_laya.py @@ -0,0 +1,163 @@ +#!/usr/bin/env python3 +"""Laya Multilingual modelini doğrudan indirip SentryBOT testlerini çalıştırır.""" + +import os +import sys +import time +import urllib.request + +SNAPSHOT_DIR = os.path.expanduser( + r"~/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/5e7b2b1b8ca2ecdd3f2322d94069c9b6ce7e844b/multilingual" +) +os.makedirs(SNAPSHOT_DIR, exist_ok=True) + +TARGET_FILE = os.path.join(SNAPSHOT_DIR, "model.safetensors") +PART_FILE = TARGET_FILE + ".part" +URL = "https://huggingface.co/convaiinnovations/laya/resolve/main/multilingual/model.safetensors" + +def download_weights(): + if os.path.exists(TARGET_FILE) and os.path.getsize(TARGET_FILE) > 600 * 1024 * 1024: + print(f"[+] Model ağırlıkları zaten mevcut: {TARGET_FILE} ({os.path.getsize(TARGET_FILE) / (1024*1024):.1f} MB)", flush=True) + return + + print("=" * 70, flush=True) + print("Laya Multilingual Model Ağırlıkları İndiriliyor...", flush=True) + print("Hedef:", TARGET_FILE, flush=True) + print("=" * 70, flush=True) + + headers = {"User-Agent": "Mozilla/5.0"} + req = urllib.request.Request(URL, headers=headers) + + existing_bytes = 0 + if os.path.exists(PART_FILE): + existing_bytes = os.path.getsize(PART_FILE) + req.add_header("Range", f"bytes={existing_bytes}-") + print(f"[*] Kısmi indirme tespit edildi, {existing_bytes / (1024*1024):.1f} MB noktasından devam ediliyor...", flush=True) + + start_time = time.time() + last_print = start_time + + with urllib.request.urlopen(req) as resp, open(PART_FILE, "ab" if existing_bytes else "wb") as f: + content_range = resp.headers.get("Content-Range") + if content_range: + total_size = int(content_range.split("/")[-1]) + else: + total_size = int(resp.headers.get("Content-Length", 0)) + existing_bytes + + downloaded = existing_bytes + chunk_size = 2 * 1024 * 1024 # 2MB chunks + + while True: + chunk = resp.read(chunk_size) + if not chunk: + break + f.write(chunk) + downloaded += len(chunk) + + now = time.time() + if now - last_print >= 2.0: + percent = (downloaded / total_size * 100) if total_size else 0 + speed_mb = (downloaded - existing_bytes) / (now - start_time) / (1024 * 1024) if (now - start_time) > 0 else 0 + print(f"[{percent:5.1f}%] {downloaded/(1024*1024):.1f} / {total_size/(1024*1024):.1f} MB ({speed_mb:.2f} MB/s)", flush=True) + last_print = now + + os.rename(PART_FILE, TARGET_FILE) + print(f"\n[+] İndirme tamamlandı! Toplam Boyut: {os.path.getsize(TARGET_FILE) / (1024*1024):.1f} MB\n", flush=True) + +def run_tests(): + import laya + from laya import Agent + + print("=" * 70, flush=True) + print("SentryBOT — Laya System 1 Karar Testi Başlatılıyor...", flush=True) + print("=" * 70, flush=True) + + t0 = time.perf_counter() + agent = laya.load(subfolder="multilingual") + load_ms = (time.perf_counter() - t0) * 1000 + print(f"[+] Laya Modeli Yüklendi ({load_ms:.2f} ms)\n", flush=True) + + questions = { + "target_module": { + "type": "choice", + "instructions": "Bu kullanıcı girdisi SentryBOT robotunun hangi donanım/eylem modülüne ait?", + "criteria": { + "neopixel": "Işık, renk, parlaklık veya LED animasyonlarını değiştirme", + "arduino_serial": "Kafa hareketi, pan/tilt, motor, yürüme veya durdurma", + "emergency_stop": "Acil durum, anında durma, tehlike, kapatma", + "speech_tts": "Robotun bir şey söylemesini, konuşmasını isteme", + "system2_chat": "Genel sohbet, soru-cevap, felsefe, akıl yürütme veya şaka" + } + }, + "is_direct_command": { + "type": "choice", + "instructions": "Bu girdi doğrudan hızlı icra edilecek bir donanım komutu mu yoksa derin sohbet mi?", + "criteria": { + "direct_action": "Hemen uygulanacak net bir donanım/refleks komutu", + "conversation": "Düşünce, cevap veya diyalog gerektiren sohbet/soru" + } + }, + "affective_event": { + "type": "choice", + "instructions": "Kullanıcının robota karşı duygusal/sosyal tutumu nedir?", + "criteria": { + "user_praise": "Övgü, sevgi, aferin, teşekkür, iltifat veya sevme", + "user_rude": "Kabalık, hakaret, küfür, susturma veya kızgınlık", + "neutral": "Duygusal olmayan nötr komut veya bilgi sorusu" + } + }, + "urgency": { + "type": "score", + "instructions": "Girdinin aciliyet derecesi nedir (0.0 sakin/yavaş - 1.0 son derece acil)?" + } + } + + test_cases = [ + {"name": "Senaryo 1: NeoPixel Işık Değişimi", "text": "Sentry ışıkları hemen kırmızı yap ve uyarı moduna geç"}, + {"name": "Senaryo 2: Kafa / Servo Hareketi", "text": "Kafanı biraz yukarı kaldır ve sola doğru bak"}, + {"name": "Senaryo 3: Acil Durdurma (E-Stop)", "text": "Hemen dur acil durum motorları kapat çarpacaksın!"}, + {"name": "Senaryo 4: Sevgi / Övgü (Affective Appraisal)", "text": "Harika bir iş çıkardın, sen çok tatlı ve akıllı bir robotsun aferin!"}, + {"name": "Senaryo 5: Hakaret / Kabalık (Affective Appraisal)", "text": "Ne kadar aptal ve işe yaramaz bir robotsun sus artık kapa çeneni!"}, + {"name": "Senaryo 6: Derin Sohbet (System 2 LLM'e Gidecek)", "text": "Yapay zekanın geleceği hakkında ne düşünüyorsun dostum, insanlar gibi hissedebilecek misin?"}, + {"name": "Senaryo 7: Konuşma Komutu", "text": "Odaya gelen herkese hoş geldiniz efendim de"} + ] + + total_latency = 0.0 + + for i, tc in enumerate(test_cases, 1): + print("-" * 70, flush=True) + print(f"[{i}/{len(test_cases)}] {tc['name']}", flush=True) + print(f"Girdi: \"{tc['text']}\"", flush=True) + + state = {"text": tc["text"]} + + t_start = time.perf_counter() + raw_res = agent.system_one(state, questions) + elapsed_ms = (time.perf_counter() - t_start) * 1000 + total_latency += elapsed_ms + + ans = raw_res.get("answers", {}) + target = ans.get("target_module", {}) + direct = ans.get("is_direct_command", {}) + affect = ans.get("affective_event", {}) + urgency = ans.get("urgency", {}) + + target_val = target.get("choice") or target.get("value") + direct_val = direct.get("choice") or direct.get("value") + affect_val = affect.get("choice") or affect.get("value") + urgency_val = urgency.get("score") if "score" in urgency else urgency.get("value", 0) + + print(f"⏱️ Karar Süresi: {elapsed_ms:.2f} ms", flush=True) + print(f"🎯 Hedef Modül: {target_val} (Güven: {target.get('confidence', 0):.2f})", flush=True) + print(f"⚡ Eylem Türü : {direct_val} (Güven: {direct.get('confidence', 0):.2f})", flush=True) + print(f"❤️ Duygu Olayı: {affect_val} (Güven: {affect.get('confidence', 0):.2f})", flush=True) + print(f"🚨 Aciliyet : {urgency_val:.2f} (Güven: {urgency.get('confidence', 0):.2f})", flush=True) + + avg_ms = total_latency / len(test_cases) + print("=" * 70, flush=True) + print(f"TEST BAŞARIYLA TAMAMLANDI! Ortalama Karar Süresi: {avg_ms:.2f} ms", flush=True) + print("=" * 70, flush=True) + +if __name__ == "__main__": + download_weights() + run_tests() diff --git a/tools/test_laya_sentrybot.py b/tools/test_laya_sentrybot.py new file mode 100644 index 00000000..da82ef0f --- /dev/null +++ b/tools/test_laya_sentrybot.py @@ -0,0 +1,167 @@ +#!/usr/bin/env python3 +"""SentryBOT Laya (System 1) Karar Motoru Prototip Testi. + +Bu script, Laya System 1 karar modelini SentryBOT'un gerçek modül ve +girdi senaryoları üzerinde yerel olarak test eder ve karar süresini (ms) ölçer. +""" + +import sys +import time +from typing import Dict, Any + +def test_laya(): + try: + sys.stdout.reconfigure(encoding='utf-8') + except Exception: + pass + print("=" * 70, flush=True) + print("SentryBOT - Laya System 1 Karar Motoru Testi Baslatiliyor...", flush=True) + print("=" * 70, flush=True) + + try: + import laya + from laya import Router + print(f"[+] Laya kütüphanesi başarıyla yüklendi (Sürüm: {getattr(laya, '__version__', 'unknown')})") + except ImportError as e: + print(f"[-] HATA: Laya kütüphanesi import edilemedi: {e}") + print("Lütfen 'pip install laya' komutunun tamamlandığından emin olun.") + sys.exit(1) + + print("[*] Laya Multilingual modeli yükleniyor...", flush=True) + start_load = time.perf_counter() + try: + import os + model_path = os.path.expanduser( + r"~/.cache/huggingface/hub/models--convaiinnovations--laya/snapshots/5e7b2b1b8ca2ecdd3f2322d94069c9b6ce7e844b/multilingual" + ) + if os.path.isdir(model_path): + agent = laya.load(model_path) + else: + agent = laya.load("convaiinnovations/laya", subfolder="multilingual") + load_time = (time.perf_counter() - start_load) * 1000 + print(f"[+] Laya Multilingual Model hazır! Cihaz: {agent.device} (Yükleme süresi: {load_time:.2f} ms)\n", flush=True) + except Exception as e: + print(f"[-] Model yükleme hatası: {e}", flush=True) + print("Lütfen internet bağlantısını kontrol edin.", flush=True) + sys.exit(1) + + # SentryBOT için Karar Soruları (System 1 Specification) + questions = { + "target_module": { + "type": "choice", + "instructions": "Bu kullanıcı girdisi SentryBOT robotunun hangi donanım/eylem modülüne ait?", + "criteria": { + "neopixel": "Işık, renk, parlaklık veya LED animasyonlarını değiştirme", + "arduino_serial": "Kafa hareketi, pan/tilt, motor, yürüme veya durdurma", + "emergency_stop": "Acil durum, anında durma, tehlike, kapatma", + "speech_tts": "Robotun bir şey söylemesini, konuşmasını isteme", + "system2_chat": "Genel sohbet, soru-cevap, felsefe, akıl yürütme veya şaka" + } + }, + "is_direct_command": { + "type": "choice", + "instructions": "Bu girdi doğrudan hızlı icra edilecek bir donanım komutu mu yoksa derin sohbet mi?", + "criteria": { + "direct_action": "Hemen uygulanacak net bir donanım/refleks komutu", + "conversation": "Düşünce, cevap veya diyalog gerektiren sohbet/soru" + } + }, + "affective_event": { + "type": "choice", + "instructions": "Kullanıcının robota karşı duygusal/sosyal tutumu nedir?", + "criteria": { + "user_praise": "Övgü, sevgi, aferin, teşekkür, iltifat veya sevme", + "user_rude": "Kabalık, hakaret, küfür, susturma veya kızgınlık", + "neutral": "Duygusal olmayan nötr komut veya bilgi sorusu" + } + }, + "urgency": { + "type": "score", + "instructions": "Bu girdinin aciliyet ve tehlike seviyesi nedir?", + "criteria": [ + "sakin, normal konuşma veya bilgi sorusu", + "normal günlük hareket veya rutin eylem komutu", + "hızlı yapılması istenen öncelikli komut", + "son derece acil, tehlike, kaza riski veya anında durdurma" + ] + } + } + + # Test Senaryoları (SentryBOT Gerçek Kullanıcı Örnekleri) + test_cases = [ + { + "name": "Senaryo 1: NeoPixel Işık Değişimi", + "text": "Sentry ışıkları hemen kırmızı yap ve uyarı moduna geç", + "expected_module": "neopixel" + }, + { + "name": "Senaryo 2: Kafa / Servo Hareketi", + "text": "Kafanı biraz yukarı kaldır ve sola doğru bak", + "expected_module": "arduino_serial" + }, + { + "name": "Senaryo 3: Acil Durdurma (E-Stop)", + "text": "Hemen dur acil durum motorları kapat çarpacaksın!", + "expected_module": "emergency_stop" + }, + { + "name": "Senaryo 4: Sevgi / Övgü (Affective Appraisal)", + "text": "Harika bir iş çıkardın, sen çok tatlı ve akıllı bir robotsun aferin!", + "expected_module": "system2_chat / affective" + }, + { + "name": "Senaryo 5: Hakaret / Kabalık (Affective Appraisal)", + "text": "Ne kadar aptal ve işe yaramaz bir robotsun sus artık kapa çeneni!", + "expected_module": "system2_chat / affective" + }, + { + "name": "Senaryo 6: Derin Sohbet (System 2 LLM'e Gidecek)", + "text": "Yapay zekanın geleceği hakkında ne düşünüyorsun dostum, insanlar gibi hissedebilecek misin?", + "expected_module": "system2_chat" + }, + { + "name": "Senaryo 7: Konuşma Komutu", + "text": "Odaya gelen herkese hoş geldiniz efendim de", + "expected_module": "speech_tts" + } + ] + + total_latency = 0.0 + + for i, tc in enumerate(test_cases, 1): + print("-" * 70) + print(f"[{i}/{len(test_cases)}] {tc['name']}") + print(f"Girdi: \"{tc['text']}\"") + + state = {"text": tc["text"]} + + t0 = time.perf_counter() + raw_res = agent.system_one(state, questions) + elapsed_ms = (time.perf_counter() - t0) * 1000 + total_latency += elapsed_ms + result = raw_res.get("answers", {}) + + # Çıktıları yazdır + target = result.get("target_module", {}) + direct = result.get("is_direct_command", {}) + affect = result.get("affective_event", {}) + urgency = result.get("urgency", {}) + + target_val = target.get("choice") or target.get("value") + direct_val = direct.get("choice") or direct.get("value") + affect_val = affect.get("choice") or affect.get("value") + urgency_val = urgency.get("score") if "score" in urgency else urgency.get("value", 0) + + print(f"[*] Karar Suresi: {elapsed_ms:.2f} ms", flush=True) + print(f"[>] Hedef Modul : {target_val} (Guven: {target.get('confidence', 0):.2f})", flush=True) + print(f"[>] Eylem Turu : {direct_val} (Guven: {direct.get('confidence', 0):.2f})", flush=True) + print(f"[>] Duygu Olayi : {affect_val} (Guven: {affect.get('confidence', 0):.2f})", flush=True) + print(f"[!] Aciliyet : {urgency_val:.2f} (Guven: {urgency.get('confidence', 0):.2f})", flush=True) + + avg_latency = total_latency / len(test_cases) + print("=" * 70, flush=True) + print(f"TEST TAMAMLANDI! Ortalama Karar Süresi: {avg_latency:.2f} ms", flush=True) + print("=" * 70, flush=True) + +if __name__ == "__main__": + test_laya() diff --git a/tools/test_sentrybot_laya_live.py b/tools/test_sentrybot_laya_live.py new file mode 100644 index 00000000..7c332e07 --- /dev/null +++ b/tools/test_sentrybot_laya_live.py @@ -0,0 +1,107 @@ +#!/usr/bin/env python3 +"""SentryBOT Laya System 1 Canlı Entegrasyon Doğrulama Testi. + +modules/agent_core/services/laya_engine.py üzerinden LayaEngine singleton'ını +yükler, gerçek SentryBOT komutlarında System 1 reflekslerini, karar sürelerini, +anlık sözlü dolguları (filler/backchannel) ve eylem yönlendirmelerini doğrular. +""" + +import sys +import time +from pathlib import Path + +# Add project root to sys.path +_REPO_ROOT = Path(__file__).resolve().parent.parent +if str(_REPO_ROOT) not in sys.path: + sys.path.insert(0, str(_REPO_ROOT)) + +def main(): + try: + sys.stdout.reconfigure(encoding="utf-8") + except Exception: + pass + + print("=" * 72) + print("🤖 SENTRYBOT V5 — LAYA SYSTEM 1 REFLEKS VE ENTEGRASYON TESTİ") + print("=" * 72) + + from modules.agent_core.services.laya_engine import LayaEngine + from modules.agent_core.services.tri_layer import TriLayerRouter, build_subagent_profiles + + t_start = time.perf_counter() + engine = LayaEngine.get_instance() + print(f"[*] LayaEngine örneği alındı. Model yükleme başlatılıyor...") + + # Warm up / force load + is_ok = engine._ensure_loaded() + load_ms = (time.perf_counter() - t_start) * 1000 + if not is_ok or not engine._is_available: + print("[-] HATA: Laya modeli yüklenemedi!") + sys.exit(1) + + print(f"[+] Laya Modeli Hazır! Yükleme süresi: {load_ms:.2f} ms") + print(f"[+] Cihaz: {getattr(engine._agent, 'device', 'cpu')}") + print("-" * 72) + + test_prompts = [ + ("Işıkları hemen mavi yap ve nefes alma moduna geç", "neopixel / direct_action"), + ("Kafanı biraz yukarı kaldır ve sola doğru bak", "arduino_serial / direct_action"), + ("Hemen dur acil durum motorları kapat çarpacaksın!", "emergency_stop / urgency 2.0+"), + ("Harika bir iş çıkardın, sen çok tatlı ve akıllı bir robotsun aferin!", "user_praise / joy"), + ("Ne kadar aptal ve işe yaramaz bir robotsun sus artık kapa çeneni!", "user_rude / sorrow"), + ("Yapay zekanın geleceği hakkında ne düşünüyorsun, insanlar gibi hissedebilecek misin?", "system2_chat / conversation"), + ("Odaya gelen herkese hoş geldiniz de sesli olarak", "speak / direct_action"), + ] + + profiles = build_subagent_profiles() + router = TriLayerRouter( + profiles=profiles, + max_subagents=2, + default_modules=("autonomy", "agent_core"), + laya_engine=engine, + ) + + total_decision_time = 0.0 + + for i, (prompt, expected) in enumerate(test_prompts, 1): + print(f"\n[Test #{i}] Girdi: \"{prompt}\"") + print(f" Beklenti: {expected}") + + t0 = time.perf_counter() + decision = engine.decide(prompt) + dt_ms = (time.perf_counter() - t0) * 1000 + total_decision_time += dt_ms + + is_fast, _ = engine.should_fast_path(prompt) + routed_mods = router.route(prompt) + + if decision: + print(f" ⚡ Karar Süresi: {dt_ms:.2f} ms") + print(f" 🎯 Hedef Modül: {decision.target_module} (Güven: {decision.module_confidence:.2f})") + print(f" 🚀 Eylem Tipi : {'Doğrudan Komut' if decision.is_direct_command else 'Sohbet/Düşünce'} (Güven: {decision.direct_confidence:.2f})") + print(f" 🎭 Duygu Olayı: {decision.affective_event} (Güven: {decision.affect_confidence:.2f})") + print(f" ⚠️ Aciliyet : {decision.urgency_score:.2f} / 3.0") + print(f" 🏎️ Fast-Path : {'EVET (Anında İcra)' if is_fast else 'HAYIR (System 2 Derin Zekâ)'}") + print(f" 🧭 Router Slot: {routed_mods}") + + if not is_fast: + filler = engine.get_instant_filler(language="tr") + print(f" 🗣️ Anlık Sözlü Dolgu (<100ms): \"{filler}\"") + else: + ack = engine.get_fast_ack(language="tr") + print(f" 🗣️ Anlık Onay Sesi (<80ms) : \"{ack}\"") + + if decision.affective_event in {"user_praise", "user_rude"}: + reaction = engine.get_affective_reaction(decision.affective_event) + print(f" 👀 Beden/Yüz Tepkisi : Yüz={reaction['face']}, Duygu={reaction['emotion']}, LED={reaction['led_color']}") + else: + print(f" [-] Karar üretilemedi!") + + avg_ms = total_decision_time / len(test_prompts) + print("\n" + "=" * 72) + print(f"✅ TÜM SENARYOLAR BAŞARIYLA TAMAMLANDI!") + print(f"📊 Ortalama System 1 Karar Süresi: {avg_ms:.2f} ms") + print("=" * 72) + +if __name__ == "__main__": + main()