Skip to content

LLM over-uses query_wikipedia instead of checking cultural memory for NZ domain terms #1074

@lokhor

Description

@lokhor

Summary

Observed across 7/10 device test scenarios for STT normalisation (PR #1070). When the user asks about a Māori/NZ cultural term (wharepaku, chocka, taniwha, kumara), the model calls query_wikipedia instead of checking the nz_truth_memories corpus — even though those terms are already in core/inference/src/main/assets/nz_truth_memories.json:2333-2388.

Root cause

The problem is prompt/tool-selection, not missing data. The cultural memory corpus already contains entries for all four terms. The model chooses Wikipedia over memory retrieval, often with worse results (e.g. Wikipedia returns the Māori language article for wharepaku, which is semantically correct but not useful).

Observed examples

  • "what is a fattybaku" → wharepaku → Wikipedia (Māori language article) instead of cultural memory
  • "what is chocka" → Wikipedia instead of cultural memory (which has the NZ slang definition)
  • "tell me about taniwha" → Wikipedia instead of cultural memory

Suggested fix

Either:

  1. System prompt guidance to search memory before Wikipedia for NZ/cultural domain terms
  2. Re-rank tool selection to prefer memory retrieval when the query contains recognised Māori/NZ cultural terms
  3. Lower the threshold for searchMemory vs query_wikipedia in the tool-selection decision

Acceptance criteria

  • Querying "what is a wharepaku" returns the cultural memory entry, not Wikipedia
  • Querying "what is chocka" returns the NZ slang definition from cultural memory
  • Wikipedia is still used for non-NZ terms (e.g. "quantum physics")
  • No regression in Wikipedia tool use for appropriate queries

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority:mediumUseful, schedule when reasonablesize:MMedium — moderate complexity, 1-2 daysvoice

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions