Skip to content

LLM response parsing lacks schema validation in beliefs and self-analysis #89

@crest-agent

Description

@crest-agent

Problem

In beliefs.ts (checkBeliefUpdate) and self-analysis.ts (parseAnalysis), the code parses LLM responses using extractJson but does not validate that the resulting object contains the required fields. For beliefs, we need newBeliefs and changelogEntry; for self-analysis, we need reflection and issues. If the LLM omits a field or returns an unexpected shape, the code will throw (e.g., calling .trim() on undefined) and crash the entire process. This fragility could cause daily runs to fail silently or with uncaught exceptions.

What to build

Use the existing validateSchema function from json-utils.ts to validate parsed data before use:

  • In checkBeliefUpdate, after obtaining parseResult.data, call validateSchema(result, { requiredFields: [{name: "newBeliefs", type: "string"}, {name: "changelogEntry", type: "string"}] }).
  • In parseAnalysis (self-analysis), similarly validate for reflection (string) and issues (array of objects with title and body).
  • If validation fails, log a warning and skip the update/issue-opening instead of throwing.

Done when

  • Both modules gracefully handle malformed LLM responses by skipping the operation and logging a clear error.
  • No uncaught exceptions occur due to missing fields.
  • The error logger records validation failures for debugging.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions