-
Notifications
You must be signed in to change notification settings - Fork 0
LLM response parsing lacks schema validation in beliefs and self-analysis #89
Copy link
Copy link
Open
Description
Problem
In beliefs.ts (checkBeliefUpdate) and self-analysis.ts (parseAnalysis), the code parses LLM responses using extractJson but does not validate that the resulting object contains the required fields. For beliefs, we need newBeliefs and changelogEntry; for self-analysis, we need reflection and issues. If the LLM omits a field or returns an unexpected shape, the code will throw (e.g., calling .trim() on undefined) and crash the entire process. This fragility could cause daily runs to fail silently or with uncaught exceptions.
What to build
Use the existing validateSchema function from json-utils.ts to validate parsed data before use:
- In
checkBeliefUpdate, after obtainingparseResult.data, callvalidateSchema(result, { requiredFields: [{name: "newBeliefs", type: "string"}, {name: "changelogEntry", type: "string"}] }). - In
parseAnalysis(self-analysis), similarly validate forreflection(string) andissues(array of objects withtitleandbody). - If validation fails, log a warning and skip the update/issue-opening instead of throwing.
Done when
- Both modules gracefully handle malformed LLM responses by skipping the operation and logging a clear error.
- No uncaught exceptions occur due to missing fields.
- The error logger records validation failures for debugging.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels