Generated HTML report is inaccurate and lacks AI summary of completed work

[caoergou]

Problem

The HTML report generated by owloop go / owloop run is mostly mechanical numbers and, in a real end-to-end run, even those numbers are wrong or misleading.

Observed report: /home/xyz/EricCao/xyzrobotics/xyz-data-website/.owloop/reports/owloop_report_latest.html

Specific defects

1. Spec status is wrong

   • All 5 specs are shown as pending even though every one of them completed successfully.
   • There is no per-spec summary of what changed or whether its acceptance criteria passed.

2. Diff summary is not the actual diff of this run

   • The report lists unrelated files such as backend/app/api/poster.py, frontend poster-template-sort changes, Docker files, and bitbucket-pipelines.yml.
   • The real owloop commits touched only 8 files (+338 / -297), but the report claims 25 files (+476 / -11).
   • Root cause: the report appears to be computed against master (or the wrong base) and includes commits that have nothing to do with the owloop run.

3. Commits section shows the wrong commits

   • It displays the last 5 commits on master (merge PRs from other work), not the 5 commits created by owloop on the owloop branch.

4. No AI-generated human-readable summary

   • The user cannot tell from the report what was actually accomplished.
   • There is no explanation of:
     • which service modules were extracted
     • which duplicate patterns were removed
     • which API files were refactored
     • what verification results were (pytest, ruff, mypy)
     • what blockers or learnings were recorded

Why this matters

The report is the main artifact the user reviews after an autonomous run. If it is inaccurate and lacks context, the user has to manually run git log, git diff, and inspect each commit to understand what happened — defeating the purpose of an autonomous loop.

Suggested improvements

1. Fix the base / branch for diff and commit listings

   • Report should compare the owloop branch against its merge base, not against HEAD of the current checkout.
   • Commits table should list only commits created by the current owloop run.

2. Correct spec status

   • Each spec should show complete / failed / in-progress based on actual execution.
   • Include iteration count and token cost per spec.

3. Add an AI-generated "What changed" summary

   • After the loop finishes, ask the agent to produce a short paragraph summary per spec:
     • What was the goal
     • What files were changed
     • What duplicate code was removed / what service was extracted
     • Verification results
   • Include this summary at the top of the report.

4. Add a "Key learnings / blockers" section

   • Surface <learning> tags collected during the run so the user sees caveats (e.g. "bulk_save_objects site left untouched because helper uses db.session.add").

5. Sanity-check numbers

   • Diff stat, token total, and commit count should be cross-checked and flagged if they look inconsistent (e.g. spec status pending but loop status completed).

Related

• Meta issue: owloop go end-to-end UX issues (Meta: owloop go end-to-end UX issues discovered during refactor test #75)

[caoergou]

Source-level root causes confirmed for the three wrong sections (same run, moved here from #78 during dedup):

1. Commits / diff: report.py:101 — get_recent_commits(self.project_dir, iterations) takes "the last N commits of whatever the main repo has checked out" (master at the time), not the run branch. That's why the table showed unrelated merge PRs and the diff summed their stats (double-counting merges). Fix: record last_good commit + branch head in the session file and render strictly from last_good..head.
2. Spec status: ReportGenerator reads the main-repo copy of .owloop/specs/; the engine marks COMPLETE only in the worktree copy. Only session_latest.json flows back today.
3. Timeline (3 events): the event stream physically split across three files — main repo .owloop/logs/events.jsonl (3 pre-worktree events), worktree root logs/events.jsonl (1 event — which got committed, see Gate integrity: vacuous acceptance criteria, Exclusions enforcement never worked (regression of #66), owloop metadata committed into user repos #78 finding 3), and the worktree's .owloop/logs/ (the rest). Root cause: paths.py legacy fallback + log_dir being a @property re-resolved per call against a mutable cwd.
4. Re: the AI-summary ask — the run emitted 5 high-value <learning> tags (e.g. the bulk_save_objects incompatibility, delete_issue's silent no-op semantics) that never left the terminal log. Surfacing collected learnings/blockers would cover most of the "what actually happened" gap without an extra LLM call.

Structural fix proposal (single run ledger rendered as the report's only source): #78 design addendum.