Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions LIMITATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,11 @@ do. Read this before trusting any single output.
- Four built-in claims (route-test-coverage, admin-authorization,
billing-webhook-signature, jwt-authentication). User-defined claims are not
supported yet, deliberately.
- Route test association is established from test imports only, and a static
import association is not execution coverage. Tests that exercise a route
through a running server (supertest, TestClient) or through an unresolved
helper are reported as unassociated, so this claim currently abstains on many
real repositories.
- Route test association accepts a test that requests the exact route path or
imports the route's implementation. It is still a static association, not
execution coverage. Tests that build URLs dynamically (an f-string or a
settings variable) and routes declared relative to an unresolved router mount
prefix are reported as unresolved rather than untested.
- Admin authorization is established only from a guard at the route's own call
site. Guards applied by a router mount, a server-wide middleware, or a
framework decorator are reported as unresolved. WEAK means DevTime found no
Expand Down
4 changes: 2 additions & 2 deletions QUICKSTART.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ python -m venv .venv
source .venv/bin/activate # Windows (PowerShell): .venv\Scripts\Activate.ps1
# Windows (Git Bash): source .venv/Scripts/activate
pip install -e ".[dev]"
pytest # optional: all tests pass (157 at v0.5.1)
pytest # optional: all tests pass (173 at v0.6.0)
```

## 3. Create the demo repo
Expand Down Expand Up @@ -132,7 +132,7 @@ A fresh-clone check was run on the current candidate:
- **OS:** Windows 11 (Git Bash)
- **Python:** 3.11.9
- **Install:** `pip install -e ".[dev]"`
- **Tests:** all passing (157 at v0.5.1)
- **Tests:** all passing (173 at v0.6.0)
- **Demo:** `dtc init` / `dtc scan` / `dtc concepts` / `dtc explain "Billing Webhooks"`
all produced the expected output from a clean `git clone`.

Expand Down
98 changes: 98 additions & 0 deletions RELEASE_NOTES_v0.6.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# DevTime v0.6.0 - freshness you can rely on, and honest scan state

v0.5.1 fixed results that were wrong. This release fixes results that were
*stale while claiming to be current*, which is the same failure wearing a
timestamp.

## A verified claim now goes stale when its real dependencies change

Verification recorded only the bounded list of evidence shown to a user, and
used that as the invalidation set. The two are not the same thing. Consequences,
all reproduced before being fixed:

- Editing the **test** that established a route/test association left the result
reported as FRESH. Only route files were fingerprinted.
- **Deleting** an evidence file left the result FRESH, because the old row
survived in the scan database and its hash still matched.
- Adding a **new route** left a claim about "all routes" FRESH, because no
previously recorded file had changed.

Results now record a complete dependency set, separate from displayed evidence:
the handler files, the tests, the guards and helpers that justified the
conclusion, and a fingerprint of the claim's surface set. A dependency that
changed, disappeared, or was not seen by the latest scan marks the claim STALE
and names the file. A result that recorded nothing to justify it is reported as
NEEDS_VERIFICATION instead of being assumed fresh.

Precision is preserved: unrelated edits still do not invalidate anything.

## dtc verify says which snapshot it used

Verification recomputes from the last persisted scan. That is not your working
tree, and stamping a fresh evaluation timestamp on old evidence implies work
that never happened. DevTime now checks and reports the relationship:

```text
These results were computed from a scan that no longer matches your working tree.
changed since the scan: src/billing/stripe-webhook.ts
Run dtc scan to verify against current code.
```

- JSON output gains `evidence_snapshot`: scan id, scan time, files scanned, and
the working-tree relationship (`matches_scan`, `changed_since_scan`,
`partially_checked`, `never_scanned`).
- The MCP `verify_claim` tool adds an explicit `staleness_warning`, because an
agent cannot see the user's files and must be told which snapshot it is
reasoning about and how to refresh.
- The working-tree check is read-only and bounded, so verification never turns
into a second full scan.

## Route test association is useful again, legitimately

v0.5.1 removed the false positives but could only see imports, so it abstained
on the most common real pattern: a test that drives a running app by URL. It now
accepts a test that **requests the exact route path** (supertest,
FastAPI TestClient) as well as one that imports the implementation.

Route identity still keeps the HTTP method, so a test requesting `GET /users`
establishes nothing about `POST /users`. Name similarity remains a suggestion
that can never raise the status.

## Unresolved is not the same as untested

Routes declared relative to a router mount prefix (`APIRouter()` with
`include_router(..., prefix=...)`) have no knowable full URL. Reporting them as
"no test found" blames the repository for a gap in DevTime's analysis. They are
now reported separately:

```text
10 route(s) are declared relative to a router mount prefix that DevTime did not
resolve, so their full URL is unknown. That is a gap in this analysis, not
evidence that they lack tests.
```

## Compatibility

- The `verifications` table gains `inventory_fingerprint`, added by an
idempotent migration. Existing rows keep NULL and are re-verified rather than
assumed fresh. Recorded decisions and scan history are untouched.
- JSON stays `schema_version: 2`; `evidence_snapshot` and `dependency_count` are
additive.
- No command, claim id, or MCP tool was renamed or removed.
- Test signals gain `requests` metadata; existing databases are re-scanned
normally.

## Notes

- 173 passing tests (16 new), including every reproduction above: test edited,
evidence deleted, new route added, unrelated change ignored, empty dependency
set, idempotent migration, stale CLI output, stale MCP disclosure, supertest
and TestClient association, method separation, and non-URL `.get()` calls that
must not be mistaken for requests.
- Known gaps, reported rather than assumed: router-level and application-level
guards, verification reached through an imported helper, dynamic request URLs,
and unresolved mount prefixes.

## Names

- PyPI distribution: `devtime-ei`. Python import: `devtime`. CLI: `dtc`.
53 changes: 41 additions & 12 deletions VERIFICATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,12 +49,20 @@ same time: the last verification supported it, but its evidence changed since.

| Freshness | Meaning |
|-----------|---------|
| FRESH | Evidence files are unchanged since the last verification. |
| STALE | At least one evidence file changed or disappeared. Re-verify. |
| NEEDS_VERIFICATION | The claim has never been verified in this repository. |
| FRESH | Every dependency is unchanged since the last verification. |
| STALE | A dependency changed, disappeared, or the claim's surface set changed. |
| NEEDS_VERIFICATION | Never verified here, or nothing was recorded to justify it. |

Freshness only tracks files that were evidence for the claim. Unrelated changes
never mark a claim stale.
Freshness is computed from a claim's **dependencies**, which are not the same as
the evidence displayed to you. Displayed evidence is a bounded selection; the
dependency set is complete and includes the test files, guards, and helpers that
justified the conclusion. A claim about a set (for example "all routes") also
depends on that set, so a newly added route invalidates it even though no
recorded file changed.

Only a claim's own dependencies participate, so unrelated edits never mark it
stale. A result that recorded no dependencies is reported as
NEEDS_VERIFICATION rather than assumed fresh.

NOT_APPLICABLE matters as much as the others. A repository with no billing code
is not "unknown" for a billing claim; the claim simply does not apply, and saying
Expand All @@ -63,13 +71,17 @@ so plainly is more useful than an ominous UNKNOWN.
## Built-in claims

- **route-test-coverage** (id kept for compatibility; now presented as *Route
Test Association*) - "HTTP routes have tests that import their
implementation." Association is established from test imports only, matched on
exact module stems so `users` does not match `superusers`. Route identity keeps
the HTTP method. A test that merely shares a word with a route path is reported
as an unverified suggestion and can never raise the status. Routes defined in
test, example, or fixture files are not application surface and are excluded
from the inventory. A static association is not execution coverage.
Test Association*) - "HTTP routes are referenced by tests that request or
import them." Two evidence levels are accepted: a test that requests the exact
route path (supertest, FastAPI TestClient), or a test that imports the route's
implementation module, matched on exact module stems so `users` does not match
`superusers`. Route identity keeps the HTTP method, so a GET test establishes
nothing about POST on the same path. A test that merely shares a word with a
route path is reported as an unverified suggestion and can never raise the
status. Routes declared relative to an unresolved mount prefix are reported as
unresolved rather than untested. Routes defined in test, example, or fixture
files are excluded from the inventory. A static association is not execution
coverage.
- **admin-authorization** (v0.5) - "Administrative routes require an
authorization check." Authorization is established only from a guard applied at
the route's own call site. Authentication is not authorization: `requireAuth`
Expand Down Expand Up @@ -120,6 +132,23 @@ When no claim applies, DevTime does not dead-end. It reports what it scanned,
what evidence it collected, what would make a claim verifiable, and states
plainly that this is a coverage limit rather than a verdict on your code.

## Scan freshness (v0.6)

Verification recomputes conclusions from the last persisted scan, which is not
the same thing as your working tree. When the two disagree, DevTime says so
before showing any result:

```text
These results were computed from a scan that no longer matches your working tree.
changed since the scan: src/billing/stripe-webhook.ts
Run dtc scan to verify against current code.
```

JSON output carries the same facts under `evidence_snapshot` (scan id, scan
time, files scanned, and the working-tree relationship), and the MCP
`verify_claim` tool adds an explicit `staleness_warning` so an agent that cannot
see your files is told which snapshot it is reasoning about.

## Trust model

- Deterministic and rule-driven. No AI, no network, no code execution.
Expand Down
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "devtime-ei"
version = "0.5.1"
version = "0.6.0"
description = "Local-first Engineering Intelligence for software repositories"
readme = "README.md"
requires-python = ">=3.11"
Expand Down Expand Up @@ -50,7 +50,7 @@ dev = [
Homepage = "https://github.com/Shakargy/devtime"
Repository = "https://github.com/Shakargy/devtime"
Issues = "https://github.com/Shakargy/devtime/issues"
"Release Notes" = "https://github.com/Shakargy/devtime/releases/tag/v0.5.1"
"Release Notes" = "https://github.com/Shakargy/devtime/releases/tag/v0.6.0"
Demo = "https://youtu.be/1Hiu3Y9J_SI"

[project.scripts]
Expand Down
4 changes: 2 additions & 2 deletions server.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@
"source": "github"
},
"websiteUrl": "https://github.com/Shakargy/devtime",
"version": "0.5.1",
"version": "0.6.0",
"packages": [
{
"registryType": "pypi",
"identifier": "devtime-ei",
"version": "0.5.1",
"version": "0.6.0",
"transport": {
"type": "stdio"
}
Expand Down
2 changes: 1 addition & 1 deletion src/devtime/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""DevTime - local-first Engineering Intelligence for repository memory."""

__version__ = "0.5.1"
__version__ = "0.6.0"

# Version metadata (Builder Edition, Chapter 20).
EVIDENCE_MODEL = "2026.06.1"
Expand Down
31 changes: 31 additions & 0 deletions src/devtime/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -157,23 +157,54 @@ def verify(
if result.status in ver.APPLICABLE_STATUSES:
ver.save_verification(conn, result)

state = ver.scan_state(conn)

if as_json:
console.print_json(
_json.dumps(
{
"schema_version": "2",
"command": "verify",
"evidence_snapshot": state,
"results": [r.to_dict() for r in results],
}
)
)
return

_print_scan_state(state)
_print_report(results, single=bool(claim))
finally:
conn.close()


def _print_scan_state(state: dict) -> None:
"""Say plainly which snapshot these conclusions came from.

Verification reads the last persisted scan, not the working tree. When the
two disagree, that must be the first thing a reader sees.
"""
tree = state.get("working_tree")
if tree == "changed_since_scan":
changed = state.get("changed_paths") or []
console.print(
"[yellow]These results were computed from a scan that no longer "
"matches your working tree.[/yellow]"
)
for p in changed[:5]:
console.print(f" changed since the scan: {p}", markup=False)
if len(changed) > 5:
console.print(f" ... and {len(changed) - 5} more", markup=False)
console.print("Run [bold]dtc scan[/bold] to verify against current code.")
console.print("")
elif tree == "partially_checked":
console.print(
"[dim]Working-tree comparison was capped; some files were not "
"re-checked against the scan.[/dim]"
)
console.print("")


def _print_report(results: list, single: bool) -> None:
"""Report card: what DevTime can and cannot verify about this repository."""
from devtime.intelligence import verification as ver
Expand Down
Loading
Loading