Repository navigation
feat(plugins): add standalone Jev context pruning example - #5731
Conversation
github-actions
Bot
added
area:deps
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Six unresolved moderate findings must be addressed before approval.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
What changed in this PR
Adds an opt-in standalone Jev extension for pruning obsolete read-only tool results before native summarization.
Changes:
- Adds the installable extension, plugin registration, and metadata.
- Implements bounded sync/async pruning with cooldowns and fail-open behavior.
- Adds documentation, attribution, live verification, and integration tests.
| File | Summary |
|---|---|
examples/deerflow-extension-jev-context/THIRD_PARTY_NOTICES.md |
Upstream attribution and license; no findings. |
examples/deerflow-extension-jev-context/scripts/verify_live.py |
Synthetic live verification script; no findings. |
examples/deerflow-extension-jev-context/README.md |
Installation and behavior documentation; no findings. |
examples/deerflow-extension-jev-context/pyproject.toml |
Moderate (1 vote): add installed-distribution coverage for the declared entry point. |
examples/deerflow-extension-jev-context/deerflow_extension_jev_context/compaction.py |
Moderate findings: fix cooldown accounting (2 votes), inspect structured tool status, include effective options in release identity, safely validate non-finite values, and continue scanning after oversized candidates. |
examples/deerflow-extension-jev-context/deerflow_extension_jev_context/__init__.py |
Plugin registration and catalog contribution; no findings. |
docs/full-stack-plugins.md |
Plugin catalog documentation; no findings. |
backend/tests/test_jev_context_extension.py |
Host integration and behavior tests; no findings. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
willem-bd
left a comment
willem-bd
left a comment
There was a problem hiding this comment.
Reviewed at d2742ef. I verified the example against the head contracts rather than the diff alone: extension-api 0.2.2 exports and PluginContribution/BackendAction/SettingsField/MiddlewarePlacement usage, ExtensionRegistry.plugin/middlewares, the ReleasePolicyProvider duck-typing (including isolation-wrapper unwrap), LoadedExtensions.plugins/middleware_contributors shapes, and the MODEL_LOGICAL anchor (outer_of(LLMErrorHandlingMiddleware), so the integration test's before_model ordering vs. summarization reflects the real host stack, not a test artifact). Also checked IsolatedMiddleware lifecycle fail-open containment (unexpected prepare() raises are contained and reported; CancelledError propagates), deerflow_tool_meta/normalize_tool_message, CI conventions (the 2.0.x-dev push branch and setup-uv 0.11.1 pin match backend-unit-tests.yml and are enforced by test_ci_uv_version_pin.py), the Gateway plugin-action dispatcher (the strict {}-only status contract matches the bookmarks example's strict-payload style and maps to a clean 422), and the host lockfile satisfying the declared ranges (langchain 1.3.14, langchain-core 1.4.9, pydantic 2.13.3, httpx 0.28.1).
No correctness bugs found. The cooldown accounting Copilot flagged is fixed at this head (full configured count stored; attempts at 1, 18, 35 for the default 16 match the README and the gap-parameterized tests). Suggestions and nits below.
| response = client.post(ENDPOINT, json=body, headers={"Authorization": f"Bearer {os.environ[self.options.api_key_env]}"}) | ||
| response.raise_for_status() | ||
| return self.finish(state, update, selected, response.json()) | ||
| except (httpx.HTTPError, ValueError, KeyError, TypeError): |
There was a problem hiding this comment.
Failing open is the right default, and the no-payload/no-credential logging policy is sound — but as written, a deployment with a revoked key, a typo'd api_key_env target, or blocked egress shows configured: true in the Capability Center forever while every attempt silently fails. Because the exception is caught here (and in abefore_model), the host's IsolatedMiddleware diagnostic reporting never fires either, so there is zero signal anywhere that pruning is dead. Would you consider one bounded line, e.g. logger.debug("Jev request failed: %s", type(exc).__name__) — exception class only, no URL, payload, key, or transcript content? That keeps the stated policy intact while making misconfiguration diagnosable.
There was a problem hiding this comment.
Addressed in 8f99bf6. Added exception-class-only DEBUG diagnostics in both before_model and abefore_model. Neither path includes exception text, traceback, URL, credentials or transcript content. Fail-open behavior, cooldown and the single-attempt policy remain unchanged. README now names the logger and clarifies that configured only reports a nonempty key environment variable, not a successful connectivity/authentication check. Sync/async regressions check the exact diagnostic and unchanged history for malformed answers, HTTP timeouts, ValueError, KeyError and TypeError.
Validation: 86 plugin tests and 5 CI configuration tests passed; blocking-I/O suite 151 passed; installed-wheel verification passed. Full backend: 18,593 passed / 14 failed, with all 14 failing IDs matching the previously verified clean PR base 41ed506; no new failures.
| return None, None | ||
| return {STATE_KEY: {"remaining": self.options.min_calls_between_attempts}}, prepared | ||
|
|
||
| def finish(self, state, update, selected, response): |
There was a problem hiding this comment.
finish() is the one moment where the consumed→produced mapping for this destructive rewrite is still known, and it is discarded — success leaves no trace either. The public contract at extension-api 0.2.2 already ships registry.context_compaction_observer() + CompactionEvent for exactly this purpose (emit the transform "where it is still true"), and the README simultaneously promises "no transcript archive" and no logging, so a future "why did the agent lose this fact" question has no forensic trail at all. If the lifecycle-hook signature (no app/task store handed to before_model) is what blocks emitting the event from inside a middleware, a sentence in the README's "Behavior and limits" acknowledging the observability gap and its cause would help deployers set expectations.
There was a problem hiding this comment.
Addressed in 8f99bf6. Added the observability limitation to Behavior and limits. Shortened messages retain their IDs and carry jev_context_shortened=true, but the plugin retains neither the original-to-replacement hash mapping nor a detailed compaction event. The marker cannot reconstruct omitted facts or explain the classifier decision. The API boundary is event emission: extension-api 0.2.2 exposes CompactionEvent and observer registration, but no public middleware emission API; notify_context_compacted is host-internal. The example therefore documents the gap without importing that internal dispatcher.
| async def main(): | ||
| if not os.environ.get("TYPESAFE_API_KEY"): | ||
| raise SystemExit("Set TYPESAFE_API_KEY before running this opt-in smoke test") | ||
| base = os.environ["TEST_CHAT_BASE_URL"].rstrip("/") |
There was a problem hiding this comment.
Nit: missing TYPESAFE_API_KEY gets a clean SystemExit message, but missing TEST_CHAT_BASE_URL or TEST_CHAT_MODEL produces a raw KeyError traceback. Cheap to align with the check above.
There was a problem hiding this comment.
Addressed in 8f99bf6. Aligned all three required smoke-test settings: missing, empty or whitespace-only TYPESAFE_API_KEY, TEST_CHAT_BASE_URL and TEST_CHAT_MODEL now produce a clean SystemExit naming the settings before any network request. Nine offline cases exercise those combinations through the actual script main function and verify that no request is sent.
| The upstream license is reproduced below. | ||
|
|
||
| MIT License | ||
|
|
There was a problem hiding this comment.
Nit: Copyright (c) 2025 names no holder, which weakens the attribution this file exists to provide (MIT's grant references "the above copyright notice"). Worth naming the upstream author(s) if the upstream LICENSE identifies them.
There was a problem hiding this comment.
Addressed in 8a61424. This is an independent DeerFlow implementation; fast-jev-compaction was an idea reference, not a vendored dependency. The extra reference-project notice was replaced with the DeerFlow repository MIT LICENSE, and the README keeps the idea-source link while explicitly identifying the implementation as independent. The package metadata and installed-wheel verification now check the actual DeerFlow license. The wheel was rebuilt, installed and verified through the host.
willem-bd
left a comment
willem-bd
left a comment
There was a problem hiding this comment.
Follow-up review of the delta since the round reviewed at d2742ef (commits 8a61424, 8f99bf6, 2e43887; head 2e43887, CI green). All four earlier findings are resolved at this head and pinned by tests: both middleware paths now emit a class-only DEBUG record with no exception text, traceback, URL or transcript content, and the README clarifies that configured only means a nonempty env var, not verified connectivity; the README documents the jev_context_shortened marker, the absence of any retained original-to-replacement mapping, and that 0.2.2 CompactionEvent observers are not notified; verify_live.py treats missing, empty, or whitespace-only values of all three required settings as one clean SystemExit, asserted for all nine combinations; and the renamed LICENSE names ByteDance/DeerFlow as copyright holders, wired through pyproject license-files and the verify_package.py metadata assertions. One new nit below; nothing else in the delta.
| monkeypatch.setenv(missing, value) | ||
| requests = transport(monkeypatch) | ||
| script = Path(__file__).resolve().parents[2] / "examples/deerflow-extension-jev-context/scripts/verify_live.py" | ||
| main = runpy.run_path(str(script))["main"] |
There was a problem hiding this comment.
Nit: runpy.run_path executes verify_live.py, whose module-level sys.path.insert(0, ...) (verify_live.py line 17) mutates the process-global sys.path and is never restored — so the example-package directory leaks into sys.path for the rest of the backend test session (nine duplicate entries across this test's parametrizations), defeating the isolation the jev fixture's monkeypatch.syspath_prepend otherwise undoes at teardown. Swapping in a copy first keeps the leak local: monkeypatch.setattr(sys, "path", list(sys.path)) before the run_path call.
There was a problem hiding this comment.
Addressed in 1a88f83. The fixture already restored sys.path at teardown, so I could not reproduce accumulation across parametrizations. However, runpy.run_path did mutate the original list during each case. The test now gives the script a copy with monkeypatch.setattr(sys, "path", list(sys.path)) and asserts that the original list is unchanged. That assertion failed for all nine cases before the change and passed for all nine afterward.
Validation: 91 focused plugin/CI tests passed, blocking-I/O tests 151 passed, and make format/make lint passed. Full backend: 18,593 passed / 14 failed; the complete 14 failing IDs match the previously verified clean PR base 41ed506, with no new failures.
willem-bd
left a comment
willem-bd
left a comment
There was a problem hiding this comment.
Static review of the current PR head.
…5731) * feat(plugins): add standalone Jev context pruning example * fix(plugins): address Jev pruning review findings * fix(examples): package Jev plugin with DeerFlow license * fix(examples): address Jev plugin review feedback * docs(examples): remove unrelated Jev project reference * test(examples): isolate Jev live script import path * Update branch pattern for push event * Fix YAML syntax for branch pattern --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Related to the full-stack plugin contract introduced in #5647.
Why
Long tool-heavy conversations can carry large obsolete results until native summarization runs. This independent example exercises whether the existing plugin catalog and middleware hooks can implement optional relevance-based pruning without adding a core compaction API or a host dependency on Jev.
What changed
deerflow-extension-jev-contextpackage, registered ascommunity.jev-contextin the read-only plugin catalog. Deployment opt-in and an environment-only Jev key are required. No custom browser page or frontend rebuild is needed.before_model/abefore_modelhooks on the lead agent to ask Jev about old read-only tool results. Keep all tool-call records and protect recent messages, host-stamped errors/partial results, skill reads, multimodal content and non-read-only operations. Shorten only low keep-probability results, preserving message IDs and metadata.The pruning is lossy and persists in graph state; disabling the plugin does not restore omitted text. Deployment opt-in sends bounded conversation excerpts to Jev. Rewriting history can invalidate a cached suffix, so reduced prompt tokens are not a guarantee of lower cost or latency. The package documents this distinction and leaves manual native compaction unchanged.
Surface area
frontend/backend/applanggraph.json, or prompt changestate; fixed question instructions contain only locally generated candidate IDs. No conversation text is inserted into framework-owned system text.docker/or sandboxed executionskills/backend/pyproject.tomlorfrontend/package.jsonValidation
cd backend && make format && make lint: passed. Example package Ruff checks passed as well.tests/test_jev_context_extension.pyplus CI uv-version checks: 91 passed (86 plugin tests). Coverage includes sync/async cooldowns and privacy-preserving failure diagnostics, host-normalized errors, middleware identity, bounded candidate selection, and all three required smoke-test settings missing, empty or whitespace-only. The opt-in smoke-test import runs against a copiedsys.path; a red/green regression checks that its module-level insertion does not mutate the original process search-path list across all nine parametrizations. The current diagnostic/preflight regressions failed before the fixes (17 failed / 2 passed); all pass now.make test-blocking-io: 151 passed.make testrun: this branch 18,593 passed / 14 failed / 206 skipped / 6 deselected. Compared the complete failure set with the previously completed clean run of the unchanged PR base41ed5063(18,518 passed / 14 failed / 206 skipped / 6 deselected), using the same command and separate locked environments with verified source imports. The complete failing-test ID sets are identical, with no new failures. They are environment-sensitive browser/fetch tests intest_browser_automation.py,test_browserless_client.py,test_crawl4ai_tools.pyandtest_fastcrw_tools.py(DNS resolution / private-address validation). The local full suite is not claimed green.scripts/verify_package.py. A deliberately broken installed entry point fails the check. The new Jev Plugin Package workflow runs that build/install/check sequence in CI; ordinary offline unit tests require no package build or network.AI assistance
Tool(s) used: Codex.
How you used it: Inspected context-pruning approaches and public extension contracts; implemented the independent package, documentation and tests; ran offline/live validation and reviewed the resulting diff. The human author attestation below is left for the author to complete.