Repository navigation
feat(plugins): add standalone Jev text classification example - #5735
Conversation
A deployment-installed package contributes one model tool, classify_texts, and a status action through the full-stack plugin contract. The tool labels a list of texts against caller-supplied categories with a deployment-configured backend: Jev (one choice question per item over a shared batch state, checked per item) or an OpenAI-compatible chat endpoint (a JSON label list validated in code). Items are packed into requests under a size limit and sent with bounded concurrency; every failure is explicit per call or per item; credentials come only from deployment-named environment variables; item text never enters instructions, question keys, results or errors, and the served model name is kept to a short identifier. Each batch runs under an asyncio timeout bounded by the call deadline, so the whole call stays inside the host's 30-second limit, and ids and category names are bounded in bytes so 300 results fit the host's 64 KiB output limit.
…tion plugin The script loads the package through the real extension path and classifies 20 synthetic English and Chinese sentences plus a blank and an over-long item with the configured backend, printing labels and statuses only. Against real Jev it agreed on all 20 labels in two requests and 1.6 seconds, with the served model reported as jev-1.13.0.
Totoro-qaq
changed the title
github-actions
Bot
added
area:deps
willem-bd
left a comment
willem-bd
left a comment
There was a problem hiding this comment.
Reviewed 56f677a against the harness plugin path at head. Overall this is in good shape: the host-side schema validation before the handler makes the plugin's direct key access safe, the per-item vs per-batch failure separation matches the Jev and chat answer shapes, the asyncio deadline correctly replaces reliance on the socket timeout, and the no-leak properties (caller ids never on the wire, no item text/keys/upstream bodies in results or logs) hold up under scrutiny. Two findings: one correctness issue in the request-size estimation, and one small coverage gap. Minor note not worth an inline: the deadline test's wall-clock assertion (elapsed < 1.6) is timing-sensitive on a loaded CI runner, though the 60% slack is reasonably generous.
| def __init__(self, options: Options, key: str, labels: Labels, instruction: str) -> None: | ||
| self.url = options.jev_base_url.rstrip("/") + JEV_ENDPOINT_PATH | ||
| self.model = options.jev_model | ||
| self.headers = {"Authorization": f"Bearer {key}", "Accept": "application/json"} |
There was a problem hiding this comment.
The size estimators measure json.dumps(..., ensure_ascii=False), but httpx serializes json= bodies with the json.dumps default (ensure_ascii=True), escaping every non-ASCII character as six bytes (\uXXXX). For CJK content the estimate is therefore up to ~2x low: with defaults (batch_size: 10, max_text_chars: 2000) plus 32 CJK category names/descriptions and 2000-char CJK texts, _pack produces batches it estimates around 200 KB whose actual bodies are around 400 KB, exceeding MAX_REQUEST_BYTES — the invariant the README states ("each request is kept under 256 KiB") and that test_requests_are_packed_under_the_size_limit... pins, since that test only uses ASCII payloads. The same mismatch applies to JevBackend.cost (line 182) and ChatBackend.base/ChatBackend.cost (lines 212 and 218). Dropping ensure_ascii=False in all four places would make the estimate match the actual wire format.
There was a problem hiding this comment.
Right, the estimators measured raw UTF-8 while httpx sends the json.dumps default. Fixed in 82a71ed: all four estimates now go through one _wire_size helper that measures exactly that encoding, and the packing test gained a CJK variant (32 categories with 600-character descriptions, 2,000-character texts) that asserts on the bytes the mock transport actually received rather than on the estimate. With that input Jev now takes 20 requests of two items (each just under the limit) and the chat backend four requests of ten; before the fix the same input produced 200 KB estimates for 400 KB bodies.
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_transport_timeouts_are_reported_per_item_without_upstream_detail(load, monkeypatch): |
There was a problem hiding this comment.
The README advertises the network per-item status, but no test exercises it — only timeout (httpx.TimeoutException) and HTTP status codes are covered. A one-case variant of this test raising e.g. httpx.ConnectError from the mock transport would also pin the httpx.HTTPError -> network branch of _post.
There was a problem hiding this comment.
Added in 82a71ed: the transport-failure test is now parametrised over ReadTimeout → timeout and ConnectError / RemoteProtocolError → network, each also checking that the upstream message never reaches the result. On the wall-clock note: the upstream in that test never answers, so returning at all is what proves the asyncio bound; the assertion is now elapsed < 3.0 with a comment saying the slack only guards tightness.
httpx serializes json= bodies with the json.dumps default, escaping every non-ASCII character as six bytes, while the size estimators measured raw UTF-8, so CJK batches could reach about twice the 256 KiB limit. The estimators now measure the same encoding; a test with CJK categories and texts checks the bytes the transport actually received. The network status gets a test through connect and protocol errors, and the deadline test's wall-clock slack is widened for loaded runners.
willem-bd
left a comment
willem-bd
left a comment
There was a problem hiding this comment.
Re-reviewed the delta at 82a71ed. The transport-failure test is now parametrised over ConnectError/RemoteProtocolError so the documented network status is covered, and the full suite passes at the new head (56 passed). One correction on the wire-size change, which also corrects my earlier finding: the pinned httpx (>=0.28.0) does not send the json.dumps default for json= — it sends compact, non-escaped UTF-8 — so the new estimator overshoots rather than matching the wire. Details inline.
|
|
||
|
|
||
| def _wire_size(value: Any) -> int: | ||
| """Bytes ``httpx`` sends for ``json=value``: the ``json.dumps`` default, non-ASCII escaped.""" |
There was a problem hiding this comment.
Correction here — to this fix and to my earlier thread. httpx >= 0.28.0 (the httpx>=0.28.0 pin in backend/pyproject.toml) does not send the json.dumps default for json=: httpx._content.encode_json serializes with ensure_ascii=False, separators=(",", ":"), allow_nan=False, so non-ASCII goes out as raw UTF-8, not \uXXXX escapes. Measured on 0.28.1 with a CJK body: 992 bytes on the wire, json.dumps(value).encode() estimates 1909 (1.92x), while the previous ensure_ascii=False form estimated 1003 (1.01x). So the cap can no longer be violated (the estimate only overshoots, and _pack still gives an oversized single item its own batch), but non-ASCII requests now pack at roughly half capacity — about twice as many requests as needed — and the docstring, the JevBackend.__init__ comment ("what httpx puts on the wire", "six-byte escape"), and the new test's name/comment assert the opposite of the pinned httpx's behavior. Could you measure with httpx's actual encoding (json.dumps(value, ensure_ascii=False, separators=(",", ":"))) and rewrite the test's comment/name, or, if the extra margin is deliberate, document it as margin rather than as the wire size?
There was a problem hiding this comment.
Thanks for the correction — I verified httpx 0.28.1's actual request bytes; my earlier reply's description of non-ASCII escaping was incorrect.
Fixed in bb5b8e0: _wire_size now uses compact UTF-8 JSON (ensure_ascii=False, separators=(",", ":"), allow_nan=False). The CJK fixture now uses 10 Jev requests instead of 20, while still checking every captured request against the 256 KiB cap. Updated the test name and comments.
I also corrected the chat estimator to count the additional escaping of JSON embedded in its message string. Regressions cover quotes, backslashes and newlines that fit the host input limit but previously exceeded the outbound cap.
Validation: 62 plugin tests and 153 blocking-I/O tests passed; lint, formatting and guidance checks passed. Full offline suite: 18,581 passed, with the same 14 failing test IDs as the unchanged baseline (18,575 passed); no new failures.
| return _error("invalid_categories", f"Category names must be unique, non-blank and at most {MAX_KEY_BYTES} bytes.") | ||
| items = [dict(item) for item in payload["items"]] | ||
| identifiers = [item["id"] for item in items] | ||
| if len(set(identifiers)) != len(identifiers) or any(not identifier.strip() or len(identifier.encode()) > MAX_KEY_BYTES for identifier in identifiers): |
There was a problem hiding this comment.
[P2] Bound IDs by their JSON-encoded size before accepting a 300-item call. This check counts raw UTF-8 bytes, but the host serializes the result with json.dumps(..., ensure_ascii=False) and rejects outputs over 64 KiB. I reproduced a schema-valid request with 300 distinct 64-byte IDs (60 NUL characters plus a four-digit index) and blank texts: the input was 116,757 bytes, while the result encoded to 128,500 bytes. The real ToolNode returned Plugin result exceeds 64 KiB. instead of per-item empty_text statuses. Control characters expand to six bytes each in JSON; category labels can expand too. Please validate against the encoded output budget or constrain escape-heavy IDs and labels, and add a maximum-item regression.
There was a problem hiding this comment.
Thanks — reproduced at bb5b8e0c: 300 schema-valid 64-byte IDs containing NUL characters produced a 128,500-byte JSON result from a 116,524-byte input, which the real plugin tool path would reject at 64 KiB.
Fixed in dad8667d: item IDs and category names now satisfy both the existing raw UTF-8 limit and a 64-byte limit for their content after host-style JSON encoding. Invalid keys return invalid_items or invalid_categories before any classifier request. The two new 300-item regressions cover both fields through the real ToolNode; existing 64-byte Chinese keys and maximum successful/error outputs still pass.
Validation: 64 plugin tests, 153 blocking-I/O tests, lint, formatting, and guidance checks passed. Full offline suite: 18,620 passed / 14 failed / 170 skipped / 6 deselected. The 14 failed node IDs match the unchanged pre-fix head; 37 Redis cases that were previously skipped ran in this environment, and no new failures appeared.
…nce#5735) * feat(examples): add a standalone text classification plugin A deployment-installed package contributes one model tool, classify_texts, and a status action through the full-stack plugin contract. The tool labels a list of texts against caller-supplied categories with a deployment-configured backend: Jev (one choice question per item over a shared batch state, checked per item) or an OpenAI-compatible chat endpoint (a JSON label list validated in code). Items are packed into requests under a size limit and sent with bounded concurrency; every failure is explicit per call or per item; credentials come only from deployment-named environment variables; item text never enters instructions, question keys, results or errors, and the served model name is kept to a short identifier. Each batch runs under an asyncio timeout bounded by the call deadline, so the whole call stays inside the host's 30-second limit, and ids and category names are bounded in bytes so 300 results fit the host's 64 KiB output limit. * docs(examples): add an opt-in live smoke test for the text classification plugin The script loads the package through the real extension path and classifies 20 synthetic English and Chinese sentences plus a blank and an over-long item with the configured backend, printing labels and statuses only. Against real Jev it agreed on all 20 labels in two requests and 1.6 seconds, with the served model reported as jev-1.13.0. * fix(examples): measure request size as sent and cover transport failures httpx serializes json= bodies with the json.dumps default, escaping every non-ASCII character as six bytes, while the size estimators measured raw UTF-8, so CJK batches could reach about twice the 256 KiB limit. The estimators now measure the same encoding; a test with CJK categories and texts checks the bytes the transport actually received. The network status gets a test through connect and protocol errors, and the deadline test's wall-clock slack is widened for loaded runners. * fix(examples): match classifier request wire encoding * fix(examples): bound classification output keys by JSON size --------- Co-authored-by: Totoro-qaq <279883115+Totoro-qaq@users.noreply.github.com>
Related: #5653 (RFC and matched-batch evaluation) and #5679 (host model invocation for extensions). This is the classification example both threads asked for; it does not change host code.
Why
DeerFlow has no way to label a list of texts against caller-supplied categories. #5653 measured that a typed classifier (Jev) and ordinary chat models both do this well in batches of ten, and the maintainers asked to see it as an optional extension rather than a core feature. With #5647 merged, a deployment-installed package can contribute a model tool, so the prototype from the RFC is reshaped into that form. #5679 asked for a concrete consumer of a future host model capability; the chat backend here is the seam where that capability plugs in.
What changed
examples/deerflow-extension-jev-classifycontributing one model tool,classify_texts, and a read-onlystatusaction under the plugin namespacecommunity.jev-classify. No browser code, no host imports, no frontend rebuild.{id, text}items, 2 to 32 categories with one-line descriptions and an optional instruction, and returns one label and status per item in input order. For a CSV the agent reads the file with its normal tool, sends rows with the row number asid, and writes labels back itself; the tool never touches files.choicequestion per item over a shared batch state (POST /v1/systemone). Chat asks an OpenAI-compatible endpoint for a JSON label list and validates the id set and every label in code. Credentials are read only from deployment-named environment variables; the plugin owns one HTTP client per call and closes it.backend_not_configured,invalid_items,invalid_categoriesandtoo_many_itemsfor the whole call;http_<status>,timeout,network,invalid_response,internal,stoppedanddeadlineper item. Jev answers are checked per item, so one bad answer fails one item; a chat label list is validated as a whole. An HTTP 401/402/403/429 stops further batches. Nothing is retried; the only log line is a type name when a plugin bug fails a batch.not_processed. Cancellation propagates and closes the client.docs/full-stack-plugins.md.Out of scope, on purpose: reading or writing files from inside the tool (the tool context carries no file access), a CSV-specific tool, and the host model invoker from #5679. Once that capability exists, the chat backend switches to it; the Jev backend stays plugin-owned, as #5731 does for its own Jev calls.
Surface area
deerflow-extension-api,httpxandpydantic, all already in the backend graph; no host dependency changesValidation
Wire-size review follow-up (
bb5b8e0c): 62 plugin tests and 153 blocking-I/O tests passed; lint, format and guidance checks passed. Full offline suite: 18,581 passed, with the same 14 failing test IDs as the unchanged baseline (18,575 passed); no new failures.Initial tests first:
backend/tests/test_jev_classify_extension.py(52 cases) was written before the package and failed with a missing module; all pass with it. It covers registration and the disabled state, the Jev request shape and label mapping, batching with per-item statuses, size-based packing for both backends, the concurrency bound, per-item and per-batch Jev failures, label canonicalisation, HTTP, authentication and rate-limit failures, transport timeouts, the enforced deadline (in-flight batches end astimeoutand the call returns within the budget even when the upstream never answers), cancellation closing the client, a backend bug failing one batch, the chat backend's output contract and prompt escaping, missing credentials, invalid requests including 65-byte ids and names, the output bound at 300 items with 64-byte ids and labels, served model-name sanitisation, real tool-node dispatch with no text leakage, the status action and rejected deployment options (including malformed endpoints).An adversarial review pass over the first version found that the deadline relied on the socket timeout, which a trickling upstream can outlive, and that 64-character ids could exceed the host's byte-measured output bound; both are fixed and pinned by tests above.
ruff checkandruff format --checkwith the backend configuration: clean.Blocking I/O suite (
make test-blocking-io): 153 passed.Full offline backend suite (
pytest -m "not live" --ignore=tests/blocking_io tests/) at 2800369: 18,569 passed, 208 skipped, 15 failed. The 15 failures are environment-specific and unrelated: 14 are the known cases that need outbound DNS (the tests' public URLs hit the local address guard here), and the remaining one,test_local_sandbox_provider_mounts.py::TestReadOnlyPath::test_bash_write_to_projected_copy_does_not_mutate_source, fails identically on a pristineupstream/maincheckout with the same environment; the diff touches onlyexamples/,docs/and the new test file, and main's own Unit Tests workflow is green.Agent guidance check: 31 guidance files, 0 errors, 0 warnings.
Wheel build:
uv buildproduces the wheel with thedeerflow.extensionsentry point and the three declared dependencies.Live Jev smoke test (
scripts/verify_live.py, added in 56f677a): 20 synthetic English and Chinese sentences across four categories plus a blank and an over-long item, batch size 10. All 20 labels matched the obvious ones, the two extra items came back asempty_textandtoo_long, 2 requests, 1.56 s end to end through the real plugin path, served model reported asjev-1.13.0for thejev-latestalias. No chat endpoint was called. The measurements that motivated the batch size and the question shape are in [RFC] Jev for text-list and CSV classification: evaluation and minimal workflow #5653.Output-bound review follow-up (
dad8667d): 64 plugin tests and 153 blocking-I/O tests passed; lint, format and guidance checks passed. Full offline suite: 18,620 passed / 14 failed / 170 skipped / 6 deselected. The 14 failed node IDs match the unchanged pre-fix head; 37 Redis cases that were previously skipped ran in this environment, and no new failures appeared.AI assistance
Tool(s) used: Claude Code; Codex for the request-size review follow-up.
How you used it: Wrote the regression tests first, then the package and docs, and ran local validation. Codex independently checked httpx serialization, added failing wire-size regressions, implemented the follow-up, and ran offline validation against the unchanged baseline.