Release 4.6.0

AGENTS.md

@@ -89,8 +89,9 @@ Mirror the existing patterns (e.g. `field-property`, `prompt-config`, `ruleset`)
 Details in [`specs/04-testing.md`](./specs/04-testing.md). The shape:
 
 - Tests live in `tests/` and drive command behaviour through `click.testing.CliRunner` — no live PolySwarm stack needed.
-- Two styles for that: **SDK-boundary mocks** (`mock.patch('polyswarm_api.api.PolyswarmAPI.<method>')`, e.g. `tests/field_property_test.py`) and **VCR cassettes** (`tests/cli_test.py`) that replay recorded HTTP for end-to-end CLI runs. Cassettes live in `tests/vcr/` (`.vcr` for the HTTP interactions, `.click` for the expected rendered output).
+- Three styles for that: **SDK-boundary mocks** (`mock.patch('polyswarm_api.api.PolyswarmAPI.<method>')`, e.g. `tests/field_property_test.py`), **SDK-transport mocks** (`PolyswarmSession.execute`, e.g. `tests/refang_test.py`, for behaviour that runs inside an SDK method; see the next bullet) and **VCR cassettes** (`tests/cli_test.py`) that replay recorded HTTP for end-to-end CLI runs. Cassettes live in `tests/vcr/` (`.vcr` for the HTTP interactions, `.click` for the expected rendered output).
 - Pure rendering logic — which line a given field set produces, no command-tree behaviour — is instead unit-tested against the formatter directly (e.g. `tests/known_good_field_test.py`); see the spec's *Style 3* for when that's the right choice.
+- Behaviour that runs *inside* an SDK endpoint method (IoC refanging) is tested at the SDK transport instead — `PolyswarmSession.execute` — because an SDK-method mock would replace the code under test (e.g. `tests/refang_test.py`); see the spec's *Style 4*.
 - VCR is an **efficiency cache, not a requirement** — the suite must pass against a live e2e stack with VCR off. Re-record a cassette by deleting it and re-running the test against a live stack; never hand-edit a cassette or `cp` one from a sibling test.
 
 ## Commit + PR hygiene

pyproject.toml

@@ -22,7 +22,7 @@ classifiers = [
 ]
 
 dependencies = [
-    "polyswarm_api>=4.5.0,<5.0.0",
+    "polyswarm_api>=4.7.0,<5.0.0",
     "click>=7.1",
     "colorama>=0.4.6",
     "click-log>=0.4.0",

specs/01-architecture.md

@@ -16,7 +16,7 @@ The components of the CLI and how a command flows from `argv` to rendered output
 
 `polyswarm_cli` is the top-level `click.Group`, constructed with `cls=ExceptionHandlingGroup`. It:
 
-1. Declares the **global options** — `--api-key` (env `POLYSWARM_API_KEY`), `--api-uri` (env `POLYSWARM_API_URI`), the **endpoint shortcuts** `--prod` / `--stage` / `--local` / `--prod-eu` / `--stage-eu`, `--output-file`, `--output-format`/`--fmt` (`text`|`json`|…), `--color/--no-color`, `--verbose`, `--community` (env `POLYSWARM_COMMUNITY`), `--parallel`, `--verify/--no-verify`, plus `--version` / `--api-version`.
+1. Declares the **global options** — `--api-key` (env `POLYSWARM_API_KEY`), `--api-uri` (env `POLYSWARM_API_URI`), the **endpoint shortcuts** `--prod` / `--stage` / `--local` / `--prod-eu` / `--stage-eu`, `--output-file`, `--output-format`/`--fmt` (`text`|`json`|…), `--color/--no-color`, `--verbose`, `--community` (env `POLYSWARM_COMMUNITY`), `--parallel`, `--verify/--no-verify`, `--refang/--no-refang` (default on; see [`02-commands.md`](./02-commands.md) §Global `--refang/--no-refang`), plus `--version` / `--api-version`.
 
    **Endpoint resolution** (`resolve_api_uri`): the shortcuts are convenience aliases for known public endpoints (`API_URI_SHORTCUTS`); `--prod` is an explicit alias for the production endpoint (identical to the no-flag default, offered for symmetry). Precedence is **explicit command-line flag → `POLYSWARM_API_URI` env var → production default** (`PROD_API_URI` = `https://api.polyswarm.network/v3`). Specifically: a shortcut and an explicit *command-line* `--api-uri` are mutually exclusive (conflict → `click.UsageError`, exit 2), as are two shortcuts; a shortcut **wins over** an ambient `POLYSWARM_API_URI` (the env var is consulted only when no shortcut is given) — so `--prod` forces production even when the env var points elsewhere; and a command-line `--api-uri` wins over the env var (click's own source precedence). The command-line-vs-env distinction uses `ctx.get_parameter_source('api_uri') == ParameterSource.COMMANDLINE`.
 2. **Seeds `ctx.obj`** — constructs a `Polyswarm(...)` client (the SDK wrapper) as `ctx.obj['api']` and the selected formatter as `ctx.obj['output']`.
@@ -82,13 +82,13 @@ The catalogue of groups and the SDK methods each wraps is in [`02-commands.md`](
 
 ## Support — `utils.py`, `exceptions.py`
 
-- **`utils.py`** — `parallelize`/`parallel_executor` (thread-pool fan-out with per-item exception aggregation: collects results, logs per-item no-results, raises an aggregate `NoResultsException`/`NotFoundException`/`InternalFailureException` at the end) and `parallel_executor_iterable_results` (the same, for SDK methods that return generators — it materialises each generator inside the worker so per-item exception handling still fires), plus `collect_files` and the detection helpers (`is_valid_id`, `is_ip`, `is_domain`, `is_url`).
+- **`utils.py`** — `parallelize`/`parallel_executor` (thread-pool fan-out with per-item exception aggregation: collects results, logs per-item no-results, raises an aggregate `NoResultsException`/`NotFoundException`/`InternalFailureException` at the end) and `parallel_executor_iterable_results` (the same, for SDK methods that return generators — it materialises each generator inside the worker so per-item exception handling still fires), plus `collect_files` and the detection helpers (`is_valid_id`, `is_ip`, `is_domain`, `is_url`), and `refang_input(api, value)` — not a detection helper but the refang one: it applies the SDK's `polyswarm_api.refang.refang_ioc` only when the client's `refang_iocs` is on, for the few places the CLI handles a value before (or instead of) an SDK endpoint method.
 - **`client/utils.py`** — `parse_hashes` and the click parameter validators (`validate_id`, `validate_hash(es)`, `validate_key`, …). Note the module is `client/utils.py`, not the top-level `utils.py` above — the two are distinct and easily confused.
 - **`exceptions.py`** — the CLI's own hierarchy, **distinct from the SDK's**: `PolyswarmException` → `NoResultsException`, `NotFoundException`, `InternalFailureException`, `PartialResultsException`. `ExceptionHandlingGroup` catches both these and the SDK's `api_exceptions.*`.
 
 ## Lifecycle of a command (end to end)
 
-1. `polyswarm_cli` parses global options, builds `Polyswarm(api_key, uri=…, community=…, parallel=…, verify=…)` and the formatter into `ctx.obj`.
+1. `polyswarm_cli` parses global options, builds `Polyswarm(api_key, uri=…, community=…, parallel=…, verify=…, refang_iocs=…)` and the formatter into `ctx.obj`.
 2. The subcommand reads `api`/`output` from `ctx.obj`, parses its own args, and calls one or more SDK methods (directly or via a wrapper fan-out method).
 3. Results are rendered through the formatter; collections are iterated.
 4. Any exception propagates to `ExceptionHandlingGroup.invoke`, which logs it and raises `Exit(<code>)`.

specs/02-commands.md

@@ -16,7 +16,7 @@ The top-level command groups, what each is for, and the primary `polyswarm-api`
 
 | Group / commands (module) | Purpose | Primary SDK methods wrapped |
 |---|---|---|
-| `search` (`search.py`) | Search by hash / URL / metadata / IOC / scans; metadata mapping; `field-property` subgroup | `search_hashes`, `search_urls`, `search_by_metadata`, `search_by_ioc`, `iocs_by_hash`, `search_scans`, `metadata_mapping`, `metadata_field_properties_{write,get,delete,list}` |
+| `search` (`search.py`) | Search by hash / URL / metadata / IOC / scans; metadata mapping; `field-property` subgroup. `search ioc <type> <value>` with an ip, domain, ttp or imphash type is the reverse IOC search and prints one sha256 per matching artifact; `--with-artifacts` asks the server for each artifact's metadata-search row instead (SDK 4.7.0, guaranteed by the pin — see [05-sdk-contract.md](./05-sdk-contract.md) §Current floor) and renders it with the `metadata` formatter, the same block `search metadata` prints, so the `sha256` / `sha1` / `md5` output formats work on it. The flag is refused (exit 2, a click `UsageError`) on the sha256 / sha1 / md5 forward lookup, which has no artifact rows to return. It needs a server that supports the parameter: an older one ignores it and answers sha256 strings, which the SDK cannot parse as metadata rows | `search_hashes`, `search_urls`, `search_by_metadata`, `search_by_ioc`, `iocs_by_hash`, `search_scans`, `metadata_mapping`, `metadata_field_properties_{write,get,delete,list}` |
 | `known` + `search known` (`search.py`) | Manage / check known-good/-bad hosts | `add_known_good_host`, `add_known_bad_host`, `update_known_good_host`, `delete_known_good_host`, `check_known_hosts` |
 | `kgb` (`kgb.py`) | Known Good **Binaries** — internal-only CRUD on a sha256-keyed known-good record (`create` / `get` / `delete`, no update). Distinct from `known`, which manages known-good/-bad *hosts*. | `known_good_create`, `known_good_get`, `known_good_delete` |
 | `scan`, `lookup`, `wait`, `rescan`, `rescan-id` (`scan.py`) | Submit files/dirs and await results; look up / rescan existing scans | `scan_file`, `scan_lookup`, `wait_for`, `rescan`, `rescan_id` (wrapper methods over `submit`/`lookup`/`rescan`/`rescan_id`) |
@@ -62,6 +62,19 @@ The top-level command groups, what each is for, and the primary `polyswarm-api`
 > likeliest origin of the original mistake — check which endpoint you are on
 > before copying a default between them.
 
+## Global `--refang/--no-refang` (IoC refanging)
+
+Threat-intel reports print indicators defanged (`hxxps[:]//evil[.]com`, `127[.]0[.]0[.]1`); pasted verbatim they never match a search, and a submitted one becomes a broken URL artifact. The root group's `--refang/--no-refang` (default on) is passed to the client as `Polyswarm(..., refang_iocs=…)`. The SDK's own default is off (its refanging is opt-in, so the SDK release stays a minor bump); the CLI opts in unless `--no-refang` is given, and the SDK refangs the URL / domain / IP inputs of its own endpoint methods (`polyswarm_api.refang`; rules and gate in the SDK's downstream-contract spec). So `search url`, `search metadata -p/-u/-d` (never the free-form query), `search ioc ip|domain`, `search known`, `known add/update`, and the `-r/--url-file` lines of `scan url` (which reach the wire only through the SDK's `submit`) need no CLI code. Positional `scan url` / `sandbox url` URLs are the exception below.
+
+**Write paths change too, not only searches.** With refanging on (the default), `known add` / `known update` store the live host (`known add domain evil[.]com feed` stores `evil.com`), so rows an earlier CLI wrote verbatim as `evil[.]com` are no longer reachable from `search known -d evil[.]com`, which now sends `evil.com`. And `scan url` / `sandbox url` given a defanged URL submit a different artifact (content, `artifact_name`, sha — and the quota it costs) than the same invocation did before. Pass `--no-refang` to keep the verbatim behaviour, e.g. to reach catalogue rows stored defanged.
+
+Two places handle the value in the CLI and call `utils.refang_input(api, value)` — the SDK's `refang_ioc`, gated on `api.refang_iocs`:
+
+- **Validation before the SDK sees the value.** `scan url` and `sandbox url` check positional URLs with `is_url`; they validate the refanged form, so `hxxps[:]//evil[.]com` is accepted instead of rejected as invalid. With `--no-refang` the defanged URL is still rejected, exactly as before.
+- **CLI-owned requests.** `metadata analyze-ip` goes through `Polyswarm.submit_url`, which builds its request with `_single` rather than an SDK endpoint method, so it refangs explicitly.
+
+Hashes and ids are never touched. Neither is a `--qrcode-file` path on `scan url` / `sandbox url`, and that exemption lives in the SDK, not in CLI code: a `preprocessing={'type': 'qrcode'}` submission skips refanging entirely (pinned here by the two qrcode tests in `tests/refang_test.py`). Tests: `tests/refang_test.py`, mocking at the SDK transport so the SDK's own refang is exercised — see [`04-testing.md`](./04-testing.md) §Style 4.
+
 ## Adding to the catalogue
 
 When you add or materially change a group, update its row (and add per-subcommand detail here if the behaviour is non-obvious). The `AGENTS.md` §"When adding a new command family" checklist covers the wiring + formatter + test steps.

specs/04-testing.md

@@ -2,12 +2,12 @@
 
 ## Scope
 
-How the CLI is tested: the `CliRunner` harness, the two mocking styles (SDK-boundary mocks vs VCR cassettes), the formatter-unit style for pure rendering, the cassette layout and record workflow, and how to run the suite. Files: `tests/`, `tests/vcr/`, `src/conftest.py`, `pyproject.toml` (`[project.optional-dependencies].tests`, `[tool.pytest.ini_options]`).
+How the CLI is tested: the `CliRunner` harness, the mocking styles (SDK-boundary mocks, SDK-transport mocks, VCR cassettes), the formatter-unit style for pure rendering, the cassette layout and record workflow, and how to run the suite. Files: `tests/`, `tests/vcr/`, `src/conftest.py`, `pyproject.toml` (`[project.optional-dependencies].tests`, `[tool.pytest.ini_options]`).
 
 ## Invariants
 
 - **Anything that is command behaviour is driven through `click.testing.CliRunner`** — argument parsing, the SDK call, the wiring, the exit code: exercise the real command tree, never an internal function standing in for it. No live PolySwarm stack is required. The one sanctioned exception is pure rendering logic — see [Style 3](#style-3--formatter-unit-tests).
-- **Mock at the SDK boundary, or replay HTTP with VCR — never both for the same path.** A test either patches `polyswarm_api.api.PolyswarmAPI.<method>` (unit-style) or lets VCR replay recorded HTTP (end-to-end). The CLI's own code is exercised either way.
+- **Mock at exactly one point per path — the SDK method, the SDK transport, or VCR.** A test patches `polyswarm_api.api.PolyswarmAPI.<method>` (unit-style, the default), patches the SDK transport `PolyswarmSession.execute` when the behaviour under test runs *inside* the SDK method ([Style 4](#style-4--sdk-transport-mocks)), or lets VCR replay recorded HTTP (end-to-end). Never two of them for the same path. The CLI's own code is exercised either way.
 
 - **VCR is an efficiency cache, not a load-bearing requirement.** The suite must pass against a live e2e stack with VCR off. Don't hardcode `record_mode='none'`; if a test only works against its recorded cassette, that's a bug in the test. Note this is about a test's *logic*, not its fixtures: a `.click` snapshot pins server-generated ids and timestamps, so re-recording needs a stack in a particular state — see [Re-recording a cassette](#re-recording-a-cassette).
 
@@ -84,6 +84,14 @@ For **rendering logic with no command-tree behaviour** — which labelled line a
 
 Use it **only** for that. Argument parsing, SDK calls, generator consumption, `ctx.obj` wiring and exit codes are command behaviour: a formatter unit test can't observe them, so those need Style 1 or Style 2. A command whose rendering is covered by Style 3 still needs at least one `CliRunner` test proving the command reaches the formatter at all.
 
+## Style 4 — SDK-transport mocks
+
+For behaviour that happens **inside** an SDK endpoint method, a Style 1 mock proves nothing: patching `PolyswarmAPI.search_url` replaces the very code under test. IoC refanging is the case today — the SDK refangs `search_url`'s argument (and the other URL / domain / IP inputs) before it builds the request, so the observable effect is the request that reaches the transport.
+
+Patch `polyswarm_api.session.PolyswarmSession.execute` — the SDK's documented session customization point — and assert on the `PolyswarmRequest` it receives: `request.params` (query string) and `request.input_json` (JSON body), read-only. Example: `tests/refang_test.py`.
+
+Use this style only when Style 1 would bypass the behaviour under test. Moving these tests "up" to endpoint-method mocks would keep them green while dropping their coverage. The dependency it adds (the `execute(request)` seam and those two request fields) is recorded in [`05-sdk-contract.md`](./05-sdk-contract.md); an SDK rename there breaks these tests, which is the intended signal.
+
 ## What to test for a new command
 
 1. The command parses its arguments and calls the expected SDK method with the expected arguments.