Release 4.8.0 with metadata search & historical hunts updates

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "polyswarm_api"
-version = "4.7.0"
+version = "4.8.0"
 description = "Client library to simplify interacting with the PolySwarm consumer API"
 readme = "README.md"
 requires-python = ">=3.10,<4"
@@ -55,7 +55,7 @@ package-dir = { "" = "src" }
 where = ["src"]
 
 [tool.bumpversion]
-current_version = "4.7.0"
+current_version = "4.8.0"
 commit = true
 tag = false
 sign_tags = true

specs/02-resources.md

@@ -31,6 +31,7 @@ BaseResource                          # holds .api, ._content, .parse_result cla
     ├── Metadata                      # /search/metadata/query
     ├── MetadataFieldProperties       # /search/metadata/properties
     ├── MetadataMapping               # /search/metadata/mappings
+    ├── MetadataRawSearch             # /search/metadata/raw
     ├── IOC                           # /ioc (known good / known bad / search)
     ├── LiveYaraRuleset               # /hunt/rule/live
     ├── HistoricalHunt                # /hunt/historical
@@ -388,7 +389,7 @@ Holds `handle`, `artifact_name`, `artifact_type`, `sha256`, `sha1`, `md5`. Also
 Several resources add domain-specific classmethods on top of the standard CRUD set:
 
 - `IOC` — `iocs_by_hash`, `ioc_search`, `check_known_hosts`, `create_known_good`, `create_known_bad`, `update_known_good`, `delete_known_good`.
-- `LiveYaraRuleset` / `HistoricalHunt` / `YaraRuleset` — standard CRUD plus list/delete-batch variants. `YaraRuleset` also parses the hunt-page tracking fields (`favorite`, `favorited_at`, `rule_count` — `None` means the server had no answer, distinct from 0 — `historical_hunt_count`, and the stored `new_results_count` with its staleness marker `new_results_counted_at`: the server refreshes the counter on a schedule and the marker says when, so `None` means "not yet refreshed / no live hunt", never 0), and `HistoricalHunt` the source-rule provenance (`rule_id`, `rule_modified`, and the tri-state `source_rule_changed` — `None` is "unknown", never "unchanged"). All additive `.get()` parses; an older server leaves them `None`.
+- `LiveYaraRuleset` / `HistoricalHunt` / `YaraRuleset` — standard CRUD plus list/delete-batch variants. `YaraRuleset` also parses the hunt-page tracking fields (`favorite`, `favorited_at`, `rule_count` — `None` means the server had no answer, distinct from 0 — `historical_hunt_count`, and the stored `new_results_count` with its staleness marker `new_results_counted_at`: the server refreshes the counter on a schedule and the marker says when, so `None` means "not yet refreshed / no live hunt", never 0), and `HistoricalHunt` the source-rule provenance (`rule_id`, `rule_modified`, and the tri-state `source_rule_changed` — `None` is "unknown", never "unchanged") and its time window `window_days` (4.8.0: the int the hunt was created with, on the single GET and the list alike; `None` means no window was requested, so the hunt reaches the platform maximum). All additive `.get()` parses; an older server leaves them `None`.
 - `YaraRulesetFavorite` — `RESOURCE_ENDPOINT = '/hunt/rule/favorite'`, **`RESOURCE_ID_KEYS = ['community']`**: a deliberate deviation from the default `['id']`, splitting the request the way the server reads it — the toggle reads BOTH `id` and `favorite` from the PUT **body** (the default would move `id` into the query string and the server would 400 with "A valid rule id must be provided"), while `community` rides the **query string** to match where the ruleset GET/list calls send it. (The server's community middleware accepts the value from either the query or the body — but never both at once — so the placement here is about consistency and the id-400, not about a silently-ignored body value.) `favorite` serialises as `1`/`0` (the `_params` bool→int coercion), which the server's boolean parser accepts. Response carries the star state plus the team's `favorites_used`/`favorites_limit`. Pinned by `test/hunt_tracking_builder_test.py` and both transports of `test/ruleset_favorite_respx_test.py`.
 - `SandboxTask` — `create_file`, `update_file`, `latest`, `my_tasks` for the various sandbox-submission shapes. **No `upload_file` instance method** in 4.0.
 - `Sample` — `create` with `endpoint_fmt={'sha256': sha256}` for the URL-parametrised path.

specs/03-endpoints.md

@@ -35,6 +35,7 @@ The full catalogue of methods on the public client surface and which transport h
 | Method | Resource builder |
 |---|---|
 | `metadata_mapping()` | inline (GET) — parsed into `MetadataMapping` |
+| `metadata_raw_search(body, community=None)` | `MetadataRawSearch.search` (POST). `body` is sent verbatim: it is an OpenSearch `_search` body, and the server owns the grammar it accepts. `community` rides the **query string**, not the body, because the server overlays the community filter itself. |
 | `metadata_field_properties_write(field_path, description, …)` | inline (POST) |
 | `metadata_field_properties_get(field_path)` | inline (GET) |
 | `metadata_field_properties_delete(field_path)` | inline (DELETE) |
@@ -71,7 +72,7 @@ Internal-only CRUD for the `/known-good` binary resource (distinct from the IOC
 
 | Method | Resource builder |
 |---|---|
-| `historical_create(rule=None, ruleset_name=None)` | `HistoricalHunt.create` |
+| `historical_create(rule=None, ruleset_name=None, window_days=None)` | `HistoricalHunt.create` — `window_days` (4.8.0) is how many days back the hunt reaches; the server accepts 30, 60, 90 or 180 and answers anything else with a 400, which surfaces as the usual typed exception (the SDK does not validate it). It is an int and rides the JSON body. `None` leaves the key out, the server keeps its default and the hunt reaches the platform maximum, so a bare call sends exactly what it always did. |
 | `historical_get(hunt=None)` | `HistoricalHunt.get` |
 | `historical_update(hunt)` | `HistoricalHunt.update` |
 | `historical_delete(hunt)` | `HistoricalHunt.delete` |

specs/04-testing.md

@@ -26,12 +26,14 @@ How the test suite is organised. Three layers: pure unit tests (no HTTP at all 
 - `test/async_client_test.py` — async, VCR-backed integration tests (not yet on the parametrised harness — follow-up work).
 - `test/hunt_matched_strings_test.py` — pure-unit tests for the three-state `matched_strings` contract and its `matched_strings_dropped` sibling, across all four hunt-result classes. The endpoint behaviour they cannot see (that the detail route actually emits the keys and the list route sends `null`) is pinned live in `client_scan_test.py::test_live` / `async_client_test.py::test_async_live` — the e2e-first + pure-unit pairing invariant 1 asks for.
 - `test/known_good_test.py` — pure-unit tests for the known-good resource fields.
+- `test/metadata_raw_search_test.py` — pure-unit request-shape and parse tests for `metadata_raw_search`: that the query body is sent verbatim and `community` rides the query string (invariant 7 — the builder is pure data, so the routing reads straight off the `PolyswarmRequest`'s `request_parameters`, with no httpx), plus the `partial` reason codes the server returns. The lifecycle is live, in `client_scan_test.py` / `async_client_test.py`.
 - `test/jmespath_test.py` — unit tests for `BaseJsonResource.jmespath`.
 - `test/vcr/*.vcr` — recorded cassettes.
 - `test/malicious` — fixture file for upload tests (`test/eicar.yara` was retired when the rules tests moved to per-test `uid_yara` bodies).
 - `test/hunt_tracking_builder_test.py` — pure-unit request-shape and parse tests for the hunt-page tracking builders/resources.
 - `test/refang_test.py` — pure-unit tests for `polyswarm_api.refang` (driven by the shared case table `test/fixtures/refang_cases.json`, kept byte-identical with the other clients that implement the same contract) and request-shape tests for every client method that refangs its IoC inputs, captured at the `_paginate` / `_single` boundary for both transports. Pure-unit because this is client-side input normalization: the server contract is unchanged, so there is no new endpoint behaviour for a cassette to pin.
 - `test/ioc_search_test.py` — `search_by_ioc(with_artifacts=)`: pure-unit request shape (the int flag, the unchanged default and bare call), pass-through on both client methods, and a dual-transport (`ClientTestCase`) respx parse of an artifact row — a stand-in until the e2e stack serves the parameter (see `99-open-questions.md`).
+- `test/historical_window_test.py` — `historical_create(window_days=)` and `HistoricalHunt.window_days`: pure-unit request shape (the int in the body, the key absent for `None`, no client-side validation) and parse (int, explicit null, missing key — on `HistoricalHunt` and `HistoricalHuntList`), plus dual-transport (`ClientTestCase`) respx cases for the body each client sends — respx because VCR does not match bodies. What the server returns is pinned live by `test_historical_window_days` / `test_async_historical_window_days` (a hunt with a window and one without, read back on create, get and list).
 - `test/ruleset_favorite_respx_test.py` — dual-transport (`ClientTestCase`) respx suite for the favorite toggle: the `FAVORITE_LIMIT` refusal envelope and the query/body split.
 - `test/sandbox_file_qrcode_respx_test.py` — dual-transport (`ClientTestCase`) respx suite pinning that a QR-code `sandbox_file` submission uploads the image's bytes (the S3 PUT body) under its unrefanged basename. Respx rather than e2e because the stack's sandbox providers cannot reasonably process a QR image.
 

src/polyswarm_api/__init__.py

@@ -1,5 +1,5 @@
 # https://www.python.org/dev/peps/pep-0008/#module-level-dunder-names
-__version__ = '4.7.0'
+__version__ = '4.8.0'
 __release_url__ = 'https://api.github.com/repos/polyswarm/polyswarm-api/releases/latest'
 
 from . import api

src/polyswarm_api/aio/api.py

@@ -288,6 +288,51 @@ async def metadata_mapping(self):
             result_parser=resources.MetadataMapping,
         )
 
+    async def metadata_raw_search(self, body, community=None):
+        """Run a raw OpenSearch query body against the metadata corpus.
+
+        The server overlays the community filter itself and caps the work, so
+        ``body`` is a plain ``_search`` body and the community travels in the
+        query string rather than in it. The accepted grammar is closed: a
+        construct the server does not recognise is refused rather than passed
+        through, and so is a top-level ``community``/``communities`` key.
+
+        Entitlement-gated and metered per query, including a query the server
+        refuses after running it.
+
+        :param body: An OpenSearch ``_search`` body (``query``, ``aggs``,
+            ``size``, ``sort``, ``_source``, ...). Omitting ``size`` returns
+            **no documents**: the server defaults it to 0 so an aggregation
+            caller does not pay for hits they did not ask for. Ask for hits
+            by setting it.
+        :param community: Community to search; defaults to the client's.
+        :return: A ``MetadataRawSearch`` whose ``json`` carries the engine's
+            ``hits`` and ``aggregations``, plus ``partial`` when the server
+            could not establish that the answer is complete.
+
+            ``partial`` is a **list to inspect, not a flag**. Its entries are
+            ``{'code': ..., 'message': ...}`` and they make different claims,
+            so branch on ``code`` - ``message`` is for a human and may be
+            reworded. The codes are a closed set:
+
+            * ``timed_out`` / ``terminated_early`` - the search stopped early.
+            * ``shards_failed`` - a shard was lost; also carries ``failed``.
+            * ``buckets_clamped`` - a ``terms`` list dropped terms.
+            * ``composite_paged`` - a ``composite`` may have more pages; page
+              it with its ``after_key``.
+            * ``bucket_count_approximate`` - a ``terms`` count is approximate
+              rather than truncated.
+            * ``hit_total_capped`` - the hit counter stopped at
+              ``track_total_hits``. This is what *every* large query that asks
+              for hits returns, so on its own it says nothing about the
+              aggregations; the server omits it when the body asked for no
+              documents. ``hits.total.relation`` carries the same fact as the
+              engine's own field, ``eq`` exact and ``gte`` capped.
+        """
+        logger.info('Running a raw metadata search')
+        return await self._single(
+            resources.MetadataRawSearch.search(self, body, community))
+
     async def metadata_field_properties_write(self, field_path, description,
                                         example=None, category=None, aliases=None):
         """Upsert a metadata field properties entry (the single write path).
@@ -597,18 +642,21 @@ async def live_result(self, result_id):
         """
         return await self._single(resources.LiveHuntResult.get(self, id=result_id))
 
-    async def historical_create(self, rule=None, ruleset_name=None):
+    async def historical_create(self, rule=None, ruleset_name=None, window_days=None):
         """
         Run a new historical hunt.
 
         :param rule: YaraRuleset object or string containing YARA rules to install
         :param ruleset_name: Name of the ruleset.
+        :param window_days: Days back the hunt reaches: 30, 60, 90 or 180. Omit it for the
+            platform maximum. The server validates the value; the SDK does not.
         :return: The created Hunt resource
         """
         logger.info('Create historical hunt %s', rule)
         rule, rule_id = self._parse_rule(rule)
         return await self._single(resources.HistoricalHunt.create(self, yara=rule.yara if rule else None, rule_id=rule_id,
-                                               ruleset_name=ruleset_name, community=self.community))
+                                               ruleset_name=ruleset_name, window_days=window_days,
+                                               community=self.community))
 
     async def historical_get(self, hunt=None):
         """

src/polyswarm_api/api.py

@@ -303,6 +303,50 @@ def metadata_mapping(self):
             result_parser=resources.MetadataMapping,
         )
 
+    def metadata_raw_search(self, body, community=None):
+        """Run a raw OpenSearch query body against the metadata corpus.
+
+        The server overlays the community filter itself and caps the work, so
+        ``body`` is a plain ``_search`` body and the community travels in the
+        query string rather than in it. The accepted grammar is closed: a
+        construct the server does not recognise is refused rather than passed
+        through, and so is a top-level ``community``/``communities`` key.
+
+        Entitlement-gated and metered per query, including a query the server
+        refuses after running it.
+
+        :param body: An OpenSearch ``_search`` body (``query``, ``aggs``,
+            ``size``, ``sort``, ``_source``, ...). Omitting ``size`` returns
+            **no documents**: the server defaults it to 0 so an aggregation
+            caller does not pay for hits they did not ask for. Ask for hits
+            by setting it.
+        :param community: Community to search; defaults to the client's.
+        :return: A ``MetadataRawSearch`` whose ``json`` carries the engine's
+            ``hits`` and ``aggregations``, plus ``partial`` when the server
+            could not establish that the answer is complete.
+
+            ``partial`` is a **list to inspect, not a flag**. Its entries are
+            ``{'code': ..., 'message': ...}`` and they make different claims,
+            so branch on ``code`` - ``message`` is for a human and may be
+            reworded. The codes are a closed set:
+
+            * ``timed_out`` / ``terminated_early`` - the search stopped early.
+            * ``shards_failed`` - a shard was lost; also carries ``failed``.
+            * ``buckets_clamped`` - a ``terms`` list dropped terms.
+            * ``composite_paged`` - a ``composite`` may have more pages; page
+              it with its ``after_key``.
+            * ``bucket_count_approximate`` - a ``terms`` count is approximate
+              rather than truncated.
+            * ``hit_total_capped`` - the hit counter stopped at
+              ``track_total_hits``. This is what *every* large query that asks
+              for hits returns, so on its own it says nothing about the
+              aggregations; the server omits it when the body asked for no
+              documents. ``hits.total.relation`` carries the same fact as the
+              engine's own field, ``eq`` exact and ``gte`` capped.
+        """
+        logger.info("Running a raw metadata search")
+        return self._single(resources.MetadataRawSearch.search(self, body, community))
+
     def metadata_field_properties_write(
         self, field_path, description, example=None, category=None, aliases=None
     ):
@@ -691,12 +735,14 @@ def live_result(self, result_id):
         """
         return self._single(resources.LiveHuntResult.get(self, id=result_id))
 
-    def historical_create(self, rule=None, ruleset_name=None):
+    def historical_create(self, rule=None, ruleset_name=None, window_days=None):
         """
         Run a new historical hunt.
 
         :param rule: YaraRuleset object or string containing YARA rules to install
         :param ruleset_name: Name of the ruleset.
+        :param window_days: Days back the hunt reaches: 30, 60, 90 or 180. Omit it for the
+            platform maximum. The server validates the value; the SDK does not.
         :return: The created Hunt resource
         """
         logger.info("Create historical hunt %s", rule)
@@ -707,6 +753,7 @@ def historical_create(self, rule=None, ruleset_name=None):
                 yara=rule.yara if rule else None,
                 rule_id=rule_id,
                 ruleset_name=ruleset_name,
+                window_days=window_days,
                 community=self.community,
             )
         )

src/polyswarm_api/resources.py

@@ -77,6 +77,25 @@ class MetadataMapping(core.BaseJsonResource):
     RESOURCE_ENDPOINT = '/search/metadata/mappings'
 
 
+class MetadataRawSearch(core.BaseJsonResource):
+    RESOURCE_ENDPOINT = '/search/metadata/raw'
+
+    @classmethod
+    def search(cls, api, body, community=None):
+        """``body`` is the caller's whole query and travels untouched. The
+        community is scope, not query, and the server reads it from the query
+        string - folding it into the body would pick scope with a key the
+        server's grammar then rejects."""
+        return core.PolyswarmRequest(
+            api=api,
+            method='POST',
+            url=f'{api.uri}{cls.RESOURCE_ENDPOINT}',
+            params={'community': community or api.community},
+            json=body,
+            result_parser=cls,
+        )
+
+
 class MetadataFieldProperties(core.BaseJsonResource):
     RESOURCE_ENDPOINT = '/search/metadata/properties'
     RESOURCE_ID_KEYS = ['field_path']
@@ -853,6 +872,8 @@ def __init__(self, content, api=None):
         # unknown (no source rule, or nothing to compare against) — not
         # "unchanged".
         self.source_rule_changed = content.get('source_rule_changed')
+        # None: no window requested (platform maximum), or a server predating the field.
+        self.window_days = content.get('window_days')
 
 
 class HistoricalHuntList(HistoricalHunt):