FEAT: Policy Scorer Compatibility (phase 2.5) (#2518)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: hannahwestra25 <hannahwestra@microsoft.com>
Copilot-Session: 9cd5233d-054f-4496-8237-4dd41b2ed544
Copilot-Session: 5e2a5ec1-82af-4b4b-a69c-3306c1507164

Author: 3 people

doc/code/framework.md

@@ -258,7 +258,7 @@ If you are contributing to PyRIT, that work will most likely land in one of the
 - Any decision an attack makes should be based on a scorer result
 - A scorer is not limited to a message, it could be anything (e.g. was this tool called or was this file written). It receives a `Scorable`, which identifies that evidence, and an optional `ScoringExpectation`.
 - `TrueFalseScorer` and `FloatScaleScorer` define result families. `MessageScorer` adds message resolution and message-only policy on top of them.
-- `Score.status` marks a verdict complete or undetermined, and the attack decides how to branch on it.
+- A scorer declares which evidence it reads, rather than the caller filtering evidence for it. A `MessageScorer` states the conversation roles and data types it reads on its `ScorerPromptValidator`.
 - **Does not own**: acting on its own result. A scorer evaluates a response and returns a score; branching on that score is the attack's job, and aggregating scores across runs is analytics'. It may call a target to evaluate, but it doesn't send the attack's objective prompt or manage the conversation.
 
 **Framework Plans**:

doc/code/scoring/0_scoring.ipynb

@@ -219,11 +219,24 @@
     "storage and stores its SHA-256 digest. The score remains resolvable after the source file is\n",
     "removed.\n",
     "\n",
-    "A complete score has `status=\"complete\"` and a typed value. An undetermined score has\n",
-    "`status=\"undetermined\"` and no value. A fully blocked response is a complete negative result\n",
-    "by default: `False` for message true/false scorers and `0.0` for message float-scale scorers.\n",
-    "`SelfAskRefusalScorer` is the intentional exception because a content-filter block is a\n",
-    "refusal, so it returns `True`. Other response errors remain undetermined."
+    "Scoring APIs return `list[Score]`. An empty list means that the scorer does not apply to the\n",
+    "evidence, such as a message with no supported role or data type. A non-empty list contains\n",
+    "completed or undetermined scores.\n",
+    "\n",
+    "A complete score has `status=\"complete\"` and a typed domain verdict. An undetermined score\n",
+    "has `status=\"undetermined\"` and no value because supported evidence failed to load. A fully\n",
+    "blocked response is a complete negative result by default: `False` for message true/false\n",
+    "scorers and `0.0` for message float-scale scorers. `SelfAskRefusalScorer` is the intentional\n",
+    "exception because a content-filter block is a refusal, so it returns `True`.\n",
+    "\n",
+    "A scorer declares which evidence it reads; the caller does not filter evidence on its behalf.\n",
+    "A message scorer names the conversation roles it reads with `supported_roles` on its\n",
+    "`ScorerPromptValidator`. Prepended (`simulated_assistant`) turns are fabricated history, so a\n",
+    "scorer must opt in to read them. Every scorer still receives a failed response, because a\n",
+    "scorer whose evidence never came from the response must run even when the response failed.\n",
+    "Explicit `role_filter` and `skip_on_error_result` values remain supported until removal.\n",
+    "Callers that rely on their historical defaults must now pass them explicitly. New code\n",
+    "should use `supported_roles` and the scorer's unreadable-evidence fallback instead."
    ]
   },
   {

doc/code/scoring/0_scoring.py

@@ -121,11 +121,24 @@
 # storage and stores its SHA-256 digest. The score remains resolvable after the source file is
 # removed.
 #
-# A complete score has `status="complete"` and a typed value. An undetermined score has
-# `status="undetermined"` and no value. A fully blocked response is a complete negative result
-# by default: `False` for message true/false scorers and `0.0` for message float-scale scorers.
-# `SelfAskRefusalScorer` is the intentional exception because a content-filter block is a
-# refusal, so it returns `True`. Other response errors remain undetermined.
+# Scoring APIs return `list[Score]`. An empty list means that the scorer does not apply to the
+# evidence, such as a message with no supported role or data type. A non-empty list contains
+# completed or undetermined scores.
+#
+# A complete score has `status="complete"` and a typed domain verdict. An undetermined score
+# has `status="undetermined"` and no value because supported evidence failed to load. A fully
+# blocked response is a complete negative result by default: `False` for message true/false
+# scorers and `0.0` for message float-scale scorers. `SelfAskRefusalScorer` is the intentional
+# exception because a content-filter block is a refusal, so it returns `True`.
+#
+# A scorer declares which evidence it reads; the caller does not filter evidence on its behalf.
+# A message scorer names the conversation roles it reads with `supported_roles` on its
+# `ScorerPromptValidator`. Prepended (`simulated_assistant`) turns are fabricated history, so a
+# scorer must opt in to read them. Every scorer still receives a failed response, because a
+# scorer whose evidence never came from the response must run even when the response failed.
+# Explicit `role_filter` and `skip_on_error_result` values remain supported until removal.
+# Callers that rely on their historical defaults must now pass them explicitly. New code
+# should use `supported_roles` and the scorer's unreadable-evidence fallback instead.
 # %% [markdown]
 # ## Scoring directly
 #

doc/code/scoring/3_combining_scorers.ipynb

@@ -100,6 +100,12 @@
     "text `ContentScorable` evidence. It returns a dynamic wrapper that remains the same scorer\n",
     "kind as its input.\n",
     "\n",
+    "An empty child result means that the scorer did not apply. A composite scorer ignores empty\n",
+    "child results and aggregates the remaining results. It returns an empty list if every child\n",
+    "result is empty. Inverter and threshold wrappers pass an empty result through unchanged.\n",
+    "A conversation wrapper returns an empty result when it finds no applicable conversation\n",
+    "evidence or its child returns no score. Any outer wrapper then applies the rules above.\n",
+    "\n",
     "Deprecated message-shaped calls remain on `MessageScorer`, but generic wrappers do not\n",
     "project those APIs from their children. Score wrappers through the canonical `Scorable` API.\n",
     "\n",

doc/code/scoring/3_combining_scorers.py

@@ -83,6 +83,12 @@
 # text `ContentScorable` evidence. It returns a dynamic wrapper that remains the same scorer
 # kind as its input.
 #
+# An empty child result means that the scorer did not apply. A composite scorer ignores empty
+# child results and aggregates the remaining results. It returns an empty list if every child
+# result is empty. Inverter and threshold wrappers pass an empty result through unchanged.
+# A conversation wrapper returns an empty result when it finds no applicable conversation
+# evidence or its child returns no score. Any outer wrapper then applies the rules above.
+#
 # Deprecated message-shaped calls remain on `MessageScorer`, but generic wrappers do not
 # project those APIs from their children. Score wrappers through the canonical `Scorable` API.
 #

pyrit/executor/attack/multi_turn/crescendo.py

@@ -713,9 +713,7 @@ async def _score_response_async(self, *, context: CrescendoAttackContext) -> Sco
                 response=context.last_response,
                 objective_scorer=self._objective_scorer,
                 auxiliary_scorers=self._auxiliary_scorers,
-                role_filter="assistant",
                 objective=context.objective,
-                skip_on_error_result=False,
             )
 
         objective_score = scoring_results["objective_scores"]

pyrit/executor/attack/multi_turn/multi_prompt_sending.py

@@ -408,9 +408,7 @@ async def _evaluate_response_async(self, *, response: Message, objective: str) -
                 response=response,
                 auxiliary_scorers=self._auxiliary_scorers,
                 objective_scorer=self._objective_scorer if self._objective_scorer else None,
-                role_filter="assistant",
                 objective=objective,
-                skip_on_error_result=True,
             )
 
         objective_scores = scoring_results["objective_scores"]

pyrit/executor/attack/multi_turn/tree_of_attacks.py

@@ -804,12 +804,11 @@ async def _score_response_async(self, *, response: Message, objective: str) -> N
         and any auxiliary scorers (which provide additional metrics). The scoring results are
         used by the TAP algorithm to decide which branches to explore further.
 
-        Blocked or errored responses are scored via the scorer's unified default behavior:
-        ``TrueFalseScorer`` returns
-        ``Score(False)`` and ``FloatScaleScorer``
-        returns ``Score(0.0)`` whenever no supported pieces remain after validator filtering
-        (the normal outcome for a blocked piece). This keeps blocked branches at the bottom
-        of the priority queue without needing attack-level error mapping.
+        Scorers apply their own unreadable-response policy. A fully blocked response uses the
+        scorer family's neutral fallback unless the scorer overrides it. An unreadable transport
+        or protocol response produces an undetermined score. A response with no supported role
+        or data type makes the scorer return ``[]``, so this method raises ``RuntimeError``. Tree
+        of Attacks does not map these outcomes to ``False`` or ``0.0``.
 
         Args:
             response (Message): The response from the objective target to evaluate.
@@ -841,9 +840,7 @@ async def _score_response_async(self, *, response: Message, objective: str) -> N
                 response=response,
                 objective_scorer=self._objective_scorer,
                 auxiliary_scorers=self._auxiliary_scorers,
-                role_filter="assistant",
                 objective=objective,
-                skip_on_error_result=False,
             )
 
         # Extract objective score

pyrit/executor/attack/single_turn/prompt_sending.py

@@ -367,9 +367,7 @@ async def _evaluate_response_async(
                 response=response,
                 objective_scorer=self._objective_scorer,
                 auxiliary_scorers=self._auxiliary_scorers,
-                role_filter="assistant",
                 objective=objective,
-                skip_on_error_result=True,
             )
 
         if not self._objective_scorer:

pyrit/score/__init__.py

@@ -39,7 +39,7 @@
     from pyrit.score.float_scale.system_prompt_extraction_scorer import SystemPromptExtractionScorer
     from pyrit.score.float_scale.video_float_scale_scorer import VideoFloatScaleScorer
     from pyrit.score.message_scorable_resolver import MessageScorableResolver
-    from pyrit.score.message_scorer import MessageScorer, MessageScoringOptions
+    from pyrit.score.message_scorer import MessageScorer
     from pyrit.score.response_handler import CallableResponseHandler, JsonSchemaResponseHandler, ResponseHandler
     from pyrit.score.scorable import ContentScorable, MessageScorable, Scorable
     from pyrit.score.scorer import Scorer
@@ -176,7 +176,6 @@
     "MessageScorableResolver": "pyrit.score.message_scorable_resolver",
     "MessageScorable": "pyrit.score.scorable",
     "MessageScorer": "pyrit.score.message_scorer",
-    "MessageScoringOptions": "pyrit.score.message_scorer",
     "MethKeywordScorer": "pyrit.score.true_false.regex.meth_keyword_scorer",
     "MetricsType": "pyrit.score.scorer_evaluation.metrics_type",
     "NerveAgentKeywordScorer": "pyrit.score.true_false.regex.nerve_agent_keyword_scorer",