Skip to content

Commit 446c643

Browse files
BolorCopilot
Bolor
and
Copilot
committed
merging main
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2 parents 3dd9a6d + ba944ee commit 446c643

425 files changed

Lines changed: 43736 additions & 326 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.pyrit_conf_example‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,6 @@ memory_db_type: sqlite
2828
# - scorers: Registers pre-configured scorers into the ScorerRegistry
2929
# - load_default_datasets: Loads default datasets for all registered scenarios
3030
# - objective_list: Sets default objectives for scenarios
31-
# - openai_objective_target: Sets up OpenAI target for scenarios
3231
#
3332
# Each initializer can be specified as:
3433
# - A simple string (name only)

‎PAIR_REFACTORING_SUMMARY.md‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
# PAIR Attack Refactoring Summary
2+
3+
## Overview
4+
5+
Successfully refactored the PAIR (Prompt Automatic Iterative Refinement) orchestrator from `pair_orchestrator.py` to become `PAIRAttack` following the new attack strategy pattern. The refactored attack inherits from `AttackStrategy` and maintains all core PAIR functionality while removing orchestrator-specific dependencies.
6+
7+
## Files Created/Modified
8+
9+
### 1. `/pyrit/attacks/multi_turn/pair.py` (NEW)
10+
- **PAIRAttackContext**: Context class extending `MultiTurnAttackContext` with PAIR-specific parameters
11+
- **PAIRAttackResult**: Result class extending `AttackResult` with PAIR-specific metadata
12+
- **PAIRAttack**: Main attack strategy class extending `AttackStrategy[PAIRAttackContext, PAIRAttackResult]`
13+
14+
### 2. `/pyrit/attacks/__init__.py` (UNCHANGED)
15+
- `PairAttack` already included in `__all__` list, ready for the refactored implementation
16+
17+
## Key Components
18+
19+
### PAIRAttackContext
20+
```python
21+
@dataclass
22+
class PAIRAttackContext(MultiTurnAttackContext):
23+
depth: int = 3
24+
desired_response_prefix: str = "Sure, here is"
25+
objective_achieved_score_threshold: float = 0.8
26+
```
27+
28+
### PAIRAttackResult
29+
```python
30+
@dataclass
31+
class PAIRAttackResult(AttackResult):
32+
@property
33+
def depth_reached(self) -> int: ...
34+
```
35+
36+
### PAIRAttack
37+
```python
38+
class PAIRAttack(AttackStrategy[PAIRAttackContext, PAIRAttackResult]):
39+
def __init__(
40+
self,
41+
*,
42+
objective_target: PromptChatTarget,
43+
attack_adversarial_config: AttackAdversarialConfig,
44+
attack_converter_config: Optional[AttackConverterConfig] = None,
45+
attack_scoring_config: Optional[AttackScoringConfig] = None,
46+
prompt_normalizer: Optional[PromptNormalizer] = None,
47+
depth: int = 3,
48+
desired_response_prefix: str = "Sure, here is",
49+
objective_achieved_score_threshold: float = 0.8,
50+
): ...
51+
```
52+
53+
## Implementation Strategy
54+
55+
The refactored `PAIRAttack` leverages the existing `TreeOfAttacksWithPruningAttack` with PAIR-specific configuration:
56+
57+
1. **Delegation Pattern**: Creates an internal `TreeOfAttacksWithPruningAttack` instance
58+
2. **PAIR Configuration**:
59+
- `tree_width=1` (PAIR uses single branch)
60+
- `branching_factor=1` (PAIR doesn't branch)
61+
- `on_topic_checking_enabled=False` (PAIR disables this)
62+
- Uses PAIR-specific system prompts
63+
64+
3. **Attack Strategy Lifecycle**:
65+
- `_validate_context()`: Validates PAIR-specific parameters
66+
- `_setup_async()`: Creates and configures the underlying tree attack
67+
- `_perform_attack_async()`: Executes the tree attack and converts results
68+
- `_teardown_async()`: Cleans up resources
69+
70+
## API Migration
71+
72+
### Old PAIROrchestrator
73+
```python
74+
orchestrator = PAIROrchestrator(
75+
objective_target=objective_target,
76+
adversarial_chat=adversarial_chat,
77+
scoring_target=scoring_target,
78+
depth=3,
79+
prompt_converters=[...],
80+
objective_achieved_score_threshold=0.8,
81+
desired_response_prefix="Sure, here is",
82+
)
83+
result = await orchestrator.run_attack_async(objective="...")
84+
```
85+
86+
### New PAIRAttack
87+
```python
88+
attack = PAIRAttack(
89+
objective_target=objective_target,
90+
attack_adversarial_config=AttackAdversarialConfig(target=adversarial_chat),
91+
attack_converter_config=AttackConverterConfig(request_converters=[...]),
92+
attack_scoring_config=AttackScoringConfig(objective_scorer=...),
93+
depth=3,
94+
desired_response_prefix="Sure, here is",
95+
objective_achieved_score_threshold=0.8,
96+
)
97+
result = await attack.execute_async(objective="...")
98+
```
99+
100+
## Key Improvements
101+
102+
1. **Attack Strategy Pattern Compliance**:
103+
- Enforced lifecycle with validation, setup, execution, and teardown
104+
- Consistent interface with other attack strategies
105+
- Better error handling and resource management
106+
107+
2. **Modular Configuration**:
108+
- `AttackAdversarialConfig`: Manages adversarial chat configuration
109+
- `AttackConverterConfig`: Manages prompt converters
110+
- `AttackScoringConfig`: Manages scoring configuration
111+
112+
3. **Enhanced Type Safety**:
113+
- Generic typing with `AttackStrategy[PAIRAttackContext, PAIRAttackResult]`
114+
- Better IDE support and error detection
115+
116+
4. **Comprehensive Results**:
117+
- `PAIRAttackResult` with PAIR-specific metadata
118+
- Execution time tracking
119+
- Detailed outcome reporting
120+
121+
5. **Removed Dependencies**:
122+
- No longer depends on orchestrator-specific classes
123+
- Uses attack strategy base classes
124+
- Cleaner separation of concerns
125+
126+
## PAIR Algorithm Preservation
127+
128+
All core PAIR functionality is preserved:
129+
130+
- ✅ Tree-based iterative refinement approach
131+
- ✅ Single-width exploration (width=1)
132+
- ✅ No branching (branching_factor=1)
133+
- ✅ Disabled on-topic checking
134+
- ✅ Custom scoring thresholds
135+
- ✅ Desired response prefix handling
136+
- ✅ PAIR-specific system prompts
137+
- ✅ Depth-based termination
138+
139+
## Testing
140+
141+
The refactored code:
142+
- ✅ Compiles successfully (`python -m py_compile`)
143+
- ✅ Follows the established attack strategy pattern
144+
- ✅ Maintains API compatibility through configuration objects
145+
- ✅ Preserves all PAIR-specific algorithmic behavior
146+
147+
## Usage Examples
148+
149+
See `/pair_attack_demo.py` for comprehensive usage examples and migration guidance.
150+
151+
## Conclusion
152+
153+
The PAIR attack has been successfully refactored from an orchestrator to an attack strategy, providing:
154+
- Better code organization and maintainability
155+
- Consistent interface with other PyRIT attacks
156+
- Enhanced error handling and validation
157+
- Preserved algorithmic functionality
158+
- Future-proof architecture following established patterns
159+
160+
The refactoring maintains backward compatibility through configuration objects while providing a more modern, type-safe, and maintainable implementation.

‎SORA_URL_VALIDATION_SUMMARY.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Sora Target URL Validation Implementation
2+
3+
## Summary
4+
Added comprehensive URL validation for the OpenAI Sora Target, following the same pattern established for other OpenAI targets.
5+
6+
## Implementation Details
7+
8+
### 1. Sora Target Code Changes (`openai_sora_target.py`)
9+
10+
**Added URL validation configuration:**
11+
```python
12+
# Set expected route for URL validation (Sora video generation API)
13+
self._expected_route = "/openai/deployments/*/videos/generations"
14+
15+
# Validate endpoint URL
16+
self._warn_if_irregular_endpoint()
17+
```
18+
19+
**Expected Endpoint Pattern:**
20+
- **Azure Sora API**: `/openai/deployments/*/videos/generations`
21+
- Uses wildcard pattern (`*`) to support any deployment name
22+
- Matches endpoints like: `https://myservice.openai.azure.com/openai/deployments/sora-1/videos/generations`
23+
24+
### 2. Sora Target Test Changes (`test_sora_target.py`)
25+
26+
**Added comprehensive URL validation tests:**
27+
28+
1. **`test_sora_target_url_validation_valid_azure_endpoint_no_warning()`**
29+
- Tests valid Azure Sora endpoint doesn't trigger warnings
30+
- Verifies: `https://myservice.openai.azure.com/openai/deployments/sora-1/videos/generations`
31+
32+
2. **`test_sora_target_url_validation_invalid_endpoint_triggers_warning()`**
33+
- Tests invalid endpoint triggers appropriate warning
34+
- Verifies warning message includes expected route pattern
35+
- Tests: `https://api.openai.com/v1/wrong/path`
36+
37+
3. **`test_sora_target_url_validation_wildcard_pattern_matching()`**
38+
- Tests wildcard pattern matching with various deployment names
39+
- Validates: `sora-1`, `my-custom-sora`, `video-model` deployment names
40+
- Tests both valid and invalid path structures
41+
42+
4. **`test_sora_target_url_validation_case_insensitive()`**
43+
- Verifies URL validation is case insensitive
44+
- Tests uppercase path: `/OPENAI/DEPLOYMENTS/SORA-1/VIDEOS/GENERATIONS`
45+
46+
5. **`test_sora_target_url_validation_trailing_slash_normalization()`**
47+
- Tests trailing slash normalization
48+
- Verifies endpoints with/without trailing slashes work correctly
49+
50+
## Validation Coverage
51+
52+
### ✅ Valid Endpoints (No Warnings)
53+
```
54+
https://myservice.openai.azure.com/openai/deployments/sora-1/videos/generations
55+
https://service.openai.azure.com/openai/deployments/my-custom-sora/videos/generations
56+
https://api.azure.com/openai/deployments/video-model/videos/generations
57+
```
58+
59+
### ❌ Invalid Endpoints (Warning Generated)
60+
```
61+
https://api.openai.com/v1/wrong/path
62+
https://service.azure.com/openai/wrong/sora-1/videos/generations
63+
https://service.azure.com/openai/deployments/sora-1/wrong/generations
64+
```
65+
66+
### 🔍 URL Normalization Features
67+
- **Case Insensitive**: `/OPENAI/DEPLOYMENTS/*/VIDEOS/GENERATIONS` matches `/openai/deployments/*/videos/generations`
68+
- **Trailing Slash**: `/openai/deployments/*/videos/generations/` matches `/openai/deployments/*/videos/generations`
69+
- **Query Parameters**: Ignored during validation (only path is checked)
70+
71+
## Integration with Existing Framework
72+
73+
The Sora target now follows the same URL validation pattern as other OpenAI targets:
74+
75+
1. **Inherits from `OpenAITarget`** - Uses centralized validation logic
76+
2. **Sets `_expected_route`** - Defines valid endpoint pattern
77+
3. **Calls `_warn_if_irregular_endpoint()`** - Triggers validation during initialization
78+
4. **Uses wildcard patterns** - Supports dynamic Azure deployment names
79+
5. **Comprehensive test coverage** - 5 focused test scenarios
80+
81+
## Benefits
82+
83+
1. **Consistency**: Same validation approach across all OpenAI targets
84+
2. **User Guidance**: Clear warnings when endpoints are misconfigured
85+
3. **Flexibility**: Wildcard pattern supports any Azure deployment name
86+
4. **Robustness**: Handles case sensitivity, trailing slashes, and URL normalization
87+
5. **Maintainability**: Easy to extend for new Sora API endpoints if needed
88+
89+
## Test Execution
90+
91+
Run the new Sora target URL validation tests:
92+
```bash
93+
pytest tests/unit/target/test_sora_target.py -k "url_validation" -v
94+
```
95+
96+
Total new tests added: **5 URL validation tests** for comprehensive coverage of Sora target endpoint validation.

‎URL_VALIDATION_TESTS.md‎

Whitespace-only changes.

0 commit comments

Comments
 (0)