Description
This proposal suggests extending the Prefix Object with an optional field to indicate agent interaction type, enabling more reliable identification and handling of automated HTTP clients.
In production environments, IP ranges alone are often insufficient to determine how an automated client interacts with content. The same infrastructure may be used across multiple interaction modes, each carrying different implications for access control, rate limiting, caching, and monitoring.
Common Interaction Modes
| Mode |
Description |
| Large-scale crawling |
For indexing |
| Training data collection |
Dataset construction |
| Real-time retrieval |
For AI-generated responses |
| User-triggered fetches |
On-demand access |
Proposed Extension
Introduce a lightweight, optional purpose field in the Prefix Object to indicate the primary interaction type.
Example
{
"ipv4Prefix": "66.249.64.0/20",
"service": "ExampleBot",
"purpose": "on-demand"
}Potential values (non-normative): discovery, indexing, on-demand, support
Rationale
Policy Alignment: Enables infrastructure-level traffic handling through stable, non-sensitive classifications, without exposing internal system behavior or requiring deep packet inspection.
Operational Stability: Provides a consistent categorization layer that remains valid even as underlying systems and use cases evolve.
Reduced Friction: Helps site operators distinguish between different classes of automated traffic more reliably, reducing overblocking and improving successful interactions.
Ecosystem Efficiency: Minimizes reliance on platform-specific documentation and manual rule maintenance, enabling more automated and standardized handling of trusted clients.
Backward Compatibility: Introduces no breaking changes; the field is optional and fully compatible with existing implementations (§ 3.2).
Context
As automated web access becomes more diverse, site operators need simple, stable signals to distinguish between different classes of automated traffic. A minimal classification layer at the prefix level enables more predictable and scalable handling without requiring deep inspection or platform-specific heuristics, improving interoperability and reducing operational overhead on both sides.
Description
This proposal suggests extending the Prefix Object with an optional field to indicate agent interaction type, enabling more reliable identification and handling of automated HTTP clients.
In production environments, IP ranges alone are often insufficient to determine how an automated client interacts with content. The same infrastructure may be used across multiple interaction modes, each carrying different implications for access control, rate limiting, caching, and monitoring.
Common Interaction Modes
Proposed Extension
Introduce a lightweight, optional
purposefield in the Prefix Object to indicate the primary interaction type.Example
{ "ipv4Prefix": "66.249.64.0/20", "service": "ExampleBot", "purpose": "on-demand" }Potential values (non-normative):
discovery,indexing,on-demand,supportRationale
Policy Alignment: Enables infrastructure-level traffic handling through stable, non-sensitive classifications, without exposing internal system behavior or requiring deep packet inspection.
Operational Stability: Provides a consistent categorization layer that remains valid even as underlying systems and use cases evolve.
Reduced Friction: Helps site operators distinguish between different classes of automated traffic more reliably, reducing overblocking and improving successful interactions.
Ecosystem Efficiency: Minimizes reliance on platform-specific documentation and manual rule maintenance, enabling more automated and standardized handling of trusted clients.
Backward Compatibility: Introduces no breaking changes; the field is optional and fully compatible with existing implementations (§ 3.2).
Context
As automated web access becomes more diverse, site operators need simple, stable signals to distinguish between different classes of automated traffic. A minimal classification layer at the prefix level enables more predictable and scalable handling without requiring deep inspection or platform-specific heuristics, improving interoperability and reducing operational overhead on both sides.