Add definition for "crawler"

[garyillyes]

Summary: This PR introduces a formal and precise definition of a "crawler" to establish a foundational understanding for the rest of the document.

Details:
Core Definition: Defines a crawler as an autonomous, non-interactive software acting as an automated HTTP client explicitly set up for bulk resource retrieval.
Behavioral Distinction: Differentiates crawlers from standard interactive clients by highlighting their asynchronous nature and batch-processing approach to link traversal, as opposed to immediate synchronous traversal.
Operational Constraints: Clarifies that crawlers operate without human supervision, relying on algorithmic prioritization and strict compliance with protocol-level instructions like the Robots Exclusion Protocol (REP).

Closes #12

[garyillyes]

@thibmeu does something like this work for y'all? re #12

[thibmeu]

I would change a few details

1. FTP seems out of place given the document is mostly about HTTP
2. this is the only use of interactive/non-interactive in the document, and is complemented with "real-time human supervision". That last framing seems sufficient. There might be limitation with software interacting with software, but that's out of scope for this document if i understand
3. "bulk resource retrieval" is an implementation detail

For instance, could be something along these lines

For the purposes of this document, a crawler is an automated HTTP {{HTTP-SEMANTICS}} client that retrieves resources across one or more web sites without direct human initiation of individual requests. A crawler discovers URIs during retrieval and schedules them for later processing. It relies on algorithmic prioritization and protocol-level instructions such as the Robots Exclusion Protocol {{REP}} to govern its behavior.

another consideration: there is a convention/definition section. It could be moved directly after the intro and the definition would be placed there.

[garyillyes]

I would change a few details

1. FTP seems out of place given the document is mostly about HTTP
2. this is the only use of interactive/non-interactive in the document, and is complemented with "real-time human supervision". That last framing seems sufficient. There might be limitation with software interacting with software, but that's out of scope for this document if i understand
3. "bulk resource retrieval" is an implementation detail

For instance, could be something along these lines

For the purposes of this document, a crawler is an automated HTTP {{HTTP-SEMANTICS}} client that retrieves resources across one or more web sites without direct human initiation of individual requests. A crawler discovers URIs during retrieval and schedules them for later processing. It relies on algorithmic prioritization and protocol-level instructions such as the Robots Exclusion Protocol {{REP}} to govern its behavior.

another consideration: there is a convention/definition section. It could be moved directly after the intro and the definition would be placed there.

Yeah, that does sound better. Stole it and added it to the draft.