A web-data reliability and recovery control plane built around Bright Data.
A scraper returning HTTP 200 and producing records does not necessarily mean the data is still correct.
Scraper-CI separates web acquisition from the intelligence and reliability logic around that acquisition. It profiles sources, infers bounded crawl policies, routes acquisition to the appropriate Bright Data capability, evaluates extracted data, diagnoses degradation, orchestrates recovery, and independently verifies the result.
▶ [Watch the 3-minute Scraper-CI demo on YouTube] (https://youtu.be/wOKMk2Btyd8?si=Z4HRqAVJ61AmIdbN)
▶ [Watch the 7-minute technical deep dive] (https://youtu.be/7tQj2uWBRwM?si=21Kq2U_aKfQhrkys)
- Scraper-CI: https://scraper-ci-ui.onrender.com/
- Flight Intelligence: https://flight-intelligence-rj3l.onrender.com/
- API: https://sci-av37.onrender.com
- Health: https://sci-av37.onrender.com/api/health
- OpenAPI / Swagger: https://sci-av37.onrender.com/docs
The two application links above are the primary entry points for the project. The backend is the shared control-plane API behind the Scraper-CI UI and the downstream consumer. If a Render service is sleeping or temporarily unavailable, a live URL may briefly return an availability error until the service wakes.
The screenshots below are intentionally kept as slots so the README can use the final website captures from the submission.
Scraper-CI is a control plane for reliable web data acquisition.
Bright Data is responsible for actually acquiring data from the web. Scraper-CI is responsible for everything that needs to happen around that acquisition to make the resulting data observable, bounded, testable, recoverable, and useful downstream.
The core lifecycle is:
PUBLIC WEB
│
▼
┌─────────────────────┐
│ SCRAPER-CI │
│ Intelligence Layer │
│ │
│ Profile │
│ Crawl Intelligence │
│ Crawl Policy │
│ Acquisition Decision│
└──────────┬──────────┘
│
policy + extraction
intent
│
▼
╔════════════════════════════╗
║ BRIGHT DATA ║
║ CORE ENGINE ║
║ ║
║ Collector execution ║
║ Web acquisition ║
║ Scraper Studio ║
╚════════════╤═══════════════╝
│
▼
STRUCTURED DATA
│
▼
┌─────────────────────┐
│ SCRAPER-CI │
│ Reliability Layer │
│ │
│ Validate │
│ Diagnose │
│ Heal │
│ Re-run │
│ Verify │
└──────────┬──────────┘
│
▼
SNAPSHOT
│
┌───────────┼───────────┐
▼ ▼ ▼
Consumer 1 Consumer 2 Consumer N
The important architectural boundary is:
```text
Bright Data
= acquisition
Scraper-CI
= intelligence + control + reliability + recovery
Consumer
= domain-specific interpretation
This keeps the acquisition system independent from whatever application eventually consumes the data.
Bright Data is the core acquisition engine of Scraper-CI. Scraper-CI does not replace the underlying web-acquisition infrastructure; it surrounds it with source intelligence, crawl policy, capability selection, reliability evaluation, diagnosis, recovery and verification. This separation allows Scraper-CI to use Bright Data's acquisition capabilities while keeping the control plane and downstream consumers independent from the mechanics of collection.
Traditional scraping systems often treat a successful request as a successful extraction:
HTTP 200
↓
records returned
↓
done
That is not enough.
A website can change its HTML structure, move a field, alter pagination, return partial content, expose a different page type, or silently stop populating a required field while the scraper continues to run.
Scraper-CI treats the actual extracted dataset as the thing that needs to be evaluated.
Instead of asking only:
"Did the scraper run?"
it asks:
"Did the scraper still produce data that satisfies the extraction contract?"
That distinction drives the rest of the system.
A target starts with a URL and an extraction intent.
Scraper-CI persists the target and uses the requested extraction description to initialize the Bright Data collector.
The system keeps its own target identity and also stores the Bright Data collector identity.
This allows the surrounding control plane to maintain run history, schemas, reliability state, recovery attempts, and verification results independently of the collector implementation.
Before treating a site as an arbitrary graph, Scraper-CI can inspect the source and build a source profile.
The profile records signals such as:
- page type
- platform
- JavaScript dependency
- internal links
- external links
- canonical links
- pagination links
- sitemap detection
- repeated-content patterns
- start domain
- relationship information
The goal is not to scrape everything.
The goal is to understand the source well enough to make an informed acquisition decision.
One of the platform's architectural additions is a policy layer between source understanding and acquisition.
The source profile is used to infer a bounded crawl policy.
Conceptually:
Observed source
│
├── internal relationships
├── canonical relationships
├── pagination
├── sitemap signals
├── external domains
└── page relationships
│
▼
Crawl Policy
│
├── what may be followed
├── what should be ignored
├── relationship classes
└── depth / boundary constraints
For example:
Internal links ✓
Canonical links ✓
Pagination ✓
Sitemap ✓
Arbitrary external ✕
The important idea is that crawl scope becomes an explicit policy rather than an accidental consequence of whatever links a scraper happens to encounter.
Bright Data itself provides sitemap loading and sophisticated Scraper Studio navigation primitives. Scraper-CI does not claim to have invented sitemap support. The distinction is that Scraper-CI makes source profiling and policy-driven crawl scoping a first-class control-plane concern.
Bright Data documents load_sitemap, pagination, multi-stage execution, browser interaction, and other scraping primitives in Scraper Studio. Scraper-CI sits above those primitives and decides how the acquisition should be bounded and routed.
This is an important implementation detail. The inferred crawl policy is not only stored as metadata in Scraper-CI. Its constraints are compiled into the natural-language acquisition description sent to Bright Data when the collector is created.
Conceptually:
User extraction intent
+
Source-derived crawl intelligence
↓
Bright Data acquisition prompt
For example, Scraper-CI can attach constraints such as:
Crawl scope: depth<=2, pages<=50
internal=1, canonical=1, pagination=1, sitemap=1, external=0
allowed_external=none
Do not follow unrelated external sites.
This matters because the policy influences the collector at creation time, rather than existing only as an after-the-fact dashboard label. Scraper-CI also persists the policy and independently validates the resulting acquisition against the intended contract.
The design therefore has two layers of protection:
- Prompt-level guidance tells the acquisition system what scope it should follow.
- Independent validation checks the resulting data and reliability state after acquisition.
This is one of the key intelligence-to-execution paths in Scraper-CI.
The Acquisition Decision is the router's recommendation for which registered acquisition capability should collect a source.
In short:
Profile the source → evaluate its signals → choose the acquisition path most likely to collect it reliably.
The decision records the selected capability, supporting evidence, and a routing-confidence score. A higher confidence means the router found stronger evidence for that method; it is not a guarantee that extraction will succeed.
For example:
Source signals
↓
Capability Router
↓
Scraper Studio — 78% confidence
↓
Acquisition
Why it matters: the system should not blindly use one acquisition mechanism for every source. Choosing an appropriate capability can improve extraction reliability and avoid unnecessary collection work.
Different sources do not require identical acquisition strategies.
Scraper-CI therefore has a capability abstraction:
SOURCE INTELLIGENCE
│
▼
CAPABILITY ROUTER
│
┌──────────────┼──────────────┐
▼ ▼ ▼
Pipeline Unlocker Scraper
Studio
│ │ │
└──────────────┼──────────────┘
▼
Result
The current registered Bright Data capability contracts are:
pipelineunlockerbrowserscraper_studiosearch
The router evaluates the source intelligence, selects a primary strategy, keeps fallback strategies available, and records the routing decision with the acquisition result.
Some capability contracts are intentionally registered ahead of a concrete executor. For example, Browser and Search are represented as capabilities in the routing model without pretending that an unsupported live executor exists.
That distinction matters: the architecture is extensible without faking execution that is not configured.
The registry is designed so new acquisition methods can be added behind the same capability contract without rewriting the rest of the control plane.
Current registered capabilities are:
| Capability | Current state | Role |
|---|---|---|
pipeline |
Live | Trigger a configured Bright Data Web Scraper API dataset |
unlocker |
Live | Use Bright Data Web Unlocker for acquisition paths that need it |
scraper_studio |
Live | Create/run Bright Data Scraper Studio collectors |
browser |
Contract | Reserved for a concrete Browser/CDP executor |
search |
Contract | Reserved for search/discovery-oriented acquisition |
Future work can add concrete adapters for additional Bright Data or other acquisition mechanisms without changing the higher-level flow:
Profile → Policy → Route → Acquire → Validate → Recover → Verify
That is the point of the capability abstraction: acquisition can expand while profiling, policy, reliability, diagnosis, recovery, and consumers remain decoupled from any single provider or execution mechanism.
Bright Data is the acquisition foundation.
Scraper-CI integrates with Bright Data for:
- Scraper Studio collector creation
- collector execution
- structured extraction
- Web Scraper API / pipeline execution
- Web Unlocker requests
- Scraper Studio self-healing
- collector lifecycle and repair approval
Bright Data's current Scraper Studio documentation describes interaction and parser code, stages, pagination, sitemap loading, structured collection, Browser Worker / Code Worker selection, and an AI-powered Self-Healing tool.
Bright Data also explicitly supports self-healing changes such as fixing broken fields and adding or removing output fields.
Scraper-CI uses those capabilities as part of a larger reliability lifecycle rather than treating Bright Data as the entire application.
After acquisition, Scraper-CI evaluates the resulting records.
The reliability layer currently includes:
- schema validation
- required-field validation
- field completeness
- uniqueness checks
- record-level comparison
- record-count drift
- historical reliability
- health scoring
- degradation classification
- recovery metrics
- regression checks
A simplified model is:
Acquired records
│
├── Schema validation
│
├── Completeness
│
├── Uniqueness
│
├── Record comparison
│
├── Drift
│
└── Health scoring
│
▼
HEALTHY / DEGRADED / INVALID
This means a technically successful acquisition can still be marked degraded when the data contract has deteriorated.
Diagnosis explains why a run is unhealthy.
The system can distinguish scenarios such as:
- missing required fields
- invalid schema values
- completeness degradation
- extraction changes
- blocked acquisition
- JavaScript-heavy source requirements
- other deterministic failure categories
Diagnosis is persisted with the run and can feed the recovery planner.
The purpose is to avoid blindly calling the same recovery mechanism for every failure.
Recovery is driven by the diagnosed failure.
The recovery planner can route different failure classes toward different strategies.
Examples include:
Missing required field
↓
Scraper Studio self-healing
Schema validation failure
↓
Scraper Studio self-healing
Blocked acquisition
↓
Unlocker strategy
High JavaScript dependency
↓
Browser strategy
Unknown failure
↓
Scraper Studio fallback
The recovery planner is deterministic and tested independently.
When an automated collection finishes with a degraded or invalid health state, Scraper-CI automatically enters the recovery path. The current automation contract allows up to two automatic healing attempts for the incident. Each attempt is followed by a re-run and independent verification.
Run
↓
DEGRADED / INVALID
↓
Healing attempt 1
↓
Re-run + verify
↓
Still degraded?
↓
Healing attempt 2
↓
Re-run + verify
If recovery succeeds, the incident is marked recovered. If the automatic attempts do not restore the extraction contract, recovery stops and the operator can retry recovery manually.
There is also an explicit manual correction path. This is useful when the scraper is technically healthy but the intended data contract has changed. The operator can provide a correction describing the desired change, and Scraper-CI sends a constrained repair prompt to the existing collector.
Manual correction therefore covers a different case from automatic healing:
Automatic healing
= something degraded; repair the existing contract
Manual correction
= operator intentionally wants the existing contract changed
Scraper-CI can trigger Bright Data's Self-Healing flow for an existing collector.
The important lifecycle is:
Existing Collector
│
▼
Latest extraction
│
▼
Validation / Diagnosis
│
▼
Recovery Plan
│
▼
Bright Data Self-Healing
│
▼
Repair state
│
▼
Approval when required
│
▼
Re-run
│
▼
Independent verification
The collector identity remains part of the persistent Scraper-CI record.
For automatic repair, Scraper-CI builds a diagnosis-derived repair prompt containing the target, observed degradation reasons, current field completeness, and explicit preservation rules. The repair instructions tell the healing system to preserve existing fields and meanings, avoid intentionally expanding crawl scope, avoid following individual item links unless already part of the intent, and make the smallest extraction change necessary to restore the contract.
This is another important boundary: the system does not send a generic "fix it" prompt. It sends the recovery intelligence produced by the reliability layer.
Scraper-CI also distinguishes between:
Used when a run is degraded or invalid and the system has enough information to attempt recovery.
Used when the scraper can be healthy from a metrics perspective but the operator knows the extraction requirement itself needs to change.
For example:
"Add the missing author field and preserve the existing extraction."
This is useful for an intentional evolution of the extraction contract, not merely a failure.
Healing is not considered successful merely because Bright Data reports that a repair completed.
Scraper-CI re-runs the collector and independently evaluates the resulting data.
Verification checks include:
- schema pass
- completeness pass
- recovery pass
- regression pass
- before/after health score
- recovery delta
- record-level differences
- diagnosis state
The result is an explicit:
REPAIR VERIFIED
or:
REPAIR REJECTED
This closes the loop:
Observe
↓
Acquire
↓
Validate
↓
Diagnose
↓
Heal
↓
Re-run
↓
Verify
Live UI: https://scraper-ci-ui.onrender.com/
The web application is an operator control plane over the same backend orchestration used by the CLI.
The scraper control surface exposes actions such as:
Automated
Inspect
Profile
Status
Reliability
Diagnose
Correct data
Verify
Open URL
The actions are not separate implementations of the system. The frontend calls the API, and the API invokes the same domain/orchestration functions used by the CLI.
The Bright Data credential stays server-side.
Inspect answers:
"Given this source, what acquisition capability should be preferred?"
It shows source intelligence and the routing decision, including:
- platform
- page type
- JavaScript dependency
- structured source
- recommended capability
- confidence
- reason
- fallback capabilities
This is the capability-selection view.
Profile answers:
"What does this source expose, and what crawl boundaries can be inferred?"
It shows the source profile and inferred crawl policy.
This is where the source intelligence layer becomes visible.
Status shows the latest operational state of a target:
- latest run
- run status
- record count
- health score
- health status
- latest healing attempt
- collector state
Reliability shows historical reliability rather than only the latest run.
It exposes persisted reliability snapshots and summary metrics.
This is useful when the question is:
"Has this scraper been stable?"
rather than:
"Did the last request work?"
Diagnose analyzes the latest or selected historical run and compares it against the expected extraction state and previous data.
It provides:
- classification
- health score
- degradation state
- recommendation
- record differences
- previous-run context
When a run is degraded or invalid, the operator can start the recovery workflow.
Scraper-CI creates a recovery plan and triggers the appropriate Bright Data repair flow.
A healthy scraper can still be wrong.
The Correct data action is for that case.
An operator can explicitly describe the desired extraction change without first requiring the reliability engine to label the run degraded.
This separates:
"the scraper broke"
from:
"the scraper works, but the data contract has changed."
Verification is intentionally gated.
The UI only exposes the verification action when the recovery lifecycle has reached a state where verification makes sense.
Verification re-runs the extraction and evaluates the before/after result independently.
The dashboard also exposes an automation control surface.
The current backend automation manager supports a collection/recovery cycle and persisted operational state. It does not pretend to be a long-term scheduler.
The Flight Intelligence production collection is scheduled separately through GitHub Actions every two hours.
This separation keeps the demo/control-plane automation from being confused with the production collection scheduler.
Live consumer: https://flight-intelligence-rj3l.onrender.com/
The repository contains one concrete downstream consumer:
Bright Data
↓
Scraper-CI acquisition
↓
Validation
↓
Trusted extraction
↓
Flight Activity Intelligence
The consumer reads structured aircraft records through the Scraper-CI data API.
It does not need to know:
- how Bright Data acquired the data
- which acquisition capability was selected
- how the collector was created
- how diagnosis works
- how healing works
- how crawl policy was inferred
It receives structured records plus the relevant snapshot/schema/intelligence state.
Flight Intelligence turns aircraft activity into domain-specific observations.
The current consumer derives:
- aircraft count
- airborne count
- on-ground count
- aircraft-type count
- average altitude
- average speed
- maximum altitude
- maximum speed
- fastest observation
- highest observation
The consumer tracks altitude bands:
40,000+ ft
30,000–39,999 ft
20,000–29,999 ft
10,000–19,999 ft
0–9,999 ft
It also exposes a configured high-altitude threshold and the number of aircraft above that threshold.
The consumer can track semantic identity fields such as:
- hex ID
- registration
- callsign
It uses these to derive:
- new aircraft
- disappeared aircraft
- retained aircraft
- cumulative distinct aircraft
- untracked records
The system persists snapshot history and exposes historical points containing:
- run ID
- capture time
- record count
- aircraft type count
- new aircraft count
- cumulative distinct aircraft
This is an important architectural point.
Flight Intelligence is not Scraper-CI itself.
It is one example of what can be built downstream.
The architecture is:
SCRAPER-CI
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Flight Intelligence Consumer 2 Consumer N
│ │ │
▼ ▼ ▼
Aircraft Another app Another app
analytics
The acquisition and reliability layers do not contain aircraft-specific assumptions.
Flight Intelligence is simply the first concrete consumer used to prove that the structured data can power a real application.
The same Scraper-CI output could feed, for example:
- monitoring dashboards
- alerting systems
- research workflows
- analytics applications
- RAG pipelines
- agent workflows
- operational data products
- domain-specific intelligence systems
The number of consumers is not structurally limited to one.
The deployed architecture is intentionally split into independent services.
PUBLIC WEB
│
▼
Bright Data
│
▼
┌─────────────────────┐
│ Scraper-CI API │
│ Render │
└─────────┬───────────┘
│
▼
┌─────────────────────┐
│ PostgreSQL / Aiven │
└─────────────────────┘
│
┌─────────┴─────────┐
▼ ▼
Scraper-CI Frontend Flight Intelligence
Render Render
│ │
└─────────┬─────────┘
│
▼
API / Data
The FastAPI backend is deployed as a Render Web Service.
It exposes:
- REST API
- OpenAPI documentation
- health check
- orchestration
- persistence
- Bright Data integration
- reliability evaluation
- recovery workflows
Render web services expose a public onrender.com URL and support environment variables/secrets for runtime configuration.
The Scraper-CI operator frontend is deployed separately on Render.
It communicates with the backend through the configured VITE_API_BASE_URL.
Flight Intelligence is a separate frontend consumer.
It reads the latest structured data and intelligence through:
GET /api/targets/{target_id}/data/latest
The deployed application currently uses PostgreSQL hosted by Aiven.
PostgreSQL stores the persistent control-plane state, including targets, collectors, schemas, runs, records, validation results, drift, healing attempts, verification results, reliability history, automation state, and Flight Intelligence history.
Local Docker Compose uses a separate PostgreSQL container so the local stack does not depend on the production database.
The Flight Intelligence consumer receives a fresh snapshot every 15 days. Production collection is triggered automatically by a GitHub Actions scheduled workflow using:
0 0 1,16 * *
The collection cadence was intentionally reduced from every 2 hours to approximately every 15 days to support the long-term sustainability, longevity, and continuity of the project while still maintaining periodic fresh data. This keeps the production pipeline active without introducing unnecessary collection frequency or operational overhead.
The scheduled workflow is intentionally automated, with manual execution disabled to preserve a predictable and consistent collection cycle.
GitHub Actions launches the collector, which runs scrape-ci run against the configured Flight Intelligence target. The resulting snapshot is persisted through the Scraper-CI backend/database and becomes available to the consumer.
The workflow also supports a manual workflow_dispatch trigger.
The scheduled workflow:
- checks out the repository
- installs
uv - installs the locked Python dependencies
- validates required secrets/configuration
- runs
scrape-ci run "$FLIGHT_INTELLIGENCE_TARGET_URL" --output flight-run.json - uploads the run result as a GitHub Actions artifact for short-term inspection
This means the Flight Intelligence dashboard is not a static demo dataset. It is a downstream consumer of a recurring acquisition pipeline.
Required scheduler configuration includes:
BRIGHTDATA_API_KEYDATABASE_URLFLIGHT_INTELLIGENCE_TARGET_URL
The backend is a FastAPI application.
Health
GET https://sci-av37.onrender.com/api/health
Expected response:
{
"status": "ok",
"host": "..."
}OpenAPI / Swagger
https://sci-av37.onrender.com/docs
| Method | Endpoint | Purpose |
|---|---|---|
| GET | /api/targets |
List registered targets with latest state |
| GET | /api/targets/resolve?url=... |
Resolve a target by URL |
| POST | /api/targets |
Create/open a target and optionally persist a schema |
| GET | /api/targets/{target_id} |
Get a target and its operational state |
| GET | /api/targets/{target_id}/data/latest |
Return latest downstream-consumable data |
| GET | /api/targets/{target_id}/runs |
List runs for a target |
| GET | /api/runs/{run_id} |
Get run, records, validation, drift, healing and verification |
| GET | /api/runs/{run_id}/records |
Return records for a run with a configurable limit |
| Method | Endpoint | Purpose |
|---|---|---|
| POST | /api/actions/init |
Initialize a scraper from URL + extraction description |
| POST | /api/actions/run |
Run an existing collector and evaluate its output |
| GET | /api/actions/inspect |
Inspect source intelligence and acquisition routing |
| GET | /api/actions/profile |
Profile a source and infer crawl policy |
| GET | /api/actions/status |
Get latest scraper operational state |
| GET | /api/actions/history |
Get recent run history |
| GET | /api/actions/reliability |
Get reliability summary/history |
| POST | /api/actions/diagnose |
Diagnose a latest or selected run |
| POST | /api/actions/heal |
Trigger recovery/self-healing or manual correction |
| POST | /api/actions/approve/repair |
Approve a Bright Data repair awaiting approval |
| POST | /api/actions/verify |
Re-run and independently verify recovery |
| POST | /api/actions/benchmark |
Run the deterministic reliability benchmark |
| POST | /api/actions/config-check |
Check Bright Data credential configuration |
| POST | /api/actions/init-db |
Initialize database state |
| POST | /api/actions/demo |
Run the deterministic local demo |
| GET | /api/actions/capabilities |
List registered acquisition capabilities |
| POST | /api/actions/replay |
Replay fixture data through the diagnosis pipeline |
| Method | Endpoint | Purpose |
|---|---|---|
| POST | /api/actions/automation/start |
Start one collection/recovery cycle |
| POST | /api/actions/automation/stop |
Stop/mark an automation cycle stopped |
| POST | /api/actions/automation/resume |
Retry recovery for the latest degraded run |
| GET | /api/actions/automation |
Read automation state |
The Python CLI is installed as:
scrape-ci
Inside Docker:
docker compose exec backend uv run scrape-ci --helpdocker compose exec backend uv run scrape-ci init-dbdocker compose exec backend uv run scrape-ci init "https://example.com" --description "Describe what should be extracted."Optional category:
docker compose exec backend uv run scrape-ci init "https://example.com" --description "Extract article title and author." --category newsdocker compose exec backend uv run scrape-ci profile "https://example.com"This prints the source profile and inferred crawl policy.
docker compose exec backend uv run scrape-ci inspect "https://example.com"Machine-readable:
docker compose exec backend uv run scrape-ci inspect "https://example.com" --jsondocker compose exec backend uv run scrape-ci run "https://example.com"Write the complete run result to JSON:
docker compose exec backend uv run scrape-ci run "https://example.com" --output run.jsondocker compose exec backend uv run scrape-ci historydocker compose exec backend uv run scrape-ci reliability "https://example.com"Custom limit:
docker compose exec backend uv run scrape-ci reliability "https://example.com" --limit 50docker compose exec backend uv run scrape-ci status "https://example.com"Latest run:
docker compose exec backend uv run scrape-ci diagnose "https://example.com"Specific run:
docker compose exec backend uv run scrape-ci diagnose "https://example.com" --run-id 123Automatic recovery:
docker compose exec backend uv run scrape-ci heal "https://example.com"Specific run:
docker compose exec backend uv run scrape-ci heal "https://example.com" --run-id 123Manual correction:
docker compose exec backend uv run scrape-ci heal "https://example.com" --manual --correction "Add the missing author field and preserve the existing extraction."Or simply:
docker compose exec backend uv run scrape-ci heal "https://example.com" --correction "Add the missing author field and preserve the existing extraction."docker compose exec backend uv run scrape-ci approve "https://example.com"docker compose exec backend uv run scrape-ci verify "https://example.com"docker compose exec backend uv run scrape-ci benchmarkdocker compose exec backend uv run scrape-ci config-checkscrape-ci init-db
scrape-ci init
scrape-ci history
scrape-ci reliability
scrape-ci status
scrape-ci approve
scrape-ci demo
scrape-ci profile
scrape-ci inspect
scrape-ci run
scrape-ci replay
scrape-ci diagnose
scrape-ci heal
scrape-ci verify
scrape-ci benchmark
scrape-ci config-check
The repository contains deterministic tests around the control plane.
Covered areas include:
- acquisition runtime behavior
- API data contracts
- benchmark scenarios
- health scoring
- completeness degradation
- schema validation
- capability routing
- crawl policy inference
- relationship classification
- diagnosis
- drift
- recovery guards
- recovery planning
- reliability history
- repair verification
The benchmark is designed to exercise controlled scenarios rather than depending exclusively on a live website.
This matters because the web is inherently nondeterministic.
A live site can change between test runs. Deterministic fixtures allow the same reliability and recovery logic to be replayed and evaluated repeatedly.
scraper-ci/
│
├── src/scraper_ci/
│ ├── acquisition/
│ │ ├── capability.py
│ │ ├── models.py
│ │ ├── router.py
│ │ └── strategies.py
│ │
│ ├── benchmark/
│ │ ├── metrics.py
│ │ ├── runner.py
│ │ └── scenarios.py
│ │
│ ├── brightdata/
│ │ ├── adapters.py
│ │ ├── client.py
│ │ ├── collector.py
│ │ └── healer.py
│ │
│ ├── intelligence/
│ │ ├── flight_activity.py
│ │ └── inspector.py
│ │
│ ├── orchestration/
│ │ ├── automation.py
│ │ └── flows.py
│ │
│ ├── profiling/
│ │ ├── policy.py
│ │ ├── profiler.py
│ │ └── relationships.py
│ │
│ ├── recovery/
│ │ └── planner.py
│ │
│ ├── reliability/
│ │ └── history.py
│ │
│ ├── storage/
│ │ ├── database.py
│ │ ├── models.py
│ │ └── repository.py
│ │
│ ├── validation/
│ │ ├── completeness.py
│ │ ├── crawl_policy.py
│ │ ├── diagnosis.py
│ │ ├── drift.py
│ │ ├── record_diff.py
│ │ ├── schema.py
│ │ └── scoring.py
│ │
│ ├── api.py
│ └── cli.py
│
├── frontend/
│ └── React operator control plane
│
├── flight-intelligence/
│ └── React/TypeScript downstream consumer
│
├── schemas/
│ ├── ecommerce.yaml
│ ├── hashnode.yaml
│ ├── jobs.yaml
│ └── news.yaml
│
├── benchmarks/
│ └── sites.yaml
│
├── tests/
│
├── docker-compose.yml
├── Dockerfile
├── pyproject.toml
└── .github/
└── workflows/
└── flight-intelligence.yml
The current deployed topology is:
Frontend
│
│ Render
▼
Scraper-CI Web UI
│
│ HTTPS
▼
Backend
│
│ Render
├──────────────► Bright Data
│
▼
PostgreSQL
│
│ Aiven
▼
Persistent control-plane state
The Flight Intelligence collector runs separately through GitHub Actions on a two-hour schedule and writes its results into the same PostgreSQL-backed control plane.
The deployed environment currently maintains two collectors. One locked collector is the production acquisition path used by the Flight Intelligence pipeline. A second collector is retained independently for controlled demonstrations, testing and recovery work, so the production consumer path does not need to be disturbed during experimentation.
Secrets and environment configuration are kept outside the source code.
Important runtime configuration includes:
BRIGHTDATA_API_KEY
DATABASE_URL
CORS_ORIGINS
BRIGHT_DATA_BASE_URL
BRIGHT_DATA_TIMEOUT_SECONDS
BRIGHT_DATA_POLL_INTERVAL_SECONDS
BRIGHT_DATA_POLL_TIMEOUT_SECONDS
Frontend deployments use:
VITE_API_BASE_URL
The Bright Data credential is never required by the browser.
Render supports environment variables/secrets for runtime configuration, and the deployed backend uses that pattern.
This project was developed with assistance from ChatGPT and OpenAI Codex.
Their use included coding assistance, iteration, debugging support, and implementation assistance.
The following were not blindly delegated:
- raw system architecture
- control-plane design
- source profiling model
- crawl-policy design
- acquisition/capability abstraction
- reliability model
- recovery logic
- verification logic
- downstream consumer design
- UI information architecture
- UI/interaction design
- infrastructure topology
- deployment logic
These parts were designed, reviewed, verified, and integrated by the project author.
AI-assisted code was treated as implementation assistance rather than as an authority over the system design.
Do not commit:
.env
or any file containing:
- Bright Data API keys
- PostgreSQL credentials
- deployment secrets
- GitHub Actions secrets
Use .env.example as the local configuration template.
For production, inject credentials through the deployment platform's secret/environment configuration.
If an actual credential has ever been committed or shared accidentally, rotate it.
Scraper-CI is built around one idea:
The web changes, so a reliable data system has to expect change.
It therefore treats acquisition as only one part of the problem.
PROFILE
↓
POLICY
↓
ROUTE
↓
ACQUIRE
↓
VALIDATE
↓
DIAGNOSE
↓
HEAL
↓
VERIFY
↓
CONSUME
Bright Data provides the acquisition infrastructure.
Scraper-CI provides the surrounding control plane.
Flight Intelligence demonstrates one concrete downstream use case.
And because the consumer is separated from acquisition and reliability, Flight Intelligence is only one example — Scraper-CI can feed an arbitrary number of downstream consumers without rebuilding the acquisition and recovery infrastructure for each one.
The easiest way to run the complete stack locally is Docker Compose.
Install:
- Docker Desktop
- Git
The project uses:
- Python 3.11+
- FastAPI
- PostgreSQL
- React/Vite
- TypeScript for Flight Intelligence
- Bright Data APIs
You do not need to install Python or Node manually when using the provided Docker Compose stack.
git clone https://github.com/Jaival-Suthar/sci.git
cd sciCreate a local .env file from .env.example.
PowerShell:
Copy-Item .env.example .envSet:
BRIGHTDATA_API_KEY=YOUR_BRIGHT_DATA_API_KEY
The Docker Compose backend reads this value and passes it into the backend container.
Do not expose the key to either frontend.
docker compose up --buildThe stack contains:
postgres
backend
frontend
flight-intelligence
Docker Compose waits for PostgreSQL and the backend health check before starting the frontend services.
| Service | URL |
|---|---|
| Scraper-CI dashboard | http://localhost:5173 |
| Flight Intelligence | http://localhost:5174 |
| Backend API | http://localhost:8000 |
| Swagger / OpenAPI | http://localhost:8000/docs |
| Health | http://localhost:8000/api/health |
docker compose up -dCheck service status:
docker compose psBackend:
docker compose logs -f backendAll services:
docker compose logs -fOpen:
http://localhost:8000/api/health
Expected response:
{
"status": "ok",
"host": "..."
}Open the interactive API documentation:
http://localhost:8000/docs
Open:
http://localhost:5173
Typical flow:
Create scraper
↓
Inspect
↓
Profile
↓
Run
↓
Reliability
↓
Diagnose
↓
Heal / Correct data
↓
Approve if required
↓
Verify
Open:
http://localhost:5174
The consumer reads the configured target from:
VITE_TARGET_ID
and retrieves the latest structured data through the backend data endpoint.
Show all commands:
docker compose exec backend uv run scrape-ci --helpFor example:
docker compose exec backend uv run scrape-ci profile "https://example.com"or:
docker compose exec backend uv run scrape-ci benchmarkdocker compose downThe local PostgreSQL data is stored in the named Docker volume:
scraper_ci_postgres
To stop the containers while preserving the local database:
docker compose downDo not use:
docker compose down -vunless you intentionally want to remove the local PostgreSQL volume and its stored data.