other_web_claw

ERP commerce ingest + anti-bot research lab
Etsy · Michaels · DHL · Ruishu (瑞数) pure-algo · Akamai skills

Quick Start · Scope · Architecture · Research · 中文

---

Why split from many_web_claw?

Concern	This repo	Sister repo
Goal	ERP rows + WAF research	Electric 9-site analysis DB
Tables	ss_crawler_*	crawler_*
Sites	Etsy, Michaels, DHL, SCU	Spirit…TSC
Skills	Akamai / Ruishu / env bridge	Fetch + OSS + QA reports

Sister: many_web_claw — keep them separate so production ingest stays boring and reverse-engineering stays sharp.

---

What lives here

Module	Role
sites/etsy	Oxylabs etsy / etsy_product → ERP
sites/michaels	Oxylabs universal → ERP
sites/dhl	Shipment tracker (Akamai path)
sites/scu	Ruishu (瑞数) case study + FLOW
skills/akamai	framework jsdom + v3 pure engine
skills/ruishu	Cookie pure-algo toolkit
skills/env_bridge	Lightweight BOM/DOM sandbox
run_all.py	Batch + --ingest-mapped

---

Architecture

flowchart LR
  subgraph ERP["ERP track"]
    E[Etsy / Michaels scrapers]
    R[run_all --ingest-mapped]
    I[core.ingest]
    T[(ss_crawler_product<br/>image / history)]
  end
  subgraph Lab["Research track"]
    A[skills/akamai]
    U[skills/ruishu]
    S[sites/scu FLOW]
  end
  E --> R --> I --> T
  A -.-> DHL[sites/dhl]
  U --> S

Loading

sequenceDiagram
  participant Dev
  participant Runner as run_all.py
  participant Oxy as Oxylabs
  participant DB as PostgreSQL ERP
  Dev->>Runner: prod --ingest-mapped etsy
  Runner->>Oxy: list + detail
  Runner->>DB: upsert by customer+sku+year
  Runner-->>Dev: inserted / updated / historyOnly

Loading

---

Quick Start

1. Clone & install

git clone https://github.com/lyt26/other_web_claw.git
cd other_web_claw
python -m venv .venv
# Windows: .\.venv\Scripts\activate
source .venv/bin/activate
pip install -r requirements.txt
pip install -r requirements-oss.txt
cp .env.example .env
# Same PG / OXYLABS / OSS vars as many_web_claw are fine

Node skills (when you touch Akamai / Ruishu):

cd skills/akamai/v3 && npm install && cd -
cd skills/akamai/framework && npm install && cd -
cd skills/env_bridge && npm install && cd -

2. List sites

python run_all.py --list

3. Safe preview (crawl, no DB write)

python run_all.py test --dry-run michaels
python run_all.py test --dry-run etsy

4. Production ERP ingest

python run_all.py prod --ingest-mapped michaels
python run_all.py prod --ingest-mapped etsy

5. DHL / Ruishu

python sites/dhl/tracker.py <tracking_number>
# Ruishu walkthrough:
# sites/scu/FLOW.md  +  skills/ruishu/SKILL.md

---

Research tracks

Akamai

Sensor / _abck pipelines under skills/akamai/ — see docs/AKAMAI.md.

Ruishu (瑞数)

Detect 202/412 → decode keys → Coder → basearr → Huffman→XOR→AES cookie — without a browser when constants are known. Start at sites/scu/FLOW.md.

Research code is for your own systems, authorized testing, and learning. Don’t use it to attack third-party production without permission.

---

Docs

Doc	Topic
AGENTS.md	Agent / human map
docs/ETSY_PITFALLS.md	Currency & field traps
docs/ERP_SS_CRAWLER_SCHEMA.md	ERP tables
docs/AKAMAI.md	Akamai framework
docs/ARCHITECTURE.md	Data flow

---

Hard rules

1. Never write Etsy/Michaels into crawler_* (electric analysis DB).
2. Electric nine sites → many_web_claw.
3. Never commit .env or cookie profiles.

---

中文

other_web_claw = ERP 站（Etsy / Michaels）入库 + DHL + 瑞数 / Akamai 研究代码。
电动 9 站分析库请用 many_web_claw。

python run_all.py test --dry-run michaels
python run_all.py prod --ingest-mapped etsy

---

Star if you work on this stack

If you ship WAF research, ERP product sync, or Oxylabs-backed marketplaces — starring both repos helps others discover a clean split: ops crawler vs lab + ERP.

Keep production boring. Keep reverse-engineering sharp.