Repository navigation
Expand file tree
/
Copy path.env.example
More file actions
141 lines (135 loc) · 7.37 KB
/
Copy path.env.example
File metadata and controls
141 lines (135 loc) · 7.37 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
# bbb-scraper — credentials and defaults
#
# Copy this file to `.env` next to the scripts and fill in what you use:
#
# cp .env.example .env
#
# `.env` is in .gitignore. Keep it there. A secret passed on a command line
# is readable by anything that can run `ps` and lands in your shell history,
# which is why none of these has a flag you are expected to type.
#
# Precedence, highest first:
#
# explicit CLI flag > exported environment variable > .env > default
#
# So a `.env` never overrides something you typed, and an already-exported
# variable (a CI secret, direnv, your shell profile) is never clobbered by a
# file you forgot to delete.
#
# Check what was picked up, without printing any secret:
#
# python3 env_config.py
#
# Every variable below is one this code actually reads, and every variable
# the code reads is below — smoke_test.py asserts the two sets are equal in
# both directions, because a documented-but-unread variable is worse than an
# undocumented one.
#
# READ THIS FIRST: on BBB you may not need any of them.
#
# `--mode search` and `--mode category` read BBB's own JSON endpoint, which
# is NOT behind Cloudflare. Measured 2026-09-16 from a datacenter VPS that
# gets HTTP 403 on every HTML page on the site: HTTP 200, 57 KB, 15 complete
# business records. A two-page run from that machine with this file absent
# returned 30 rows and `status: complete`. No key, no proxy, no account.
#
# What the credentials below buy is `--mode profile`, which is the one page
# kind BBB gates — and there is no endpoint for it, so the rendered page is
# the only route to accreditation dates, complaint counts and the rest.
# --- 2Captcha -------------------------------------------------------------
# One key, four separately-billed products: captcha solving, the Scraping
# Browser API, proxies and fingerprints. https://2captcha.com
#
# WORTH KNOWING BEFORE YOU BUY SOLVING FOR THIS SITE. BBB refuses in two
# shapes and only one of them is solvable:
#
# "Just a moment..." with a Turnstile widget a real challenge. This is
# what a solver is for.
# "You have been blocked | Better Business a refusal. No widget, no
# Bureau(R)" sitekey, nothing to solve —
# a solve here buys nothing.
#
# Both answer HTTP 403 and both wear BBB's own branding in the title, which
# is why the scrapers tell them apart structurally rather than by the title.
# From a datacenter address the second is what arrives, and what clears it is
# the EXIT, not a solver. Solving is wired up and capped at one purchase per
# page; it is not the happy path here.
#
# This key IS what --fingerprint uses (a separate subscription from solving:
# a working solver key is not automatically enabled for fingerprints, and the
# client says so by name if it is not).
TWOCAPTCHA_KEY=your_2captcha_api_key_here
# --- Scraping Browser API -------------------------------------------------
# A remote browser you do not run or patch, with a persistent profile and a
# chosen exit country. Used by --cdp-endpoint, and on this site it is what
# makes `--mode profile` work at all.
#
# country- picks the exit country. Use `country-us` (or `country-ca`):
# BBB is a US and Canadian directory, and a datacenter address is
# refused wherever it is. Measured: a `country-us` profile was
# served normally where a German datacenter VPS got 403 on every
# request, byte-identical over six polls.
# pid- is a profile with persistent cookies. ONE live connection each,
# capped per account — reuse pids rather than minting one per
# run, and expect a 500 (`profile_locked`, exit 5) while another
# run still holds this one. That is also why --concurrency is
# refused with --cdp-endpoint: several workers would collide.
#
# Note that Selenium cannot use this: chromedriver's `debuggerAddress` takes
# a bare host:port with nowhere to put a password. Playwright and pyppeteer
# authenticate on the WebSocket upgrade and work. selenium_scraper.py refuses
# a credentialed endpoint with exit 2 and says which engine to use instead —
# so `--mode profile` is the one mode that needs a specific engine.
#
# One more thing to expect: a credential here does not last. Of the eight
# Scraping Browser profiles and six proxy logins inherited from this
# family's sibling repos, exactly one still authenticated when this repo was
# written; the rest returned 401 `deny_no_user` or 407. That is why the
# canary workflow is dispatch-only rather than nightly.
BBB_CDP_ENDPOINT=ws://{login}-zone-scraping_browser-country-us-pid-{profileId}:{password}@cb.2captcha.com:9222
# --- Proxy ----------------------------------------------------------------
# A residential exit for `--mode profile`. One URL here, or --proxy-file with
# one per line for a pool; a pool is what spreads a run's volume, and
# --concurrency without one just sends N times the traffic from a single
# address.
#
# Region: BBB serves the US and Canada, so `-region-us` or `-region-ca`. What
# the site scores is the ADDRESS rather than its geography — but it scores a
# datacenter address in any country, so a residential one is not optional
# for the profile path.
#
# The credentials here never reach the browser's command line — they go
# through the driver's own fields. Logs keep the host and port and mask the
# rest, because which exit a run used is the point of the log and is not the
# secret. Selenium's --proxy-server cannot authenticate at all, so that
# engine strips the credentials and warns rather than letting you believe a
# user:pass URL is doing something.
#
# Not needed for the two listing modes. See the note at the top.
BBB_PROXY=http://{user}:{password}@na.proxy.2captcha.com:2334
# --- Default target -------------------------------------------------------
# Saves retyping --url. BBB serves the United States and Canada from ONE
# host, `www.bbb.org`, with the country in the PATH — so there is no
# per-country hostname to get wrong, and `find_country=USA|CAN` (or the
# `--country` flag) is what selects the directory.
#
# The three URL kinds, one per mode:
#
# /search?find_text=…&find_loc=… --mode search. `?page=N` addresses
# every page, up to the 15 BBB serves.
# /{cc}/category/{slug} --mode category. The slug is a human
# address; the endpoint filters on a
# numeric tobId, which the scraper
# resolves from BBB's own response.
# /{cc}/{st}/{city}/profile/{cat}/{n} --mode profile. One page, and the
# only one that is Cloudflare-gated.
#
# `--text` and `--location` build a search URL for you, so this is optional
# even for a search, and it is commented out so that a copied example does
# not become a default nobody chose. A URL set here is a default only: query
# flags typed on the command line replace it.
# BBB_URL=https://www.bbb.org/search?find_country=USA&find_text=restaurants&find_loc=New+York%2C+NY
# Deliberately NOT here: an output prefix. `--out` already carries a
# non-empty default, so the loader — which only fills UNSET values — would
# never see it, and the variable would be silently inert: a setting that
# looks configurable and is not.