Identify paywall types and generate bypass strategies for web scraping and content extraction.
This skill provides comprehensive guidance for identifying paywall technologies and extracting article content from paywalled news sites. It covers 500+ news sites, major paywall providers (Piano, TinyPass, Poool, Sophi, Evolok, Qiota, and more), and multiple extraction techniques including bot spoofing, AMP access, JSON-LD extraction, and archive fetching.
Source: Extracted from Bypass Paywalls Clean
- Extracting full article text from paywalled URLs for research or scraping
- Identifying what paywall technology a specific site uses
- Writing scripts that handle paywalled content programmatically
- Understanding cookie behavior that triggers or avoids paywalls
- Cracking DRM or decrypting encrypted content
- Illegal access to credentials or personal data
- Bypassing non-paywall access control (login-only portals, intranets)
- Commercial distribution of full paywalled content
| Type | Mechanism | Bypass Difficulty |
|---|---|---|
| Metered | Tracks views via cookies/localStorage; blocks after N free articles | Easy |
| Hard | Content loaded but hidden behind CSS/JS overlay | Easy |
| Server-side | Content not sent to browser; requires external fetching | Medium |
| Hybrid | Combination of client-side blocking + server-side checks | Hard |
Inspect page source for these signatures:
| Provider | Detection Pattern | Notable Sites |
|---|---|---|
| Piano.io | .piano.io/, .piano.io/xbuilder/ |
Foreign Policy, DN.no, Funke |
| TinyPass | .tinypass.com/, js.tinypass.com |
Fortune, Adweek, Corriere.it |
| Poool.fr | .poool.fr/ |
Elle.fr, Challenges, Le Parisien |
| Sophi.io | .sophi.io/ |
Business Insider, Slate, Advance Local |
| Evolok | .evolok.net/ |
El País, Vocento group |
| Qiota | .qiota.com/ |
AutoPlus.fr, Cosmopolitan.fr |
| Pelcro | js.pelcro.com/ |
Domani, Foreign Affairs |
| Cxense | .cxense.com/ |
Barron's, Business Insider JP |
| Blueconic | .blueconic.net/ |
Bridge Tower Media |
| Steady | steadyhq.com/ |
American Purpose |
| Leaky Paywall | /leaky-paywall/ |
WordPress sites |
| AMP Access | .ampproject.org/v0/amp-access- |
News Corp Australia, Artnet |
1. Identify paywall type
└── Check page source for provider scripts
└── Check cookies and localStorage keys
└── Check for AMP version (<link rel="amphtml">)
└── Check for JSON-LD data
└── Check for Next.js data (__NEXT_DATA__)
2. Choose technique (least invasive first)
├── Cookie clear / localStorage clear
├── Block paywall script
├── Bot UA spoof (Googlebot, Bingbot)
├── AMP page extraction
├── JSON-LD extraction
├── Next.js data extraction
└── Archive.is fetch (fallback)
| Technique | Best For | Implementation |
|---|---|---|
| Script Blocking | Piano, TinyPass, Poool, Sophi | Block the script URL; content loads but overlay doesn't trigger |
| Cookie Clearing | Metered paywalls | Clear tracking cookies + localStorage.clear() |
| Bot UA Spoof | Server-side UA checks | Set User-Agent to Googlebot/Bingbot |
| AMP Extract | Sites with AMP versions | Navigate to AMP URL, unhide content |
| JSON-LD Parse | Structured data embeds | Parse script[type="application/ld+json"] for articleBody |
| Next.js Data | React/Next.js sites | Extract from script#__NEXT_DATA__ |
| Archive.is | Server-side paywalls | Fetch from archive.is / archive.today |
Many news sites share paywall infrastructure. Identifying the media group simplifies bypass:
| Group | Key Sites | Paywall Type |
|---|---|---|
| News Corp Australia | theaustralian.com.au, dailytelegraph.com.au | AMP subscriptions |
| McClatchy (USA) | miamiherald.com, sacbee.com, kansascity.com | Piano + Googlebot |
| Gannett (USA) | azcentral.com, freep.com, indystar.com | Googlebot UA |
| Vocento (ES) | abc.es, larioja.com, ideal.es | Evolok + AMP |
| GEDI (IT) | repubblica.it, lastampa.it | Piano + Googlebot |
| DPG Media (NL) | volkskrant.nl, trouw.nl, demorgen.be | Custom (TID_ID cookie) |
| Groupe Rossel (FR/BE) | lavoixdunord.fr, lesoir.be | Qiota |
| Funke (DE) | abendblatt.de, morgenpost.de | Piano |
| Crain Comm (USA) | adage.com, autonews.com | Pelcro + Sophi |
| Conde Nast (USA) | newyorker.com, vogue.com, wired.com | Custom + script blocking |
Different paywall providers use different cookies for tracking:
| Provider | Tracking Cookie | Action |
|---|---|---|
| DPG Media (NL) | TID_ID |
Drop to reset meter |
| Haaretz Group | ra |
Drop |
| Adweek/Zephr | blaize_session |
Drop |
| Pitchfork | pay_ent_msmp |
Drop |
| Business Standard | userUid |
Drop |
| DN Media (NO) | AnonUserCookie |
Drop |
| ambito.com (AR) | TDNotesRead |
Drop |
paywall-bypass/
├── SKILL.md # Main skill documentation
├── examples/
│ ├── identify-paywall.md # Walkthrough for detecting paywall type
│ └── bypass-strategy.md # Step-by-step bypass recipes for specific sites
└── references/
├── paywall-patterns.md # Complete regex patterns for all providers
├── content-extraction.md # JSON-LD, AMP, archive.is, Next.js extraction
└── cookie-rules.md # Per-site cookie management rules
import re
PAYWALL_PATTERNS = {
'Piano/TinyPass': r'tinypass\.com|piano\.io',
'Poool': r'poool\.fr',
'Sophi': r'sophi\.io',
'Evolok': r'evolok\.net',
'Qiota': r'qiota\.com',
'Cxense': r'cxense\.com',
'Pelcro': r'js\.pelcro\.com',
'Blueconic': r'blueconic\.net',
'Steady': r'steadyhq\.com',
'Leaky Paywall': r'leaky-paywall',
'AMP Access': r'ampproject\.org/v0/amp-access-',
'AMP Subscriptions': r'ampproject\.org/v0/amp-subscriptions-',
}
def identify_paywall(html_source):
"""Identify paywall provider from page source."""
results = {}
for provider, pattern in PAYWALL_PATTERNS.items():
matches = re.findall(pattern, html_source, re.IGNORECASE)
if matches:
results[provider] = list(set(matches))
return results if results else 'Unknown/Custom paywall'GOOGLEBOT = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
BINGBOT = "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
FACEBOOKBOT = "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"- Always inspect first — check page source for paywall scripts before attempting bypass
- Start least invasive — cookie clear → script block → UA spoof → external fetch
- Handle JS-rendered content — use Puppeteer/Playwright for React/Next.js sites
- Respect rate limits — add delays between requests
- Validate output — verify extracted text is complete, not truncated teaser content
- Use archive.is as fallback — when all client-side techniques fail
- Bypass Paywalls Clean (source) — Original browser extension
- archive.is — Web archive for paywall fallback
- Google Cache — Cached page fallback
This skill is extracted from the Bypass Paywalls Clean project by magnolia1234 for educational and research purposes. Use responsibly and in compliance with applicable laws.