Skip to content

Add Firefox 152-158 emulation profiles - #12

Open
galetahub wants to merge 1 commit into
mainfrom
firefox-152-158
Open

galetahub wants to merge 1 commit into
mainfrom
firefox-152-158

Conversation

@galetahub

Copy link
Copy Markdown

Farmed cookies for the Bing engine work only with Firefox 158 (https://github.com/serpapi/SerpApi/pull/10812). So it would also be useful to support replaying requests through wreq.

Fable 5.2 High:

Summary

Adds firefox_152 … firefox_158. The pinned wreq-util (0.2.0) stops at Firefox 151, so these profiles are defined locally in ext/wreq_rb/src/firefox.rs on top of its Firefox 151 profile. A name that wreq-util knows still resolves to its profile, and a unit test fails once it ships one of these releases, so the local entry can be dropped then.

What each profile sets

Versions Cipher suites Supported groups
152, 153 16 with ffdhe2048 / ffdhe3072
154, 155 15 (no ECDHE_ECDSA_AES_256_CBC_SHA) with ffdhe2048 / ffdhe3072
156 – 158 15 without ffdhe

All seven also correct three things the bundled Firefox 151 profile gets wrong: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA is no longer offered, Accept-Language is en-US,en;q=0.9, and the HTTP/2 HEADERS priority weight is 42.

HTTP/1.1 request

These profiles also reproduce Firefox's HTTP/1.1 request, on every emulation_os:

  • Host first, header names spelled and ordered as Firefox sends them on a navigation, and Connection: keep-alive (the HTTP/2 client strips it).
  • Headers the caller adds (Cookie, Referer, Origin, Content-Type, cache validators, …) take Firefox's position, over HTTP/2 as well.
  • Headers the profile does not know keep the caller's order and spelling, in the place where Firefox writes headers set by a script.
  • TE: trailers is left out for http1_only clients. It cannot be dropped when the protocol is negotiated.
  • header_order keeps precedence over the profile's order.

Bundled profiles are unchanged on the wire.

Verification

  • Compared against the official macOS release builds of Firefox 152.0 – 158.0: JA3, JA4, the Akamai fingerprint, the ClientHello (cipher suites, extension order and bodies, groups, key share groups, signature algorithms), the HTTP/2 SETTINGS / WINDOW_UPDATE / HEADERS priority, and header names, values and order all match. The plain HTTP/1.1 GET is byte-identical with http1_only.
  • Clean clone of this commit builds, packages and installs as a gem on Ruby 3.4.8 and 2.7.8 (macOS arm64); 18 Rust unit tests and 76 Ruby tests pass on both.
  • Request bytes of all 133 bundled profiles are identical to released 0.6.1 across 3,112 option scenarios on HTTP/1.1, plus HTTP/2 and ClientHello comparisons.
  • Not run locally: the Linux CI job.

Known gaps

None of these is visible in JA3, JA4 or the Akamai fingerprint; all come from the TLS / HTTP stack and also apply to the bundled Firefox profiles:

  • GREASE ECH: Firefox sends a fixed payload length (239 up to 154, 240 from 155); BoringSSL picks 144/176/208/240 at random.
  • Firefox reuses its X25519 key inside the X25519MLKEM768 key share; BoringSSL generates two keys.
  • HTTP/2: no stream-level WINDOW_UPDATE after HEADERS, and :path: / is HPACK-indexed.
  • http1_only narrows ALPN to http/1.1, which Firefox never offers, so over TLS it is either an extra TE header or a different ALPN.
  • Android and iOS platforms only swap the User-Agent.

The pinned wreq-util stops at Firefox 151, so define firefox_152 to firefox_158 locally on top of its Firefox 151 profile. Each overrides what differs on the wire: the cipher suites (154 dropped ECDHE_ECDSA_AES_256_CBC_SHA), the supported groups (156 dropped ffdhe2048/ffdhe3072), Accept-Language q=0.9 and HEADERS priority weight 42. JA3, JA4, the Akamai fingerprint, the ClientHello extensions and the HTTP/2 request match the official macOS release build of each version.

Reproduce Firefox's HTTP/1.1 request for these profiles as well: Host first, header names spelled and ordered as on a navigation, and Connection: keep-alive. Headers the caller adds take Firefox's position on both protocols, and headers the profile does not know keep the caller's order and spelling. TE is left out for http1_only clients. header_order keeps precedence, and bundled profiles are unchanged on the wire.

Add unit tests for the profile table and the header order, and local wire tests for the HTTP/1.1 request.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Wire-fingerprint correctness requires human verification of the pinned vendored wreq implementation and Firefox capture comparisons, both absent from the review snapshot.

No open issues 🎉

What changed in this PR

Adds Firefox 152–158 emulation to wreq-rb so callers can replay requests using newer Firefox profiles beyond those bundled with wreq-util.

Changes:

  • Defines local profiles with release-specific TLS settings, updated headers, and HTTP/2 priority.
  • Integrates Firefox header spelling and ordering into client, per-request, and batch requests while preserving explicit header_order precedence.
  • Adds Rust and Ruby coverage and documents protocol limitations and profile-mixing behavior.
File Description
test/​emulation_test.rb Tests new profile selection, User-Agent values, platform overrides, and unknown releases.
test/​client_test.rb Tests HTTP/1.1 header spelling, ordering, overrides, and batch behavior.
README.md Documents new profiles, header behavior, and known limitations.
ext/​wreq_rb/​src/​lib.rs Registers the Firefox profile module.
ext/​wreq_rb/​src/​firefox.rs Defines Firefox 152–158 profiles and profile-level unit tests.
ext/​wreq_rb/​src/​client.rs Integrates local profiles and tracks header ordering across request paths.

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants