fix cloudflare 403 with curl_cffi

[bjia56]

Cloudflare frequently returns a 403 on the Arlo login, which to the best of my understanding is decided on client heuristics, i.e. if a client is determined with some confidence to not be a browser, the request is blocked. One of the heuristics used is the TLS fingerprint of the client. The curl_cffi project aims to enable Python scripts to bypass this TLS fingerprinting check through creating TLS handshakes that are identical to a real Chrome browser. This is accomplished through the use of the curl-impersonate project, which compiles curl with the exact TLS/SSL libraries used by Chrome and Firefox.

We have seen reasonable success in using this option over cloudscraper in the Scrypted home automation project. A caveat is that curl_cffi exposes a requests-like API, but does not implement the full API; therefore, curl_cffi is only used for the initial login, and subsequent requests fall back on requests so sseclient can continue working. Arlo appears to be happy with just the token in the header, and performs no additional TLS fingerprinting checks after login succeeds.

Note that this is not tested comprehensively, since I do not use this version of the arlo library.

Fixes #204

[Johnathan-Aretos]

Can confirm that this works as expected. Before this I had problem with authenticating.

[ahass-thedev]

Works as expected - Was not able to authenticate before

[scuc]

I tried to run this fix, but it throws an error - see below.
I can confirm that /usr/lib/libSystem.B.dylib is present on my system.
I have tried to run this code from both pipenv and just a python3.10 local install.
but I get the same error each time.
only part of the traceback that stands out to me is - "built for Mac OS X 11.0"
i'm running on Mojave 10.14.6 - have not been able to test on Mac OS 11 system.

Traceback (most recent call last):
  File "/Users/<user>/_Github/arlo-fix/arlo-cloudflarefix/main.py", line 4, in <module>
    from arlo import Arlo
  File "/Users/<user>/_Github/arlo-fix/arlo-cloudflarefix/arlo.py", line 27, in <module>
    from request import Request
  File "/Users/<user>/_Github/arlo-fix/arlo-cloudflarefix/request.py", line 19, in <module>
    from curl_cffi import requests as curl_cffi_requests
  File "/Users/<user>/.local/share/virtualenvs/arlo-cloudflarefix-ZHEWMQYK/lib/python3.10/site-packages/curl_cffi/__init__.py", line 4, in <module>
    from ._wrapper import ffi, lib  # type: ignore
ImportError: dlopen(/Users/<user>/.local/share/virtualenvs/arlo-cloudflarefix-ZHEWMQYK/lib/python3.10/site-packages/curl_cffi/_wrapper.abi3.so, 2): Symbol not found: ___darwin_check_fd_set_overflow
  Referenced from: /Users/<user>/.local/share/virtualenvs/arlo-cloudflarefix-ZHEWMQYK/lib/python3.10/site-packages/curl_cffi/.dylibs/libcurl-impersonate-chrome.4.dylib (which was built for Mac OS X 11.0)
  Expected in: /usr/lib/libSystem.B.dylib
 in /Users/<user>/.local/share/virtualenvs/arlo-cloudflarefix-ZHEWMQYK/lib/python3.10/site-packages/curl_cffi/.dylibs/libcurl-impersonate-chrome.4.dylib

[bjia56]

The published curl_cffi package may have been built on a newer MacOS version, which unfortunately means it is not guaranteed to be backwards compatible to run on older versions

[wranglerdriver]

I have tried this patch and it is not working for me. I tried Saturday and then again this morning thinking maybe my IP was still blocked. Mac OS 15.1.1 python 3.13

There are no errors other than the "403 Client Error: Forbidden for url: https://ocapi-app.arlo.com/api/auth"

[wranglerdriver]

I have tried this patch and it is not working for me. I tried Saturday and then again this morning thinking maybe my IP was still blocked. Mac OS 15.1.1 python 3.13

There are no errors other than the "403 Client Error: Forbidden for url: https://ocapi-app.arlo.com/api/auth"

Sorry - it does work. I somehow had screwed up my venv and it was using system packages instead of the virtual environment.

Thanks for working out a fix for this!

[kidtangerine]

This was working for me for months and today at 4 pm CST, I started getting the 403 again. Anyone else start having this issue with arlo?

[bjia56]

Per investigation here: twrecked/hass-aarlo#1033

The old Netgear "Arlo" user-agent is no longer being accepted. Updating to the "Linux" user-agent allows things to keep working.

Note that streaming will return a MPD url when the user-agent matches a desktop/browser. I have found that this "Android" user-agent works to request RTSP like before.

[kidtangerine]

@bjia56 you are awesome. My app is working again with the Android user-agent.

[kidtangerine]

Randomly started getting this error:

Failed to start eventstream thread: 403 Client Error: Forbidden for url: https://myapi.arlo.com.
I'm getting past the login but stopping here.

Anyone else seeing this?

[bjia56]

From inspection of Arlo's web dashboard, it seems they are not currently using SSE in the browser, and instead using MQTT over websockets. The old endpoint might still work with some tweaking however it will be difficult to know what exactly they're looking for without a reference. You could try updating curl_cffi or changing the impersonate target in request.py to latest supported Chrome (impersonate="chrome").

Since this project doesn't have MQTT support, I would suggest moving off of this into one that's more maintained, such as pyaarlo. You could also try the one in our Scrypted plugin, which is a heavily modified version of this project to convert everything into async semantics. Or, take our implementation of MQTT and convert it into something that this project can use.

[kidtangerine]

@bjia56 grateful for your responses:

I simple need a snapshot of the cameras to build out a time lapse video project. Based on your experience, which repo should I focus on for such a basic task, pyaarlo or your Scrypted Plugin?

[bjia56]

I'm not as familiar with how to use pyaarlo. It's probably more lightweight than setting up an entire Scrypted instance, however with Scrypted you could configure things up so that a snapshot is a simple curl command to a local url (using Arlo plugin + webhook plugin). Using the arlo module from within the Scrypted plugin might be a bit less straightforward since it wasn't designed to be used outside of the plugin.