Error while authenticating

[adam-sikora]

I have encountered an error while authenticating CapellaConsoleClient. A lot of clients were initiated simultaneously from separate machines so it might be potentially connected to some rate limiting. Most of the authentications passed fine but some of them failed.

Relevant part of the traceback:

  File "/usr/local/lib/python3.11/dist-packages/capella_console_client/client.py", line 83, in __init__
    self._sesh.authenticate(email, password, token, no_token_check)
  File "/usr/local/lib/python3.11/dist-packages/capella_console_client/session.py", line 64, in authenticate
    self._basic_auth(email, password)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/capella_console_client/session.py", line 113, in _basic_auth
    resp = self.post("/token", headers={"Authorization": f"Basic {basic_token}"})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 1136, in post
    return self.request(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 821, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/capella_console_client/session.py", line 146, in send
    ret = super().send(*fct_args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 908, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 936, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 993, in _send_handling_redirects
    raise exc
  File "/usr/local/lib/python3.11/dist-packages/httpx/_client.py", line 976, in _send_handling_redirects
    hook(response)
  File "/usr/local/lib/python3.11/dist-packages/capella_console_client/hooks.py", line 26, in translate_error_to_exception
    handle_error_response_and_raise(response)
  File "/usr/local/lib/python3.11/dist-packages/capella_console_client/exceptions.py", line 94, in handle_error_response_and_raise
    error = response.json()
            ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/httpx/_models.py", line 755, in json
    return jsonlib.loads(self.content.decode(encoding), **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)"

[Shaunakde]

Do you have an idea of how many clients were simultaneously trying to auth simultaneously? I tested on about ~12 multi-processing-ed clients and didn't hit a limit.

One solution I can think of is to add a re-try with exponential back-off to the auth function to mitigate, but would be nice to know how many simultaneous clients hit these limits so we can set the back-off parameters, I think.

[adam-sikora]

We are using potentially hundreds of workers at the same time as our platform scales dynamically. I unfortunately can not give you anything more specific as we have since added session persistence and retrying so it is no longer an issue for us.

I have reported this issue as the Python package fails to properly raise some subclass of CapellaConsoleClientError and instead fails with JSONDecodeError

[adam-sikora]

We have hit this error again on our platform and retrying did not help. It was happening from 2024-02-20T08:49:31Z to 2024-02-20T08:53:12Z. I don't see any errors outside of this time range in our logs and the requests work correctly now.

This suggests some temporary outage of Capella API and subsequent error in handling of error response.

[Shaunakde]

Thanks for sending this report in. I can try to look at this over the weekend. I’ll set up an experiment and try to trap the point at which we could raise a better error message. More wholistically, I’ll try to investigate to the root cause. From: Adam Sikora ***@***.***> Date: Tuesday, February 20, 2024 at 1:56 AM To: capellaspace/console-client ***@***.***> Cc: Shaunak De ***@***.***>, Comment ***@***.***> Subject: Re: [capellaspace/console-client] Error while authenticating (Issue #52) We have hit this error again on our platform and retrying did not help. It was happening from 2024-02-20T08:49:31Z to 2024-02-20T08:53:12Z. I don't see any errors outside of this time range in our logs and the requests work correctly now. This suggests some temporary outage of Capella API and subsequent error in handling of error response. — Reply to this email directly, view it on GitHub<#52 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AATNL2NXANAQWBWSQA4H2QLYURXOHAVCNFSM6AAAAABASWSHZCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJTHA2TGMRXGQ>. You are receiving this because you commented.Message ID: ***@***.***>