From d1898bdf152f4f2c620e08d83c32988a809509f1 Mon Sep 17 00:00:00 2001 From: Tate McCauley Date: Tue, 17 Nov 2026 13:08:31 -0700 Subject: [PATCH 1/2] fix(jira): say how to fix a scoped token sent to the site URL Jira answers "401 Unauthorized; scope does not match" on every call when a scoped API token is used with JIRA_BASE_URL set to the site URL, since scoped tokens only work through the API gateway. Before, the run failed with a generic auth_failed. Now it stops at the first call with bad_config and names the gateway URL and the site's _edge/tenant_info link for the cloudId. Through the gateway the same answer means a missing scope and is reported as not_authorized. The README now lists all four Jira fetchers that need no admin permission, and adds sections on running without Jira admin access and on reading the "scope does not match" error. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 11 +++ fetchers/jira/README.md | 41 ++++++++- fetchers/jira/_shared/jira_common.py | 36 ++++++++ ...est_jira_admin_and_remediation_fetchers.py | 84 +++++++++++++++++++ 4 files changed, 168 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8f32a221..ff32213f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2626,6 +2626,17 @@ schemas and the `paramify` CLI — not the internal code. ### Fixed +- **Jira fetchers: a scoped API token sent to the site URL now gives the fix, + not a generic `auth_failed`.** Jira answers `401 Unauthorized; scope does not + match` on every call when a scoped token is used with + `JIRA_BASE_URL=https://.atlassian.net`. The run now stops with + `bad_config` and names the gateway URL to use + (`https://api.atlassian.com/ex/jira/`, or the Government Cloud + gateway) and the site's `_edge/tenant_info` link that gives the cloudId. + Through the gateway, the same answer is reported as `not_authorized`, a + missing scope. `fetchers/jira/README.md` now lists all four Jira fetchers that + need no admin access. New sections say how to run without Jira admin access + and how to read this error. - **Disk, Key Vault, storage and SQL Managed Instance fetchers: an unread list gives null counts, not 0.** When a list the run needs is refused (403), throttled or times out, every total, count and percentage built from it is diff --git a/fetchers/jira/README.md b/fetchers/jira/README.md index 197a3dbb..b0813064 100644 --- a/fetchers/jira/README.md +++ b/fetchers/jira/README.md @@ -32,10 +32,24 @@ Declared once on the category ([`_categories/jira.yaml`](../_categories/jira.yam Use a dedicated service account. It needs **Browse Projects** on every project you want in the evidence — projects it cannot see are simply absent, not reported as errors — and, for Jira Service Management sites, agent access to the -service desks to be listed. `jira_site_inventory` and `jira_remediation_items` -need no Jira administrator permission; `jira_audit_records` and -`jira_access_roles` need **Administer Jira**, so run them under a separate admin -collector account (override `secrets:` for those two in the manifest). +service desks to be listed. `jira_site_inventory`, `jira_remediation_items`, +`jira_change_requests` and `jira_incident_postmortems` need no Jira +administrator permission; `jira_audit_records` and `jira_access_roles` need +**Administer Jira**, so run them under a separate admin collector account +(override `secrets:` for those two in the manifest). + +### Without Jira admin access + +The **Administer Jira** requirement is on the account, not the token. Jira +checks it on the server for the audit log (`GET /rest/api/3/auditing/record`), +application roles and other projects' role members, and no token scope can +replace it. If your policy does not give a service account Jira admin, leave +`jira_audit_records` and `jira_access_roles` out of the manifest. The other four +fetchers run with Browse Projects and the read scopes below, with no admin +access. The evidence those two would have collected (the Jira audit log, Jira +admin group membership and privileged project grants) then has to come from +another source, for example your identity provider's group membership or an +audit log export from Atlassian Administration. ### Read-only scoped token (recommended) @@ -61,6 +75,25 @@ A scoped token only works through the API gateway. Find the site's `cloudId` at required. Scopes limit what the token can do; they do not grant permissions, so the admin collector account still needs Administer Jira. +#### `401 Unauthorized; scope does not match` + +Jira gives this answer in two cases: + +- **On every call, including `curl` tests:** `JIRA_BASE_URL` is the site URL + (`https://.atlassian.net`), but the token is scoped. Adding scopes does + not help. Switch `JIRA_BASE_URL` to the gateway URL above. A fetcher run stops + at the first call with `bad_config` and gives the `tenant_info` link to use. +- **On some calls, through the gateway:** the token lacks a scope that call + needs. The run records the call as `not_authorized`. Add the scope from the + table above. Do not add `manage:jira-configuration` to fix it. + +To test a scoped token by hand, call the gateway, not the site: + +```sh +curl -sS -u "$JIRA_EMAIL:$JIRA_API_TOKEN" -H 'Accept: application/json' \ + "https://api.atlassian.com/ex/jira//rest/api/3/myself" +``` + ### `jira_site_inventory` | Config key | Env | Default | Description | diff --git a/fetchers/jira/_shared/jira_common.py b/fetchers/jira/_shared/jira_common.py index e32dc7d4..0731886d 100644 --- a/fetchers/jira/_shared/jira_common.py +++ b/fetchers/jira/_shared/jira_common.py @@ -144,6 +144,21 @@ def request(self, method, path, operation, *, params=None, body=None, allow_404= return None message = error_message(resp) + if resp.status_code == 401 and SCOPE_MISMATCH in message.lower(): + # A scoped API token. Sent to a site URL every call fails this way, so + # the run stops with the fix; through the gateway it is one missing scope. + if not self.authenticated and not is_gateway_url(self.base_url): + self.fail(operation, "HTTP401", scoped_token_on_site_message(self.base_url), "bad_config") + raise Abort + self.fail( + operation, "HTTP401", + f"{message[:200]} - the scoped token lacks a scope this call needs; " + "see the scope table in fetchers/jira/README.md", + "not_authorized", + ) + if not self.authenticated: + raise Abort + return None if resp.status_code == 401: # Jira also answers 401 for a missing project permission (measured on # /project/{key}/role), so only a 401 before any success is a bad credential. @@ -633,6 +648,27 @@ def validate_base_url(raw: str): ) +# Atlassian's 401 body for a scoped token that does not fit the request: +# {"code":401,"message":"Unauthorized; scope does not match"}. +SCOPE_MISMATCH = "scope does not match" + + +def is_gateway_url(base_url: str) -> bool: + return (urlsplit(base_url).hostname or "") in _GATEWAY_HOSTS + + +def scoped_token_on_site_message(base_url: str) -> str: + """The fix for a scoped API token sent to a site URL (base_url is already validated).""" + host = urlsplit(base_url).hostname or "" + gateway = "api.atlassian-us-gov-mod.com" if host.endswith(".atlassian-us-gov-mod.net") else "api.atlassian.com" + return ( + "Jira answered 'scope does not match': JIRA_API_TOKEN is a scoped API token, and Jira " + "accepts a scoped token only through the API gateway, never at the site URL. Set " + f"JIRA_BASE_URL=https://{gateway}/ex/jira/, taking cloudId from " + f"{base_url}/_edge/tenant_info" + ) + + def connection_settings(): """Read the category settings. Returns (base_url, email, token), or an error string.""" raw_url = os.environ.get("JIRA_BASE_URL", "").strip() diff --git a/tests/test_jira_admin_and_remediation_fetchers.py b/tests/test_jira_admin_and_remediation_fetchers.py index 8ed784b4..e888fda4 100644 --- a/tests/test_jira_admin_and_remediation_fetchers.py +++ b/tests/test_jira_admin_and_remediation_fetchers.py @@ -352,6 +352,90 @@ def no_network(*a, **k): assert status["code"] == "bad_config" and "evil.example" not in status["error"] +SCOPE_401 = {"code": 401, "message": "Unauthorized; scope does not match"} +JQL_ENV = { + "JIRA_REMEDIATION_JQL": "labels = x", + "JIRA_CHANGE_JQL": "labels = x", + "JIRA_INCIDENT_JQL": "labels = x", +} +ALL_JIRA = ["site_inventory", "remediation_items", "audit_records", "access_roles", + "change_requests", "incident_postmortems"] + + +def _gateway_run(monkeypatch, tmp_path, name, router, base_url): + calls = [] + monkeypatch.setattr("time.sleep", lambda s: None) + + def fake_request(self, method, url, params=None, json=None, timeout=None, **kw): + assert url.startswith(base_url) + path = url[len(base_url):] + calls.append(path) + status, body = router(method, path) + return FakeResponse(status, body) + + monkeypatch.setattr(requests.Session, "request", fake_request) + for k, v in {"JIRA_BASE_URL": base_url, "JIRA_EMAIL": "svc@example.com", "JIRA_API_TOKEN": "t", + "EVIDENCE_DIR": str(tmp_path), "FETCHER_STATUS_FILE": str(tmp_path / "s.json"), + **JQL_ENV}.items(): + monkeypatch.setenv(k, v) + code = load(name).main() + return code, calls, json.loads((tmp_path / "s.json").read_text()) + + +@pytest.mark.parametrize("name", ALL_JIRA) +def test_scoped_token_at_site_url_says_to_use_the_gateway(monkeypatch, tmp_path, name): + """A scoped token sent to the site URL gets 'scope does not match' on every call.""" + code, calls, status = _gateway_run( + monkeypatch, tmp_path, name, lambda m, p: (401, SCOPE_401), "https://example.atlassian.net" + ) + assert code == 1 and len(calls) == 1 # stops at the first call, not one failure per endpoint + assert status["code"] == "bad_config" + assert "https://api.atlassian.com/ex/jira/" in status["error"] + assert "https://example.atlassian.net/_edge/tenant_info" in status["error"] + + +def test_scoped_token_at_gov_site_url_names_the_gov_gateway(monkeypatch, tmp_path): + _, _, status = _gateway_run( + monkeypatch, tmp_path, "audit_records", lambda m, p: (401, SCOPE_401), + "https://example.atlassian-us-gov-mod.net", + ) + assert status["code"] == "bad_config" + assert "https://api.atlassian-us-gov-mod.com/ex/jira/" in status["error"] + + +def test_scope_mismatch_through_the_gateway_is_a_missing_scope(monkeypatch, tmp_path): + """Through the gateway the URL is right; the token lacks a scope for one call.""" + def router(method, path): + if path == "/rest/api/3/myself": + return 200, ME + return 401, SCOPE_401 + + code, _, status = _gateway_run( + monkeypatch, tmp_path, "audit_records", router, + "https://api.atlassian.com/ex/jira/eadac007-8666-4024-a026-0bfb4d34aa2e", + ) + assert code == 1 and status["code"] == "not_authorized" + assert "lacks a scope" in status["error"] and "tenant_info" not in status["error"] + + +def test_scope_mismatch_on_first_gateway_call_is_not_a_bad_url(monkeypatch, tmp_path): + code, calls, status = _gateway_run( + monkeypatch, tmp_path, "remediation_items", lambda m, p: (401, SCOPE_401), + "https://api.atlassian.com/ex/jira/eadac007-8666-4024-a026-0bfb4d34aa2e", + ) + assert code == 1 and len(calls) == 1 + assert status["code"] == "not_authorized" and "lacks a scope" in status["error"] + + +def test_plain_401_is_still_auth_failed(monkeypatch, tmp_path): + _, _, status = _gateway_run( + monkeypatch, tmp_path, "audit_records", + lambda m, p: (401, {"errorMessages": ["Client must be authenticated"]}), + "https://example.atlassian.net", + ) + assert status["code"] == "auth_failed" and "gateway" not in status["error"] + + def test_redirect_is_not_followed_and_fails(run): def router(method, path, params, body): if path == "/rest/api/3/myself": From 4969ccec2caf22c7f60356f1ca19ebdf536d7214 Mon Sep 17 00:00:00 2001 From: Tate McCauley Date: Tue, 17 Nov 2026 13:16:15 -0700 Subject: [PATCH 2/2] docs(jira): scope mismatch on every gateway call means missing /myself scopes Co-Authored-By: Claude Opus 5.5 --- fetchers/jira/README.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/fetchers/jira/README.md b/fetchers/jira/README.md index b0813064..2508d09c 100644 --- a/fetchers/jira/README.md +++ b/fetchers/jira/README.md @@ -79,15 +79,19 @@ the admin collector account still needs Administer Jira. Jira gives this answer in two cases: -- **On every call, including `curl` tests:** `JIRA_BASE_URL` is the site URL - (`https://.atlassian.net`), but the token is scoped. Adding scopes does - not help. Switch `JIRA_BASE_URL` to the gateway URL above. A fetcher run stops - at the first call with `bad_config` and gives the `tenant_info` link to use. -- **On some calls, through the gateway:** the token lacks a scope that call - needs. The run records the call as `not_authorized`. Add the scope from the - table above. Do not add `manage:jira-configuration` to fix it. - -To test a scoped token by hand, call the gateway, not the site: +- **`JIRA_BASE_URL` is the site URL** (`https://.atlassian.net`) and the + token is scoped. Every call fails, and adding scopes does not help. Switch + `JIRA_BASE_URL` to the gateway URL above. A fetcher run stops at the first call + with `bad_config` and gives the `tenant_info` link to use. +- **Through the gateway:** the token lacks a scope that call needs. The run + records the call as `not_authorized`. Add the scope from the table above. Do + not add `manage:jira-configuration` to fix it. If *every* call fails through + the gateway, check the four scopes `GET /rest/api/3/myself` needs first + (`read:application-role:jira`, `read:group:jira`, `read:user:jira`, + `read:avatar:jira`). Each fetcher calls it first, and a token made only for + browsing projects usually lacks them. + +To test a scoped token by hand, call the gateway, not the site, starting with `/myself`: ```sh curl -sS -u "$JIRA_EMAIL:$JIRA_API_TOKEN" -H 'Accept: application/json' \