Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2626,6 +2626,17 @@ schemas and the `paramify` CLI — not the internal code.

### Fixed

- **Jira fetchers: a scoped API token sent to the site URL now gives the fix,
not a generic `auth_failed`.** Jira answers `401 Unauthorized; scope does not
match` on every call when a scoped token is used with
`JIRA_BASE_URL=https://<site>.atlassian.net`. The run now stops with
`bad_config` and names the gateway URL to use
(`https://api.atlassian.com/ex/jira/<cloudId>`, or the Government Cloud
gateway) and the site's `_edge/tenant_info` link that gives the cloudId.
Through the gateway, the same answer is reported as `not_authorized`, a
missing scope. `fetchers/jira/README.md` now lists all four Jira fetchers that
need no admin access. New sections say how to run without Jira admin access
and how to read this error.
- **Disk, Key Vault, storage and SQL Managed Instance fetchers: an unread list
gives null counts, not 0.** When a list the run needs is refused (403),
throttled or times out, every total, count and percentage built from it is
Expand Down
45 changes: 41 additions & 4 deletions fetchers/jira/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,24 @@ Declared once on the category ([`_categories/jira.yaml`](../_categories/jira.yam
Use a dedicated service account. It needs **Browse Projects** on every project
you want in the evidence — projects it cannot see are simply absent, not
reported as errors — and, for Jira Service Management sites, agent access to the
service desks to be listed. `jira_site_inventory` and `jira_remediation_items`
need no Jira administrator permission; `jira_audit_records` and
`jira_access_roles` need **Administer Jira**, so run them under a separate admin
collector account (override `secrets:` for those two in the manifest).
service desks to be listed. `jira_site_inventory`, `jira_remediation_items`,
`jira_change_requests` and `jira_incident_postmortems` need no Jira
administrator permission; `jira_audit_records` and `jira_access_roles` need
**Administer Jira**, so run them under a separate admin collector account
(override `secrets:` for those two in the manifest).

### Without Jira admin access

The **Administer Jira** requirement is on the account, not the token. Jira
checks it on the server for the audit log (`GET /rest/api/3/auditing/record`),
application roles and other projects' role members, and no token scope can
replace it. If your policy does not give a service account Jira admin, leave
`jira_audit_records` and `jira_access_roles` out of the manifest. The other four
fetchers run with Browse Projects and the read scopes below, with no admin
access. The evidence those two would have collected (the Jira audit log, Jira
admin group membership and privileged project grants) then has to come from
another source, for example your identity provider's group membership or an
audit log export from Atlassian Administration.

### Read-only scoped token (recommended)

Expand All @@ -61,6 +75,29 @@ A scoped token only works through the API gateway. Find the site's `cloudId` at
required. Scopes limit what the token can do; they do not grant permissions, so
the admin collector account still needs Administer Jira.

#### `401 Unauthorized; scope does not match`

Jira gives this answer in two cases:

- **`JIRA_BASE_URL` is the site URL** (`https://<site>.atlassian.net`) and the
token is scoped. Every call fails, and adding scopes does not help. Switch
`JIRA_BASE_URL` to the gateway URL above. A fetcher run stops at the first call
with `bad_config` and gives the `tenant_info` link to use.
- **Through the gateway:** the token lacks a scope that call needs. The run
records the call as `not_authorized`. Add the scope from the table above. Do
not add `manage:jira-configuration` to fix it. If *every* call fails through
the gateway, check the four scopes `GET /rest/api/3/myself` needs first
(`read:application-role:jira`, `read:group:jira`, `read:user:jira`,
`read:avatar:jira`). Each fetcher calls it first, and a token made only for
browsing projects usually lacks them.

To test a scoped token by hand, call the gateway, not the site, starting with `/myself`:

```sh
curl -sS -u "$JIRA_EMAIL:$JIRA_API_TOKEN" -H 'Accept: application/json' \
"https://api.atlassian.com/ex/jira/<cloudId>/rest/api/3/myself"
```

### `jira_site_inventory`

| Config key | Env | Default | Description |
Expand Down
36 changes: 36 additions & 0 deletions fetchers/jira/_shared/jira_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,21 @@ def request(self, method, path, operation, *, params=None, body=None, allow_404=
return None

message = error_message(resp)
if resp.status_code == 401 and SCOPE_MISMATCH in message.lower():
# A scoped API token. Sent to a site URL every call fails this way, so
# the run stops with the fix; through the gateway it is one missing scope.
if not self.authenticated and not is_gateway_url(self.base_url):
self.fail(operation, "HTTP401", scoped_token_on_site_message(self.base_url), "bad_config")
raise Abort
self.fail(
operation, "HTTP401",
f"{message[:200]} - the scoped token lacks a scope this call needs; "
"see the scope table in fetchers/jira/README.md",
"not_authorized",
)
if not self.authenticated:
raise Abort
return None
if resp.status_code == 401:
# Jira also answers 401 for a missing project permission (measured on
# /project/{key}/role), so only a 401 before any success is a bad credential.
Expand Down Expand Up @@ -633,6 +648,27 @@ def validate_base_url(raw: str):
)


# Atlassian's 401 body for a scoped token that does not fit the request:
# {"code":401,"message":"Unauthorized; scope does not match"}.
SCOPE_MISMATCH = "scope does not match"


def is_gateway_url(base_url: str) -> bool:
return (urlsplit(base_url).hostname or "") in _GATEWAY_HOSTS


def scoped_token_on_site_message(base_url: str) -> str:
"""The fix for a scoped API token sent to a site URL (base_url is already validated)."""
host = urlsplit(base_url).hostname or ""
gateway = "api.atlassian-us-gov-mod.com" if host.endswith(".atlassian-us-gov-mod.net") else "api.atlassian.com"
return (
"Jira answered 'scope does not match': JIRA_API_TOKEN is a scoped API token, and Jira "
"accepts a scoped token only through the API gateway, never at the site URL. Set "
f"JIRA_BASE_URL=https://{gateway}/ex/jira/<cloudId>, taking cloudId from "
f"{base_url}/_edge/tenant_info"
)


def connection_settings():
"""Read the category settings. Returns (base_url, email, token), or an error string."""
raw_url = os.environ.get("JIRA_BASE_URL", "").strip()
Expand Down
84 changes: 84 additions & 0 deletions tests/test_jira_admin_and_remediation_fetchers.py
Original file line number Diff line number Diff line change
Expand Up @@ -352,6 +352,90 @@ def no_network(*a, **k):
assert status["code"] == "bad_config" and "evil.example" not in status["error"]


SCOPE_401 = {"code": 401, "message": "Unauthorized; scope does not match"}
JQL_ENV = {
"JIRA_REMEDIATION_JQL": "labels = x",
"JIRA_CHANGE_JQL": "labels = x",
"JIRA_INCIDENT_JQL": "labels = x",
}
ALL_JIRA = ["site_inventory", "remediation_items", "audit_records", "access_roles",
"change_requests", "incident_postmortems"]


def _gateway_run(monkeypatch, tmp_path, name, router, base_url):
calls = []
monkeypatch.setattr("time.sleep", lambda s: None)

def fake_request(self, method, url, params=None, json=None, timeout=None, **kw):
assert url.startswith(base_url)
path = url[len(base_url):]
calls.append(path)
status, body = router(method, path)
return FakeResponse(status, body)

monkeypatch.setattr(requests.Session, "request", fake_request)
for k, v in {"JIRA_BASE_URL": base_url, "JIRA_EMAIL": "svc@example.com", "JIRA_API_TOKEN": "t",
"EVIDENCE_DIR": str(tmp_path), "FETCHER_STATUS_FILE": str(tmp_path / "s.json"),
**JQL_ENV}.items():
monkeypatch.setenv(k, v)
code = load(name).main()
return code, calls, json.loads((tmp_path / "s.json").read_text())


@pytest.mark.parametrize("name", ALL_JIRA)
def test_scoped_token_at_site_url_says_to_use_the_gateway(monkeypatch, tmp_path, name):
"""A scoped token sent to the site URL gets 'scope does not match' on every call."""
code, calls, status = _gateway_run(
monkeypatch, tmp_path, name, lambda m, p: (401, SCOPE_401), "https://example.atlassian.net"
)
assert code == 1 and len(calls) == 1 # stops at the first call, not one failure per endpoint
assert status["code"] == "bad_config"
assert "https://api.atlassian.com/ex/jira/<cloudId>" in status["error"]
assert "https://example.atlassian.net/_edge/tenant_info" in status["error"]


def test_scoped_token_at_gov_site_url_names_the_gov_gateway(monkeypatch, tmp_path):
_, _, status = _gateway_run(
monkeypatch, tmp_path, "audit_records", lambda m, p: (401, SCOPE_401),
"https://example.atlassian-us-gov-mod.net",
)
assert status["code"] == "bad_config"
assert "https://api.atlassian-us-gov-mod.com/ex/jira/<cloudId>" in status["error"]


def test_scope_mismatch_through_the_gateway_is_a_missing_scope(monkeypatch, tmp_path):
"""Through the gateway the URL is right; the token lacks a scope for one call."""
def router(method, path):
if path == "/rest/api/3/myself":
return 200, ME
return 401, SCOPE_401

code, _, status = _gateway_run(
monkeypatch, tmp_path, "audit_records", router,
"https://api.atlassian.com/ex/jira/eadac007-8666-4024-a026-0bfb4d34aa2e",
)
assert code == 1 and status["code"] == "not_authorized"
assert "lacks a scope" in status["error"] and "tenant_info" not in status["error"]


def test_scope_mismatch_on_first_gateway_call_is_not_a_bad_url(monkeypatch, tmp_path):
code, calls, status = _gateway_run(
monkeypatch, tmp_path, "remediation_items", lambda m, p: (401, SCOPE_401),
"https://api.atlassian.com/ex/jira/eadac007-8666-4024-a026-0bfb4d34aa2e",
)
assert code == 1 and len(calls) == 1
assert status["code"] == "not_authorized" and "lacks a scope" in status["error"]


def test_plain_401_is_still_auth_failed(monkeypatch, tmp_path):
_, _, status = _gateway_run(
monkeypatch, tmp_path, "audit_records",
lambda m, p: (401, {"errorMessages": ["Client must be authenticated"]}),
"https://example.atlassian.net",
)
assert status["code"] == "auth_failed" and "gateway" not in status["error"]


def test_redirect_is_not_followed_and_fails(run):
def router(method, path, params, body):
if path == "/rest/api/3/myself":
Expand Down
Loading