diff --git a/docs/sources.md b/docs/sources.md index 6b50d6083..4281125e6 100644 --- a/docs/sources.md +++ b/docs/sources.md @@ -148,8 +148,16 @@ visible rather than failing the fetch. - **First run:** Fetches from the last 24 hours - **Subsequent runs:** Uses `since=` (inclusive: the cursor's own second is listed again, and an unchanged re-listed thread is skipped by storage) with `Z` suffix (not `+00:00` — the `+` in a URL query string is decoded as a space, breaking the filter) - **Filter:** `sync.github.inbox` decides which threads are listed. `participating` (the default) sends `participating=true`: only the threads the account takes part in (assigned, review requested, mentioned, its own, commented on). `all` leaves that parameter out, so GitHub lists the account's whole inbox as its GitHub watch settings produce it: the participating threads plus `subscribed` threads of watched repositories (new issues and pull requests, per the repository's watch or Custom watch setting), `ci_activity` for the account's own workflow runs, and whatever else GitHub puts in the inbox; the guardrails pick within it (see [GitHub reason guardrail](#github-reason-guardrail)). Both modes send `all=true` (read threads too). GitHub marks a thread read as soon as a mail client renders the notification email with its images, which can happen seconds after the notification, so a listing of unread threads alone can miss a thread before anyone has seen it; a re-listed thread whose stored record is unchanged is skipped by storage, and the record carries no read state. A thread that GitHub lists as read and that was last read more than a minute before its update is left out: GitHub moves the thread of the account's own merge or review and leaves it read. Note that in participating mode it never returns `ci_activity` — workflow-run notifications are not "participating", so CI never reaches the inbox through this source in participating mode (verified against a live account: 25 `ci_activity` listed with `all=true` alone, 0 once `participating=true` is added) -- **Enrichment:** Each notification is enriched with actual content from the subject (PR/issue body, state, assignees, labels) and the latest comment, fetched in parallel (up to 5 concurrent `gh api` calls) -- **Metadata:** `reason`, `repo_name`, `actors` (every login involved) and `ci_branch` (branch of a CheckSuite run, `""` otherwise) — all four are guardrail-filterable, see [Guardrails](#guardrails-inbox-filtering) +- **Enrichment:** Each notification is enriched with actual content from the subject (PR/issue body, state, assignees, labels) and the latest comment, fetched in parallel (up to 5 concurrent `gh api` calls). A `mention`, `assign`, `review_requested` or `team_mention` thread also gets the human comments posted since the readers of the inbox last read a record of the thread, newest first (the latest comment is often a bot's, and a human comment before it would be lost). They are listed with `since`: the listing of one thread's comments takes no sort order and returns the oldest first +- **Mentions:** GitHub's `reason` belongs to the thread, not to the update: once the account was mentioned anywhere in a thread, every later notification of that thread says `mention` (GitHub: "The reason remains as mention, regardless of whether you're ever mentioned again"). For an issue or pull request with that reason, the source reads what was posted since the readers of the inbox last read a record of the thread, and says in the record whether that mentions the account: + - `mention: new`: it does. With `mention_at`, and with `mention_by` and `mention_url` when the source read the text; the text is quoted in the content under `--- Mention ---`, also when a bot wrote it + - `mention: old`: it does not, and an earlier mention (`mention_at`) explains GitHub's reason. The summary line ends with `(update, no new mention)` instead of `(mention)` + - no `mention` key: not known, and the record says `mention` as GitHub does. That is the case when the login could not be read, when a read failed, and when no mention of the account was found at all. A record is moved away from `mention` on evidence only + + `reason` stays `mention` in all three cases, so the [reason guardrail](#github-reason-guardrail) works as before. The description and the comments are checked against GitHub's own mention events of the thread (one GraphQL call), which also date a mention that an edit added. A pull request's reviews and inline review comments are checked by their text, as a whole `@login` outside Markdown code: GitHub writes no mention event for them, though it notifies. The login is the account `gh` acts as, read in every run that has such a thread and never kept between runs, so another account or a renamed one is used from the next run on. Not checked: `team_mention`, and subjects other than issues and pull requests + + **New until read.** Storage keeps one record per thread and replaces it when the thread moves, and a reader comes on its own schedule (by default the inbox job every 30 minutes, this source every 15). So "new" is counted from the last record of the thread that every live consumer of the source has read: a mention that one record says is `new` is still `new`, with its text, in the record that replaces it, and it becomes `old` only in a record that follows one a reader has read. A consumer that stopped reading counts until its cursor expires (`sync.consumer_cursor_ttl_days`); until then a mention can be `new` again in later records of the thread, with the same `mention_at`. Where each look ended is remembered in the process. After a restart a thread is checked for the time since the sync cursor, and no later than a mention that an unread record says is `new` +- **Metadata:** `reason`, `repo_name`, `actors` (every login involved) and `ci_branch` (branch of a CheckSuite run, `""` otherwise) — all four are guardrail-filterable, see [Guardrails](#guardrails-inbox-filtering). A `mention` thread can also carry `mention`, `mention_at`, `mention_by` and `mention_url` (see Mentions above). The summary line and the metadata reach a reader of the inbox as they are, also when the content was condensed - **Default schedule:** `*/15 * * * *` (every 15 min) - **Hosted mode:** with `sync.github.mode: hosted` the channel gateway polls GitHub instead; see below @@ -289,7 +297,7 @@ on the `github_pr_activity` gate. - **First run:** Fetches the latest batch to establish a cursor baseline — no backfill of historical events - **Subsequent runs:** Fetches newest events, stops at the last-seen event ID - **Purpose:** Captures the user's **own** GitHub activity (pushes, PR creates/merges, reviews, comments, branch operations). Complements the notification source which only shows what *others* did involving you -- **Username:** Auto-detected from `gh auth` on first call, or set manually via `username` config +- **Username:** Set manually via `username` config, or else the account `gh` is signed in as, read in every run (a failed read falls back to the last login it returned) - **Repo filter:** Optional `repos` list — empty means all repos - **Event types:** PushEvent, PullRequestEvent, IssueCommentEvent, IssuesEvent, PullRequestReviewEvent, CreateEvent, DeleteEvent, ForkEvent (with type-specific formatting); other event types get generic formatting - **Note:** GitHub's Events API returns truncated payloads (e.g., PR titles and URLs may be missing). The source constructs URLs from repo name + number and handles missing fields gracefully @@ -771,6 +779,12 @@ thread), `mention`, `team_mention`, `assign`, `review_requested`, `comment` (you on it once), `subscribed` (you only watch it), `manual` (you subscribed by hand), `ci_activity`, `state_change`, `security_alert`. +A reason describes the thread, not the update, and it can outlive its cause: GitHub keeps +`mention` on a thread after the first mention, so a later comment that mentions nobody +arrives as `mention` too. The guardrail filters on the value as GitHub gives it. Whether +the update itself mentions the account is in the `mention` metadata key (`new` or `old`, +see Mentions under [GitHub](#github)). + ```yaml sync: github: diff --git a/nerve/db/sources.py b/nerve/db/sources.py index 66cdaab95..8bdc4cfc9 100644 --- a/nerve/db/sources.py +++ b/nerve/db/sources.py @@ -453,6 +453,36 @@ async def list_consumer_cursors(self, consumer: str | None = None) -> list[dict] ) as cursor: return [dict(row) async for row in cursor] + async def unread_source_message_metadata( + self, source: str, ids: list[str], + ) -> dict[str, dict]: + """Metadata of the stored records among *ids* that a consumer has yet to read. + + A record is unread while a live consumer cursor of *source* sits below + its rowid. An expired cursor does not count: its consumer starts again + at the source's newest message and never reads the record. Left out + are a record that every live consumer has read, a record of a source + that no consumer reads, and an id with no stored record. Read-only. + """ + unread: dict[str, dict] = {} + now = datetime.now(UTC).isoformat() + for first in range(0, len(ids), 500): + chunk = ids[first:first + 500] + marks = ", ".join("?" for _ in chunk) + async with self.db.execute( + f"""SELECT sm.id, sm.metadata FROM source_messages sm + WHERE sm.source = ? AND sm.id IN ({marks}) + AND EXISTS ( + SELECT 1 FROM consumer_cursors cc + WHERE cc.source = sm.source + AND (cc.expires_at IS NULL OR cc.expires_at > ?) + AND cc.cursor_seq < sm.rowid)""", + (source, *chunk, now), + ) as cursor: + async for row in cursor: + unread[row[0]] = json.loads(row[1]) if row[1] else {} + return unread + async def consumer_has_unread(self, consumer: str) -> bool: """True if any source this consumer tracks has messages past its cursor. diff --git a/nerve/sources/github.py b/nerve/sources/github.py index 3f221f446..65b6b9c1d 100644 --- a/nerve/sources/github.py +++ b/nerve/sources/github.py @@ -14,6 +14,14 @@ the latest comment that triggered the notification, and for PRs the latest review state (APPROVED, CHANGES_REQUESTED, etc.) plus any inline review comments attached to specific code lines. + +GitHub's ``reason`` belongs to the thread, not to the update: once the account +was mentioned anywhere in a thread, every later notification of that thread +says ``mention`` ("The reason remains as mention, regardless of whether you're +ever mentioned again", GitHub's REST documentation). So the record of a +``mention`` thread also says whether the update itself mentions the account +(the ``mention`` metadata key, ``new`` or ``old``; see +:meth:`GitHubSource._check_mention`). """ from __future__ import annotations @@ -22,6 +30,9 @@ import json import logging import re +from collections import OrderedDict +from collections.abc import Awaitable, Callable +from dataclasses import dataclass, field from datetime import datetime, timedelta, timezone from typing import TYPE_CHECKING, Any @@ -54,6 +65,132 @@ # are the reviewers and commenters. PR_ACTIVITY_RECORD_TYPE = "github_pr_activity" +# Reasons whose record also carries the comments posted since the readers of +# the inbox last read a record of the thread (see GitHubSource._look_start). +_NEW_COMMENT_REASONS = ("mention", "assign", "review_requested", "team_mention") + +# How far before the sync cursor the first look at a thread starts. GitHub +# stamps a thread's ``updated_at`` some twenty seconds after the comment that +# moved it (23 and 24 seconds on two measured threads), so the comment behind +# a notification that is new in this run can be older than the cursor. +_FIRST_LOOK_MARGIN = timedelta(minutes=2) + +# Threads whose last look is remembered; the least recently looked at is +# forgotten first. +_REMEMBERED_THREADS = 2048 + +# Pages of 100 that one read of a comments listing follows. +_COMMENT_PAGES = 5 + +# Pages of 100 of a thread's mention log read, newest first, to find a mention +# of the account. +_MENTION_LOG_PAGES = 3 + +# GitHub writes the mention event of a text 1 to 3 seconds after the text +# (measured on 29 mentions). A text this close before an event is the text +# the event was written for. +_EVENT_LAG = timedelta(seconds=15) + +# Markdown code. GitHub notifies nobody for a handle written inside it. +_CODE_RE = re.compile(r"```.*?```|~~~.*?~~~|`[^`\n]*`", re.DOTALL) + +# https://api.github.com/repos/{owner}/{repo}/issues|pulls/{number} +_SUBJECT_URL_RE = re.compile( + r"/repos/(?P[^/]+)/(?P[^/]+)/(?:issues|pulls)/(?P\d+)$" +) + +# The mention events of a thread, newest last, and for a pull request its +# latest reviews. GitHub writes a ``MentionedEvent`` (``actor`` is the account +# that WAS mentioned) for a mention in the description or in a comment, also +# when an edit adds it. It writes none for a mention in a review or in an +# inline review comment, so those texts are read instead. +_MENTION_LOG_QUERY = """ +query($owner: String!, $repo: String!, $number: Int!, $before: String) { + repository(owner: $owner, name: $repo) { + issueOrPullRequest(number: $number) { + ... on Issue { + timelineItems(last: 100, before: $before, itemTypes: [MENTIONED_EVENT]) { + pageInfo { hasPreviousPage startCursor } + nodes { ... on MentionedEvent { createdAt actor { login } } } + } + } + ... on PullRequest { + timelineItems(last: 100, before: $before, itemTypes: [MENTIONED_EVENT]) { + pageInfo { hasPreviousPage startCursor } + nodes { ... on MentionedEvent { createdAt actor { login } } } + } + reviews(last: 50) { nodes { author { login } body submittedAt url } } + } + } + } +} +""" + + +def _when(value: Any) -> datetime | None: + """A GitHub timestamp as an aware datetime, or None when it is not one.""" + if not isinstance(value, str) or not value: + return None + try: + parsed = datetime.fromisoformat(value.replace("Z", "+00:00")) + except ValueError: + return None + return parsed if parsed.tzinfo else parsed.replace(tzinfo=timezone.utc) + + +def _stamp(moment: datetime) -> str: + """*moment* the way GitHub writes a timestamp (UTC, ``Z`` suffix).""" + return moment.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + +def _mention_re(login: str) -> re.Pattern[str]: + """Matches ``@login`` as a whole handle, in any letter case. + + Not a longer handle (``@login-bot``, ``@loginx``), not a team of an + organization with that name (``@login/team``) and not the end of an email + address (``someone@login``). + """ + return re.compile( + rf"(? bool: + """Whether *text* mentions the handle of *pattern* outside Markdown code.""" + if not isinstance(text, str) or not text: + return False + return bool(pattern.search(_CODE_RE.sub(" ", text))) + + +def _comment_time(comment: dict[str, Any]) -> datetime | None: + """When a comment was last written: its edit, or else its creation.""" + return _when(comment.get("updated_at")) or _when(comment.get("created_at")) + + +@dataclass +class _Run: + """What one fetch run shares with the enrichment of its notifications.""" + + # Login of the account gh acts as in this run; "" when it was not needed + # or could not be read. + login: str = "" + # Where the look at a thread starts that the source has no memory of. + first_look: datetime | None = None + # Metadata of the stored records, by thread id, that a reader of the inbox + # has yet to read (see GitHubSource._look_start). None when that could + # not be read, and every thread then counts as unread. + unread: dict[str, dict[str, Any]] | None = field(default_factory=dict) + + +@dataclass +class _Look: + """The source's last look at one thread (see GitHubSource._look_start).""" + + updated_at: str # the thread's ``updated_at`` at that look + start: datetime # what came after this moment was new for that look + end: datetime # the newest moment that look covered + def _ci_branch(subject_type: str, title: str) -> str: """Branch of a CI run, or ``""`` for anything else. @@ -77,6 +214,7 @@ def _collect_actors( inline_comments: list[dict[str, Any]], recent_comments: list[dict[str, Any]], assigner: str = "", + mentioner: str = "", ) -> list[str]: """Every GitHub login involved in a notification, de-duplicated. @@ -96,8 +234,13 @@ def _collect_actors( hands it back), every other candidate below is that same account. Without this, the notification has no third-party login at all and a non-empty ``allow_actors`` allowlist can never pass it. + + ``mentioner`` is the login whose text mentions the account in this update + (see :meth:`GitHubSource._check_mention`). The record quotes that text, and + its author can be absent from every other candidate: a bot is left out of + the new comments, and a mention in a review comment is no comment at all. """ - candidates: list[str] = [subject_user, *assignees, assigner] + candidates: list[str] = [subject_user, *assignees, assigner, mentioner] if comment: candidates.append(comment.get("user", "")) if latest_review: @@ -184,12 +327,21 @@ def notification_record(notif: dict[str, Any], extra: dict[str, Any] | BaseExcep labels = extra.get("labels", []) comment = extra.get("latest_comment") assigner = extra.get("assigner", "") + # Whether this update mentions the account: {} when that was not checked, + # so the record says what GitHub says and no more. + mention = extra.get("mention") or {} + mention_state = mention.get("state", "") + mention_at = mention.get("at", "") + mention_by = mention.get("by", "") + mention_url = mention.get("url", "") + mention_key = (mention_by, mention.get("created_at", "")) content_parts = [ f"Repository: {repo_name}", f"Type: {subject_type}", f"Title: {subject_title}", f"Reason: {reason}", + _mention_line(mention), f"State: {subject_state}" if subject_state else None, f"Author: {subject_user}" if subject_user else None, f"Assignees: {', '.join(assignees)}" if assignees else None, @@ -218,6 +370,17 @@ def notification_record(notif: dict[str, Any], extra: dict[str, Any] | BaseExcep f"{comment_body}" ) + # Add the text that mentions the account, unless it is the latest comment + # above. Its author can be a bot, which the new comments below leave out, + # and it can be a review comment, which is no comment of the thread. + mention_body = mention.get("body", "") + latest_key = (comment.get("user", ""), comment.get("created_at", "")) if comment else None + if mention_state == "new" and mention_body and mention_key != latest_key: + content_parts.append( + f"\n--- Mention ({mention_by or '?'}, {mention_at}) ---\n" + f"{mention_body[:_MAX_COMMENT_CHARS]}" + ) + # Add latest review state (APPROVED, CHANGES_REQUESTED, etc.) latest_review = extra.get("latest_review") if latest_review: @@ -247,14 +410,19 @@ def notification_record(notif: dict[str, Any], extra: dict[str, Any] | BaseExcep f"\n{ic_user} on {loc}:\n{ic_body}" ) - # Add recent human comments (catches mentions buried by - # bot comments — see _enrich_recent_comments). + # Add the human comments posted since the readers last read a record of + # the thread (catches a comment buried by bot comments — see + # _enrich_since_last_read). The mention text is already above. recent_comments = extra.get("recent_comments", []) - if recent_comments: + shown_comments = [ + rc for rc in recent_comments + if (rc.get("user", ""), rc.get("created_at", "")) != mention_key + ] + if shown_comments: content_parts.append( - "\n--- Recent Comments (human, newest first) ---" + "\n--- New Comments (human, newest first) ---" ) - for rc in recent_comments: + for rc in shown_comments: rc_user = rc.get("user", "?") rc_date = rc.get("created_at", "") rc_body = rc.get("body", "") @@ -267,38 +435,88 @@ def notification_record(notif: dict[str, Any], extra: dict[str, Any] | BaseExcep actors = _collect_actors( subject_user, assignees, comment, latest_review, inline_comments, recent_comments, assigner, + mention_by if mention_state == "new" else "", ) + metadata: dict[str, Any] = { + "reason": reason, + "subject_type": subject_type, + "subject_url": html_url, + "repo_name": repo_name, + "repo_url": repo.get("html_url", ""), + "actors": actors, + "ci_branch": _ci_branch(subject_type, subject_title), + } + # The summary line and the metadata reach a reader of the inbox as they + # are, also when the content was condensed, so the answer is in both. + if mention_state: + metadata["mention"] = mention_state + if mention_at: + metadata["mention_at"] = mention_at + if mention_state == "new": + if mention_by: + metadata["mention_by"] = mention_by + if mention_url: + metadata["mention_url"] = mention_url + label = "update, no new mention" if mention_state == "old" else reason + return SourceRecord( id=notif.get("id", ""), source="github", record_type="github_notification", - summary=f"[{repo_name}] {subject_title} ({reason})", + summary=f"[{repo_name}] {subject_title} ({label})", content="\n".join(p for p in content_parts if p), timestamp=updated_at or datetime.now(timezone.utc).isoformat(), - metadata={ - "reason": reason, - "subject_type": subject_type, - "subject_url": html_url, - "repo_name": repo_name, - "repo_url": repo.get("html_url", ""), - "actors": actors, - "ci_branch": _ci_branch(subject_type, subject_title), - }, + metadata=metadata, ) +def _mention_line(mention: dict[str, Any]) -> str | None: + """The content line that says whether the update mentions the account.""" + state = mention.get("state") + at = mention.get("at", "") + if state == "new": + who = f" by {mention['by']}" if mention.get("by") else "" + where = f", {mention['url']}" if mention.get("url") else "" + return f"Mention: new{who} ({at}{where})" + if state == "old": + return ( + "Mention: none in this update. GitHub keeps the reason of a thread " + f"at mention after an earlier one (the last was at {at})" + ) + return None + + class GitHubSource(Source): """GitHub notification source using the gh CLI. ``config["inbox"]`` is ``sync.github.inbox``: ``"all"`` lists the whole inbox; anything else, or no config, the participating threads only. + + ``unread`` tells, for thread ids, which stored records a reader of the + inbox has yet to read, with their metadata + (:meth:`nerve.db.sources.SourceStore.unread_source_message_metadata`). + Without it nothing is known about readers and every record counts as read + (see :meth:`_look_start`). """ source_name = "github" - def __init__(self, config: dict[str, Any] | None = None): + def __init__( + self, + config: dict[str, Any] | None = None, + unread: Callable[[list[str]], Awaitable[dict[str, dict[str, Any]]]] | None = None, + ): self._config = config or {} + self._unread = unread + # Set by every fetch run for the enrichment of its notifications. + self._run = _Run() + # Last look at each thread, by notification thread id. Kept in this + # process only: after a restart the first look at a thread starts + # shortly before the sync cursor again (see _look_start). + self._looks: OrderedDict[str, _Look] = OrderedDict() + # The account the looks above belong to. + self._looks_login = "" async def fetch(self, cursor: str | None, limit: int = 100) -> FetchResult: """Fetch new notifications since cursor (ISO 8601 timestamp). @@ -310,6 +528,7 @@ async def fetch(self, cursor: str | None, limit: int = 100) -> FetchResult: # IMPORTANT: use Z suffix, not +00:00 — the `+` in a URL query # string is interpreted as a space, silently breaking the filter. since = None + since_dt: datetime | None = None if cursor: try: if cursor.isdigit(): @@ -318,6 +537,7 @@ async def fetch(self, cursor: str | None, limit: int = 100) -> FetchResult: raise ValueError(cursor) cursor_dt = datetime.fromisoformat(cursor.replace("Z", "+00:00")) since = cursor_dt.strftime("%Y-%m-%dT%H:%M:%SZ") + since_dt = _when(since) except ValueError: # Not a timestamp, for example the buffer position that the # hosted source keeps under the same name. GitHub refuses such a @@ -326,8 +546,10 @@ async def fetch(self, cursor: str | None, limit: int = 100) -> FetchResult: logger.warning( "GitHub cursor %r is not a timestamp; fetching the last 24 hours", cursor, ) - if since is None: - since = (datetime.now(timezone.utc) - timedelta(hours=24)).strftime("%Y-%m-%dT%H:%M:%SZ") + if since is None or since_dt is None: + since_dt = datetime.now(timezone.utc) - timedelta(hours=24) + since = since_dt.strftime("%Y-%m-%dT%H:%M:%SZ") + self._run = _Run(first_look=since_dt - _FIRST_LOOK_MARGIN) try: # Query params go in the URL — -f flags would force POST @@ -374,6 +596,42 @@ async def fetch(self, cursor: str | None, limit: int = 100) -> FetchResult: if not self._own_update(n) ] + # Which stored records a reader has yet to read (see _look_start). + # A state that cannot be read counts as unread for every thread: + # that can repeat a mention, and cannot lose one. + if self._unread is not None: + looked_at = [ + str(n.get("id", "")) for n in notifications + if n.get("reason") in _NEW_COMMENT_REASONS + ] + if looked_at: + try: + self._run.unread = await self._unread(looked_at) + except Exception as e: + logger.warning("Read state of GitHub records is not known: %s", e) + self._run.unread = None + + # Whose mentions to look for. Read in every run that has a mention + # thread and never kept between runs, so the account that gh acts + # as is always the one whose notifications were just listed: a + # different account, or a renamed one, is used from the next run + # on. Without a login no mention is checked in this run. + if any(self._has_mention_reason(n) for n in notifications): + self._run.login = await self._gh_login() + if not self._run.login: + logger.warning( + "GitHub login of gh could not be read; " + "mentions are not checked in this run", + ) + elif self._run.login.lower() != self._looks_login.lower(): + if self._looks_login: + logger.info( + "gh acts as %s now, was %s; forgetting %d thread looks", + self._run.login, self._looks_login, len(self._looks), + ) + self._looks.clear() + self._looks_login = self._run.login + # Enrich notifications with actual content in parallel sem = asyncio.Semaphore(_MAX_CONCURRENT_FETCHES) enrich_tasks = [ @@ -484,14 +742,18 @@ async def _enrich_notification( if subject.get("type") == "PullRequest" and subject_url: await self._enrich_pr_reviews(subject_url, result) - # Fetch recent human comments to catch mentions buried by bot - # comments. GitHub's latest_comment_url always points to the - # most recent comment — when a bot (e.g. some-ci[bot]) - # posts after a human @mention, the mention is lost. + # Read what was posted since the readers last read a record of the + # thread (see _look_start): the + # human comments (GitHub's latest_comment_url always points to the + # most recent comment — when a bot, e.g. some-ci[bot], posts + # after a human comment, that comment is lost) and, for a mention + # thread, whether any of it mentions the account. reason = notif.get("reason", "") s_type = subject.get("type", "") - if reason in ("mention", "assign", "review_requested", "team_mention") and s_type in ("PullRequest", "Issue"): - await self._enrich_recent_comments(subject_url, s_type, result) + if reason in _NEW_COMMENT_REASONS and s_type in ("PullRequest", "Issue"): + await self._enrich_since_last_read( + notif, subject_data if isinstance(subject_data, dict) else {}, result, + ) # For assignments: fetch who assigned it. The notification payload # names the assignee but never the assigner, and an assignment has @@ -617,20 +879,130 @@ async def _enrich_pr_reviews( for rc in rc_data[:5] # cap to keep payload reasonable ] - async def _enrich_recent_comments( - self, subject_url: str, subject_type: str, result: dict[str, Any], + async def _enrich_since_last_read( + self, notif: dict, subject_data: dict[str, Any], result: dict[str, Any], ) -> None: - """Fetch recent human comments for a PR/issue notification. + """Add what was posted since the readers last read a record of the thread. + + :meth:`_look_start` says which moment that is. + + Mutates *result* in place: ``recent_comments`` (the human comments + posted since then, newest first, without the already-fetched + ``latest_comment``) and, for a ``mention`` thread whose answer is + known, ``mention`` (see :meth:`_check_mention`). GitHub's ``latest_comment_url`` points to the *most recent* comment, which is often a bot (CI coverage, AI review, merge conflict warning). - When a human @mentions us and then a bot comments seconds later, the - mention is buried and never appears in the enriched content. + When a human writes and a bot comments seconds later, the human + comment is buried and never appears in the enriched content. + """ + subject = notif.get("subject", {}) + subject_url = subject.get("url", "") + subject_type = subject.get("type", "") + thread_id = str(notif.get("id", "")) + updated_at = str(notif.get("updated_at", "")) + + start = self._look_start(thread_id, updated_at) + comments = await self._comments_since(subject_url, subject_type, start) + complete = comments is not None + covered: list[datetime | None] = [_when(updated_at)] + covered += [_comment_time(c) for c in comments or []] + self._attach_new_comments(comments or [], result) + + if notif.get("reason") == "mention": + checked, seen = await self._check_mention( + subject_url, subject_type, subject_data, comments or [], start, result, + ) + complete = complete and checked + covered += seen + + self._remember_look(thread_id, updated_at, start, covered if complete else []) + + def _look_start(self, thread_id: str, updated_at: str) -> datetime: + """The moment after which a comment or a mention is new for this update. + + It is the end of the last look whose record every reader of the inbox + has read. Storage keeps one record per thread and replaces it when the + thread moves, and a reader comes on its own schedule (by default the + inbox job every 30 minutes, this source every 15). So while a reader + has yet to read the stored record, the look starts where the look of + that record started: a mention that one record says is new is still + new in the record that replaces it, with its text, and is not lost to + a reader that only ever sees the later one. + + A thread that is listed again with the same ``updated_at`` gets the + start that its look had the first time: its record comes out the same + and storage skips it (the cursor's own second is listed again in every + run). A thread the source has no look of, which is every thread after + a restart, starts shortly before the sync cursor (what was posted + since the last run), and no later than the mention that an unread + stored record says is new. + """ + look = self._looks.get(thread_id) + if look is not None and look.updated_at == updated_at: + return look.start + unread = self._unread_record(thread_id) + if look is not None: + return look.end if unread is None else look.start + start = self._run.first_look + if start is None: + # No run set a start: the enrichment was called on its own. + start = (_when(updated_at) or datetime.now(timezone.utc)) - _FIRST_LOOK_MARGIN + if unread and unread.get("mention") == "new": + mentioned = _when(unread.get("mention_at")) + if mentioned is not None: + start = min(start, mentioned - _EVENT_LAG) + return start + + def _unread_record(self, thread_id: str) -> dict[str, Any] | None: + """Metadata of the thread's stored record while a reader has yet to read it. + + None when every reader has read the record, and when there is no + record or no reader. ``{}`` when the read state is not known: the + record then counts as unread. + """ + if self._run.unread is None: + return {} + return self._run.unread.get(thread_id) + + def _remember_look( + self, + thread_id: str, + updated_at: str, + start: datetime, + covered: list[datetime | None], + ) -> None: + """Keep the look at a thread for the thread's next update. - This method fetches the last few issue-level comments, filters out - bots, and attaches human comments that are missing from the - already-fetched ``latest_comment``. It ensures the actual triggering - comment is always available to downstream consumers. + *covered* holds the moments of everything this look read. It is empty + when a read failed, and the look of the next update then repeats the + range instead of skipping what this one could not see. + """ + if not thread_id: + return + previous = self._looks.get(thread_id) + same_update = previous is not None and previous.updated_at == updated_at + end = previous.end if previous is not None and same_update else start + for moment in covered: + if moment is not None and moment > end: + end = moment + self._looks[thread_id] = _Look(updated_at=updated_at, start=start, end=end) + self._looks.move_to_end(thread_id) + while len(self._looks) > _REMEMBERED_THREADS: + self._looks.popitem(last=False) + + async def _comments_since( + self, subject_url: str, subject_type: str, start: datetime, + ) -> list[dict[str, Any]] | None: + """The thread's comments created or edited after *start*, oldest first. + + None when a read failed, or when the listing is longer than the pages + followed: nothing is concluded from a part of it. + + The listing of one thread's comments takes ``since`` but no sort + order, and returns the OLDEST first (measured: with + ``sort=created&direction=desc`` it returned the ten oldest comments + of a long thread). """ # PR issue-comments live under /issues/{n}/comments, not /pulls/ if subject_type == "PullRequest": @@ -638,21 +1010,36 @@ async def _enrich_recent_comments( else: comments_url = subject_url + "/comments" - # Fetch last 10 comments (newest first) — enough to find the mention - # even if several bots posted afterwards. - comments_data = await self._gh_api_get( - f"{comments_url}?per_page=10&sort=created&direction=desc", - ) - if not isinstance(comments_data, list) or not comments_data: - return + listed: list[dict[str, Any]] = [] + for page in range(1, _COMMENT_PAGES + 1): + page_arg = f"&page={page}" if page > 1 else "" + data = await self._gh_api_get( + f"{comments_url}?since={_stamp(start)}&per_page=100{page_arg}", + ) + if not isinstance(data, list): + return None + listed.extend(c for c in data if isinstance(c, dict)) + if len(data) < 100: + break + else: + return None + return [c for c in listed if (_comment_time(c) or start) > start] + @staticmethod + def _attach_new_comments( + comments: list[dict[str, Any]], result: dict[str, Any], + ) -> None: + """Set ``recent_comments``: the human ones of *comments*, newest first.""" # Deduplicate against the latest_comment already in result existing = result.get("latest_comment") or {} existing_key = (existing.get("user", ""), existing.get("created_at", "")) - recent_human: list[dict[str, str]] = [] - for c in comments_data: - user = c.get("user", {}).get("login", "") + humans: list[dict[str, str]] = [] + newest_first = sorted( + comments, key=lambda c: str(c.get("created_at") or ""), reverse=True, + ) + for c in newest_first: + user = (c.get("user") or {}).get("login", "") body = (c.get("body", "") or "") created = c.get("created_at", "") @@ -664,18 +1051,274 @@ async def _enrich_recent_comments( if (user, created) == existing_key: continue - recent_human.append({ + humans.append({ "user": user, "body": body[:_MAX_COMMENT_CHARS], "created_at": created, }) # Cap at 5 human comments — enough context without bloat - if len(recent_human) >= 5: + if len(humans) >= 5: + break + + if humans: + result["recent_comments"] = humans + + async def _check_mention( + self, + subject_url: str, + subject_type: str, + subject_data: dict[str, Any], + comments: list[dict[str, Any]], + start: datetime, + result: dict[str, Any], + ) -> tuple[bool, list[datetime | None]]: + """Say whether the update of a ``mention`` thread mentions the account. + + Sets ``result["mention"]`` when the answer is known: + + * ``new``: the account is mentioned in something posted after *start* + (see :meth:`_look_start`), with who wrote it and where when + this look read that text. + * ``old``: nothing posted after *start* mentions the account, and an + earlier mention explains GitHub's reason. + + Left unset otherwise, and the record then says ``mention`` as GitHub + does: when the login is not known, when a read failed, and when no + mention of the account was found at all (GitHub knows of one that + this look cannot see). A record is moved away from ``mention`` on + evidence only. + + Where a mention is looked for, one place per kind of text: + + * The description and the comments: GitHub's own mention events of + the thread (:meth:`_mention_log`). They are GitHub's decision of + what a mention is, and they date a mention that an edit added. + * A pull request's reviews and inline review comments: their texts. + GitHub writes no mention event for them (measured: 0 of 6 mentions, + against 30 of 30 in comments), though it does notify. + + Returns whether every read succeeded and reached back to *start*, and + the moments of what was read. + """ + login = self._run.login + if not login: + return False, [] + pattern = _mention_re(login) + complete = True + seen: list[datetime | None] = [] + # The mentions of the account: {"at"} for a mention event, and with + # by / url / body / created_at for a review text that holds one. + found: list[dict[str, Any]] = [] + reviews: list[dict[str, Any]] = [] + + log = await self._mention_log(subject_url, login) + if log is None: + complete = False + else: + mine, reviews, event_times = log + seen += event_times + found += [{"at": at} for at in mine] + + if subject_type == "PullRequest": + review_times: list[datetime] = [] + for review in reviews: + at = _when(review.get("submittedAt")) + if at is None: # a pending review of the account itself + continue + review_times.append(at) + if _mentions(pattern, review.get("body")): + found.append({ + "at": at, + "by": (review.get("author") or {}).get("login", ""), + "url": review.get("url", ""), + "body": review.get("body", ""), + "created_at": review.get("submittedAt", ""), + }) + seen += review_times + if len(reviews) >= 50 and review_times and min(review_times) > start: + complete = False # more reviews since *start* than were read + + # Newest edit first, so an edit that adds a mention is read too. + inline = await self._gh_api_get( + f"{subject_url}/comments?sort=updated&direction=desc&per_page=100", + ) + if not isinstance(inline, list): + complete = False + else: + inline_times: list[datetime] = [] + for c in inline: + at = _comment_time(c) if isinstance(c, dict) else None + if at is None: + continue + inline_times.append(at) + if _mentions(pattern, c.get("body")): + found.append({ + "at": at, + "by": (c.get("user") or {}).get("login", ""), + "url": c.get("html_url", ""), + "body": c.get("body", ""), + "created_at": c.get("created_at", ""), + }) + seen += inline_times + if len(inline) >= 100 and inline_times and min(inline_times) > start: + complete = False # more inline comments since *start* + + new = [m for m in found if m["at"] > start] + if new: + newest = max(new, key=lambda m: m["at"]) + if "by" not in newest: + newest = self._mention_text( + pattern, newest["at"], comments, subject_data, + ) or newest + mention: dict[str, Any] = {"state": "new", "at": _stamp(newest["at"])} + for key in ("by", "url", "body", "created_at"): + if newest.get(key): + mention[key] = newest[key] + result["mention"] = mention + elif found and complete: + last = max(m["at"] for m in found) + result["mention"] = {"state": "old", "at": _stamp(last)} + return complete, seen + + @staticmethod + def _mention_text( + pattern: re.Pattern[str], + event_at: datetime, + comments: list[dict[str, Any]], + subject_data: dict[str, Any], + ) -> dict[str, Any] | None: + """The text that a mention event was written for, or None. + + The event carries neither author nor link. GitHub writes it 1 to 3 + seconds after the text (measured on 29 mentions), so the text is + the comment that mentions the account and was written, or edited, + shortly before the event; or the description, for a thread that was + opened then. + """ + for c in sorted(comments, key=lambda c: _comment_time(c) or event_at, reverse=True): + at = _comment_time(c) + if at is None or not timedelta(0) <= event_at - at <= _EVENT_LAG: + continue + if _mentions(pattern, c.get("body")): + return { + "at": at, + "by": (c.get("user") or {}).get("login", ""), + "url": c.get("html_url", ""), + "body": c.get("body", ""), + "created_at": c.get("created_at", ""), + } + opened = _when(subject_data.get("created_at")) + if ( + opened is not None + and timedelta(0) <= event_at - opened <= _EVENT_LAG + and _mentions(pattern, subject_data.get("body")) + ): + # No body: the record already carries the description. + return { + "at": opened, + "by": (subject_data.get("user") or {}).get("login", ""), + "url": subject_data.get("html_url", ""), + } + return None + + async def _mention_log( + self, subject_url: str, login: str, + ) -> tuple[list[datetime], list[dict[str, Any]], list[datetime | None]] | None: + """GitHub's mention events of a thread, and a pull request's reviews. + + Returns the moments at which *login* was mentioned, the latest reviews + (empty for an issue) and the moments of every event read; None when + the read failed. + + The events come newest last, 100 a page. Older pages are read only + until one holds a mention of *login*: one such mention is enough to + explain GitHub's reason, and a mention since the thread's previous + update is on the newest page. + """ + match = _SUBJECT_URL_RE.search(subject_url) + if match is None: + return None + variables: dict[str, Any] = { + "owner": match["owner"], + "repo": match["repo"], + "number": int(match["number"]), + } + mine: list[datetime] = [] + reviews: list[dict[str, Any]] | None = None + event_times: list[datetime | None] = [] + for _ in range(_MENTION_LOG_PAGES): + data = await self._gh_graphql(_MENTION_LOG_QUERY, variables) + thread = ((data or {}).get("repository") or {}).get("issueOrPullRequest") + if not isinstance(thread, dict): + return None + items = thread.get("timelineItems") + if not isinstance(items, dict): + return None + if reviews is None: + nodes = (thread.get("reviews") or {}).get("nodes") or [] + reviews = [r for r in nodes if isinstance(r, dict)] + for node in items.get("nodes") or []: + at = _when((node or {}).get("createdAt")) + if at is None: + continue + event_times.append(at) + actor = ((node or {}).get("actor") or {}).get("login", "") + if actor.lower() == login.lower(): + mine.append(at) + page = items.get("pageInfo") or {} + if mine or not page.get("hasPreviousPage") or not page.get("startCursor"): break + variables["before"] = page["startCursor"] + return mine, reviews or [], event_times + + async def _gh_login(self) -> str: + """Login of the account gh acts as now, or "" when it cannot be read.""" + user = await self._gh_api_get("user") + login = user.get("login") if isinstance(user, dict) else "" + return login if isinstance(login, str) else "" - if recent_human: - result["recent_comments"] = recent_human + @staticmethod + def _has_mention_reason(notif: dict) -> bool: + """Whether *notif* is of a ``mention`` thread whose mentions can be read.""" + subject = notif.get("subject") or {} + return ( + notif.get("reason") == "mention" + and subject.get("type") in ("PullRequest", "Issue") + and bool(subject.get("url")) + ) + + @staticmethod + async def _gh_graphql( + query: str, variables: dict[str, Any], timeout: float = 30, + ) -> dict | None: + """Run a GraphQL query with gh; return its ``data``, or None on failure. + + A response with ``errors`` is a failure, whatever data came with it. + """ + args = ["gh", "api", "graphql", "-f", f"query={query}"] + for name, value in variables.items(): + # -F reads an integer as a number. A string goes with -f, which + # takes it as it is: -F would read "@name" as a file to send. + args += ["-F" if isinstance(value, int) else "-f", f"{name}={value}"] + try: + proc = await asyncio.create_subprocess_exec( + *args, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=timeout) + if proc.returncode != 0: + logger.debug("gh api graphql failed: %s", stderr.decode()[:200]) + return None + payload = json.loads(stdout.decode()) + except Exception as e: + logger.debug("gh api graphql error: %s", e) + return None + if not isinstance(payload, dict) or payload.get("errors"): + return None + data = payload.get("data") + return data if isinstance(data, dict) else None @staticmethod async def _gh_api_get(url: str, timeout: float = 30) -> dict | list | None: diff --git a/nerve/sources/github_events.py b/nerve/sources/github_events.py index 39cf0e9d0..411be573f 100644 --- a/nerve/sources/github_events.py +++ b/nerve/sources/github_events.py @@ -41,9 +41,17 @@ def __init__(self, config: dict[str, Any] | None = None): # Lowercase for case-insensitive matching self._repos_lower = [r.lower() for r in self._repos] self._username: str = self._config.get("username", "") + # The login gh reported last; used only when a later lookup fails. + self._detected_username = "" async def _resolve_username(self) -> str: - """Resolve GitHub username, auto-detecting from gh auth if not configured.""" + """The configured GitHub username, or else the account gh acts as now. + + The account is looked up in every run and not kept as the answer: once + gh is signed in as another account, or the account is renamed, the + next run reads the events of the current one. A failed lookup falls + back to the login the last one returned. + """ if self._username: return self._username @@ -55,12 +63,16 @@ async def _resolve_username(self) -> str: ) stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=15) if proc.returncode == 0 and stdout.strip(): - self._username = stdout.decode().strip() - logger.info("Auto-detected GitHub username: %s", self._username) - return self._username + detected = stdout.decode().strip() + if detected != self._detected_username: + logger.info("Auto-detected GitHub username: %s", detected) + self._detected_username = detected + return detected except Exception as e: logger.error("Failed to detect GitHub username: %s", e) + if self._detected_username: + return self._detected_username raise RuntimeError("GitHub username not configured and auto-detection failed") async def fetch(self, cursor: str | None, limit: int = 100) -> FetchResult: diff --git a/nerve/sources/registry.py b/nerve/sources/registry.py index 55275e9d4..8f4d9c540 100644 --- a/nerve/sources/registry.py +++ b/nerve/sources/registry.py @@ -167,7 +167,14 @@ def build_source_runners( else: from nerve.sources.github import GitHubSource - source = GitHubSource(config={"inbox": gh.inbox}) + source = GitHubSource( + config={"inbox": gh.inbox}, + # A mention stays new in a thread's record until every reader + # of the inbox has read a record that says so. + unread=lambda ids: db.unread_source_message_metadata( + GitHubSource.source_name, ids, + ), + ) gh_filter = github_inbox_filter(gh) gh_runner = SourceRunner( source=source, diff --git a/tests/test_github_source_mentions.py b/tests/test_github_source_mentions.py new file mode 100644 index 000000000..ba31fad8d --- /dev/null +++ b/tests/test_github_source_mentions.py @@ -0,0 +1,1193 @@ +"""GitHub source: whether an update of a ``mention`` thread mentions the account. + +GitHub's ``reason`` belongs to the thread: once the account was mentioned in a +thread, every later notification of that thread says ``mention``. The source +reads what was posted since the record of the thread that the readers of the +inbox have read, and says in the record whether that mentions the account +(``mention``: ``new`` or ``old``). + +The fake below answers the calls the source makes for one thread: the REST +reads (subject, comments, inline review comments, the gh login) and the +GraphQL read of the mention events and the reviews. +""" + +from __future__ import annotations + +import asyncio +import json +from datetime import datetime, timedelta, timezone +from urllib.parse import parse_qs, urlsplit + +import pytest + +from nerve.config import NerveConfig +from nerve.sources import github as gh +from nerve.sources.github import GitHubSource, notification_record +from nerve.sources.github_events import GitHubEventsSource +from nerve.sources.registry import build_source_runners +from nerve.sources.runner import SourceRunner + +_API = "https://api.github.com/repos/acme/widgets" +_WEB = "https://github.com/acme/widgets" +_T0 = datetime(2026, 3, 2, 10, 0, 0, tzinfo=timezone.utc) + +# The account gh acts as, and the moment its previous look at the thread ended. +_ME = "alice" +_START = _T0 + + +def _ts(seconds: int) -> str: + return (_T0 + timedelta(seconds=seconds)).strftime("%Y-%m-%dT%H:%M:%SZ") + + +def _comment(cid: int, user: str, body: str, at: int, edited: int | None = None) -> dict: + return { + "id": cid, + "user": {"login": user}, + "body": body, + "created_at": _ts(at), + "updated_at": _ts(at if edited is None else edited), + "html_url": f"{_WEB}/issues/7#issuecomment-{cid}", + } + + +def _inline(cid: int, user: str, body: str, at: int) -> dict: + return { + "id": cid, + "user": {"login": user}, + "body": body, + "path": "src/widget.py", + "line": 3, + "created_at": _ts(at), + "updated_at": _ts(at), + "html_url": f"{_WEB}/pull/7#discussion_r{cid}", + } + + +def _review(user: str, body: str, at: int) -> dict: + return { + "author": {"login": user}, + "body": body, + "submittedAt": _ts(at), + "url": f"{_WEB}/pull/7#pullrequestreview-{at}", + } + + +def _notif( + kind: str = "Issue", + *, + reason: str = "mention", + updated_at: int = 60, + thread_id: str = "t1", +) -> dict: + path = "pulls" if kind == "PullRequest" else "issues" + return { + "id": thread_id, + "reason": reason, + "unread": True, + "updated_at": _ts(updated_at), + "subject": { + "title": "Widget breaks", + "type": kind, + "url": f"{_API}/{path}/7", + "latest_comment_url": None, + }, + "repository": {"full_name": "acme/widgets", "html_url": _WEB}, + } + + +class _FakeGitHub: + """One thread on GitHub, as the source reads it.""" + + def __init__(self, kind: str = "Issue"): + self.kind = kind + self.login: str | None = _ME + self.subject = { + "html_url": f"{_WEB}/{'pull' if kind == 'PullRequest' else 'issues'}/7", + "body": "It breaks.", + "state": "open", + "user": {"login": "carol"}, + "assignees": [], + "labels": [], + "created_at": _ts(-86_400), + } + self.comments: list[dict] = [] + self.inline: list[dict] | None = [] + # Mention events, oldest first: (seconds, login of the mentioned account). + self.events: list[tuple[int, str]] = [] + self.event_page_size = 100 + self.reviews: list[dict] = [] + self.graphql_fails = False + self.calls: list[str] = [] + + async def get(self, url: str, timeout: float = 30): + self.calls.append(url) + if url == "user": + return {"login": self.login} if self.login else None + parts = urlsplit(url) + query = parse_qs(parts.query) + if parts.path.endswith(("/issues/7", "/pulls/7")): + return self.subject + if parts.path.endswith("/issues/7/comments"): + since = query["since"][0] + page = int(query.get("page", ["1"])[0]) + listed = [c for c in self.comments if c["updated_at"] >= since] + return listed[(page - 1) * 100:page * 100] + if parts.path.endswith("/pulls/7/comments"): + if self.inline is None: + return None + assert query["sort"] == ["updated"] and query["direction"] == ["desc"] + return sorted(self.inline, key=lambda c: c["updated_at"], reverse=True)[:100] + if parts.path.endswith("/pulls/7/reviews"): + return [] + return None + + async def graphql(self, query: str, variables: dict, timeout: float = 30): + self.calls.append(f"graphql before={variables.get('before')}") + if self.graphql_fails: + return None + assert variables["owner"] == "acme" and variables["repo"] == "widgets" + assert variables["number"] == 7 + # Pages are cut from the newest end, as `last: 100, before: ` does. + end = int(variables["before"]) if variables.get("before") else len(self.events) + begin = max(0, end - self.event_page_size) + thread: dict = { + "timelineItems": { + "pageInfo": {"hasPreviousPage": begin > 0, "startCursor": str(begin)}, + "nodes": [ + {"createdAt": _ts(at), "actor": {"login": login}} + for at, login in self.events[begin:end] + ], + }, + } + if self.kind == "PullRequest": + thread["reviews"] = {"nodes": self.reviews} + return {"repository": {"issueOrPullRequest": thread}} + + +def _source(monkeypatch, fake: _FakeGitHub, *, login: str = _ME, unread=None) -> GitHubSource: + src = GitHubSource(unread=unread) + monkeypatch.setattr(src, "_gh_api_get", fake.get) + monkeypatch.setattr(src, "_gh_graphql", fake.graphql) + src._run = gh._Run(login=login, first_look=_START) + return src + + +async def _record(src: GitHubSource, notif: dict): + extra = await src._enrich_notification(notif, asyncio.Semaphore(5)) + return extra, notification_record(notif, extra) + + +# --------------------------------------------------------------------------- +# What counts as a mention in a text +# --------------------------------------------------------------------------- + +@pytest.mark.parametrize("text,expected", [ + ("cc @alice", True), + ("@Alice, please look", True), + ("(@alice)", True), + ("thanks @alice.", True), + ("@alice's change", True), + ("@alice-bot ping", False), # another, longer handle + ("@alicex", False), + ("@alice2", False), + ("@alice/reviewers", False), # a team of an organization of that name + ("mail carol@alice", False), # the end of an address + ("see `@alice` in the log", False), + ("```\n@alice\n```", False), + ("~~~\n@alice\n~~~", False), + ("", False), + (None, False), +]) +def test_what_text_mentions_the_login(text, expected): + assert gh._mentions(gh._mention_re("alice"), text) is expected + + +def test_a_login_with_a_hyphen_is_matched_whole(): + pattern = gh._mention_re("alice-dev") + assert gh._mentions(pattern, "ping @alice-dev") + assert not gh._mentions(pattern, "ping @alice") + assert not gh._mentions(pattern, "ping @alice-developer") + + +# --------------------------------------------------------------------------- +# new / old / not known +# --------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_a_mention_in_a_new_comment_is_new(monkeypatch): + fake = _FakeGitHub() + fake.comments = [ + _comment(11, "bob", "@alice can you look at this?", 30), + _comment(12, "some-ci[bot]", "Build finished.", 40), + ] + fake.events = [(32, "alice")] # GitHub writes the event after the comment + src = _source(monkeypatch, fake) + + extra, record = await _record(src, _notif()) + + assert extra["mention"] == { + "state": "new", + "at": _ts(30), + "by": "bob", + "url": f"{_WEB}/issues/7#issuecomment-11", + "body": "@alice can you look at this?", + "created_at": _ts(30), + } + assert record.summary == "[acme/widgets] Widget breaks (mention)" + assert record.metadata["reason"] == "mention" + assert record.metadata["mention"] == "new" + assert record.metadata["mention_at"] == _ts(30) + assert record.metadata["mention_by"] == "bob" + assert record.metadata["mention_url"] == f"{_WEB}/issues/7#issuecomment-11" + assert "bob" in record.metadata["actors"] + assert f"Mention: new by bob ({_ts(30)}, {_WEB}/issues/7#issuecomment-11)" in record.content + # The mention text is quoted once, under its own heading. + assert f"--- Mention (bob, {_ts(30)}) ---\n@alice can you look at this?" in record.content + assert record.content.count("@alice can you look at this?") == 1 + assert "New Comments" not in record.content + + +@pytest.mark.asyncio +async def test_a_mention_by_a_bot_is_quoted_and_its_author_is_an_actor(monkeypatch): + # A bot is left out of the new comments, so without the mention text the + # record would name neither the bot nor what it wrote. + fake = _FakeGitHub() + fake.comments = [ + _comment(11, "some-ci[bot]", "@alice the review found an issue.", 30), + _comment(12, "bob", "Looking.", 45), + ] + fake.events = [(31, "alice")] + src = _source(monkeypatch, fake) + + extra, record = await _record(src, _notif()) + + assert extra["mention"]["by"] == "some-ci[bot]" + assert "--- Mention (some-ci[bot], " in record.content + assert "some-ci[bot]" in record.metadata["actors"] + assert "--- New Comments (human, newest first) ---" in record.content + assert "\nbob (" in record.content + + +@pytest.mark.asyncio +async def test_an_update_without_a_mention_on_a_mentioned_thread_is_old(monkeypatch): + fake = _FakeGitHub() + fake.comments = [_comment(12, "bob", "Fixed on all branches.", 30)] + fake.events = [(-9_000_000, "alice"), (-8_000_000, "dave"), (29, "dave")] + src = _source(monkeypatch, fake) + + extra, record = await _record(src, _notif()) + + assert extra["mention"] == {"state": "old", "at": _ts(-9_000_000)} + assert record.summary == "[acme/widgets] Widget breaks (update, no new mention)" + # GitHub's reason stays as it is: the reason guardrail reads it. + assert record.metadata["reason"] == "mention" + assert record.metadata["mention"] == "old" + assert record.metadata["mention_at"] == _ts(-9_000_000) + assert "mention_by" not in record.metadata + assert "mention_url" not in record.metadata + assert "Mention: none in this update." in record.content + assert f"(the last was at {_ts(-9_000_000)})" in record.content + assert "--- Mention" not in record.content + assert "Fixed on all branches." in record.content + + +@pytest.mark.asyncio +async def test_no_known_mention_leaves_the_record_as_github_says(monkeypatch): + # GitHub says mention, and nothing this look can read explains it: the + # record is not moved away from mention on a guess. + fake = _FakeGitHub() + fake.comments = [_comment(12, "bob", "Any news?", 30)] + fake.events = [(20, "dave")] + src = _source(monkeypatch, fake) + + extra, record = await _record(src, _notif()) + + assert "mention" not in extra + assert record.summary == "[acme/widgets] Widget breaks (mention)" + assert not [k for k in record.metadata if k.startswith("mention")] + assert "Mention:" not in record.content + + +@pytest.mark.asyncio +async def test_a_failed_read_of_the_mention_events_leaves_the_record(monkeypatch): + fake = _FakeGitHub() + fake.events = [(-9_000_000, "alice")] + fake.graphql_fails = True + src = _source(monkeypatch, fake) + + extra, record = await _record(src, _notif()) + + assert "mention" not in extra + assert record.summary.endswith("(mention)") + assert "mention" not in record.metadata + + +@pytest.mark.asyncio +async def test_without_a_login_nothing_is_checked(monkeypatch): + fake = _FakeGitHub() + fake.events = [(-9_000_000, "alice")] + src = _source(monkeypatch, fake, login="") + + extra, record = await _record(src, _notif()) + + assert "mention" not in extra + assert record.summary.endswith("(mention)") + assert not [c for c in fake.calls if c.startswith("graphql")] + + +@pytest.mark.asyncio +async def test_other_reasons_are_not_checked(monkeypatch): + fake = _FakeGitHub() + fake.comments = [_comment(11, "bob", "@alice look", 30)] + fake.events = [(31, "alice")] + src = _source(monkeypatch, fake) + + extra, record = await _record(src, _notif(reason="review_requested")) + + assert "mention" not in extra + assert record.summary.endswith("(review_requested)") + assert not [c for c in fake.calls if c.startswith("graphql")] + # The new comments are still added. + assert "@alice look" in record.content + + +@pytest.mark.asyncio +async def test_a_mention_added_by_an_edit_is_new(monkeypatch): + # The comment is older than the start of the look; the edit is not, and + # GitHub writes the mention event at the edit. + fake = _FakeGitHub() + fake.comments = [_comment(11, "bob", "Reproduced. cc @alice", -500, edited=30)] + fake.events = [(31, "alice")] + src = _source(monkeypatch, fake) + + extra, _ = await _record(src, _notif()) + + assert extra["mention"]["state"] == "new" + assert extra["mention"]["by"] == "bob" + assert extra["mention"]["at"] == _ts(30) + + +@pytest.mark.asyncio +async def test_a_mention_in_the_description_of_a_new_thread_is_new(monkeypatch): + fake = _FakeGitHub() + fake.subject["created_at"] = _ts(30) + fake.subject["body"] = "Found by @alice in testing." + fake.events = [(31, "alice")] + src = _source(monkeypatch, fake) + + extra, record = await _record(src, _notif()) + + assert extra["mention"] == { + "state": "new", "at": _ts(30), "by": "carol", "url": f"{_WEB}/issues/7", + } + # The description is in the record already and is not quoted twice. + assert "--- Mention" not in record.content + assert record.content.count("Found by @alice in testing.") == 1 + + +@pytest.mark.asyncio +async def test_a_mention_event_without_its_text_is_still_new(monkeypatch): + # The text is gone (a deleted comment) or is one this look does not read. + fake = _FakeGitHub() + fake.events = [(31, "alice")] + src = _source(monkeypatch, fake) + + extra, record = await _record(src, _notif()) + + assert extra["mention"] == {"state": "new", "at": _ts(31)} + assert record.metadata["mention"] == "new" + assert "mention_by" not in record.metadata + assert f"Mention: new ({_ts(31)})" in record.content + + +# --------------------------------------------------------------------------- +# Reviews and inline review comments: GitHub writes no mention event for them +# --------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_a_mention_in_an_inline_review_comment_is_new(monkeypatch): + fake = _FakeGitHub("PullRequest") + fake.inline = [ + _inline(21, "bob", "Old remark.", -400), + _inline(22, "bob", "@alice is this path still used?", 30), + ] + src = _source(monkeypatch, fake) + + extra, record = await _record(src, _notif("PullRequest")) + + assert extra["mention"]["state"] == "new" + assert extra["mention"]["by"] == "bob" + assert extra["mention"]["url"] == f"{_WEB}/pull/7#discussion_r22" + assert record.metadata["mention_url"] == f"{_WEB}/pull/7#discussion_r22" + assert "--- Mention (bob, " in record.content + assert "@alice is this path still used?" in record.content + + +@pytest.mark.asyncio +async def test_a_mention_in_a_review_is_new(monkeypatch): + fake = _FakeGitHub("PullRequest") + fake.reviews = [ + _review("bob", "Looks fine.", -400), + _review("bob", "One question for @alice below.", 30), + {"author": {"login": "alice"}, "body": "draft", "submittedAt": None, "url": ""}, + ] + src = _source(monkeypatch, fake) + + extra, _ = await _record(src, _notif("PullRequest")) + + assert extra["mention"]["state"] == "new" + assert extra["mention"]["by"] == "bob" + assert extra["mention"]["url"] == f"{_WEB}/pull/7#pullrequestreview-30" + + +@pytest.mark.asyncio +async def test_an_earlier_inline_mention_explains_the_reason(monkeypatch): + fake = _FakeGitHub("PullRequest") + fake.inline = [_inline(22, "bob", "@alice is this path still used?", -400)] + fake.comments = [_comment(12, "bob", "Rebased.", 30)] + src = _source(monkeypatch, fake) + + extra, record = await _record(src, _notif("PullRequest")) + + assert extra["mention"] == {"state": "old", "at": _ts(-400)} + assert record.summary.endswith("(update, no new mention)") + + +@pytest.mark.asyncio +async def test_a_failed_read_of_inline_comments_gives_no_old(monkeypatch): + # The events explain the reason, but a mention may sit in the inline + # comments that could not be read. + fake = _FakeGitHub("PullRequest") + fake.events = [(-9_000_000, "alice")] + fake.inline = None + src = _source(monkeypatch, fake) + + extra, record = await _record(src, _notif("PullRequest")) + + assert "mention" not in extra + assert record.summary.endswith("(mention)") + + +@pytest.mark.asyncio +async def test_a_failed_read_does_not_hide_a_mention_that_was_found(monkeypatch): + fake = _FakeGitHub("PullRequest") + fake.graphql_fails = True + fake.inline = [_inline(22, "bob", "@alice is this path still used?", 30)] + src = _source(monkeypatch, fake) + + extra, _ = await _record(src, _notif("PullRequest")) + + assert extra["mention"]["state"] == "new" + + +@pytest.mark.asyncio +async def test_more_inline_comments_than_one_page_since_the_start_give_no_old(monkeypatch): + fake = _FakeGitHub("PullRequest") + fake.events = [(-9_000_000, "alice")] + fake.inline = [_inline(100 + i, "bob", "nit", 1 + i % 50) for i in range(120)] + src = _source(monkeypatch, fake) + + extra, _ = await _record(src, _notif("PullRequest")) + + assert "mention" not in extra + + +# --------------------------------------------------------------------------- +# The mention events are read newest first +# --------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_older_pages_of_mention_events_are_read_until_one_is_of_the_account(monkeypatch): + fake = _FakeGitHub() + fake.event_page_size = 2 + fake.events = [(-900, "alice"), (-800, "dave"), (-700, "dave"), (-600, "erin"), (-500, "dave")] + src = _source(monkeypatch, fake) + + extra, _ = await _record(src, _notif()) + + assert extra["mention"] == {"state": "old", "at": _ts(-900)} + assert [c for c in fake.calls if c.startswith("graphql")] == [ + "graphql before=None", "graphql before=3", "graphql before=1", + ] + + +@pytest.mark.asyncio +async def test_reading_stops_at_the_first_page_with_a_mention_of_the_account(monkeypatch): + fake = _FakeGitHub() + fake.event_page_size = 2 + fake.events = [(-900, "alice"), (-800, "dave"), (-700, "alice"), (-600, "dave")] + src = _source(monkeypatch, fake) + + extra, _ = await _record(src, _notif()) + + assert extra["mention"] == {"state": "old", "at": _ts(-700)} + assert [c for c in fake.calls if c.startswith("graphql")] == ["graphql before=None"] + + +@pytest.mark.asyncio +async def test_a_mention_beyond_the_pages_read_is_not_known(monkeypatch): + fake = _FakeGitHub() + fake.event_page_size = 1 + fake.events = [(-900, "alice")] + [(-800 + i, "dave") for i in range(gh._MENTION_LOG_PAGES)] + src = _source(monkeypatch, fake) + + extra, _ = await _record(src, _notif()) + + assert "mention" not in extra + assert len([c for c in fake.calls if c.startswith("graphql")]) == gh._MENTION_LOG_PAGES + + +@pytest.mark.asyncio +async def test_the_login_is_compared_in_any_letter_case(monkeypatch): + fake = _FakeGitHub() + fake.events = [(31, "Alice")] + src = _source(monkeypatch, fake, login="ALICE") + + extra, _ = await _record(src, _notif()) + + assert extra["mention"]["state"] == "new" + + +# --------------------------------------------------------------------------- +# From one update of a thread to the next +# --------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_the_same_update_listed_again_gives_the_same_record(monkeypatch): + # The cursor's own second is listed again in every run. A record that came + # out differently would be stored again and shown to its readers twice. + fake = _FakeGitHub() + fake.comments = [_comment(11, "bob", "@alice can you look at this?", 30)] + fake.events = [(32, "alice")] + src = _source(monkeypatch, fake) + + _, first = await _record(src, _notif()) + # A later run has a later cursor, so a later first look. + src._run = gh._Run(login=_ME, first_look=_T0 + timedelta(seconds=50)) + _, again = await _record(src, _notif()) + + assert again == first + assert again.metadata["mention"] == "new" + + +@pytest.mark.asyncio +async def test_a_mention_that_was_read_is_old_in_the_next_update(monkeypatch): + # No reader has anything left to read here (the source was given no read + # state), so the second look starts where the first ended. + fake = _FakeGitHub() + fake.comments = [_comment(11, "bob", "@alice can you look at this?", 30)] + fake.events = [(32, "alice")] + src = _source(monkeypatch, fake) + + _, first = await _record(src, _notif(updated_at=52)) + assert first.metadata["mention"] == "new" + + fake.comments.append(_comment(12, "bob", "Never mind, found it.", 300)) + fake.calls.clear() + _, second = await _record(src, _notif(updated_at=320)) + + assert second.metadata["mention"] == "old" + assert second.metadata["mention_at"] == _ts(32) + assert second.summary.endswith("(update, no new mention)") + # The second look starts where the first ended: at the thread's update, + # the newest moment the first look covered. + assert f"{_API}/issues/7/comments?since={_ts(52)}&per_page=100" in fake.calls + assert "Never mind, found it." in second.content + assert "@alice can you look at this?" not in second.content + + +@pytest.mark.asyncio +async def test_a_mention_stays_new_until_a_reader_has_read_its_record(monkeypatch): + # Storage keeps one record per thread. The source runs more often than + # the reader comes, so the record that says new can be replaced before + # anyone read it: the record that replaces it must still say new. + fake = _FakeGitHub() + fake.comments = [_comment(11, "bob", "@alice can you look at this?", 30)] + fake.events = [(32, "alice")] + src = _source(monkeypatch, fake) + + _, first = await _record(src, _notif(updated_at=52)) + assert first.metadata["mention"] == "new" + + # A bot comment moves the thread. The reader has not come yet. + fake.comments.append(_comment(12, "some-ci[bot]", "Build finished.", 300)) + fake.calls.clear() + src._run = gh._Run(login=_ME, first_look=_START, unread={"t1": first.metadata}) + _, second = await _record(src, _notif(updated_at=320)) + + assert f"{_API}/issues/7/comments?since={_ts(0)}&per_page=100" in fake.calls + assert second.summary == "[acme/widgets] Widget breaks (mention)" + assert second.metadata["mention"] == "new" + assert second.metadata["mention_by"] == "bob" + assert second.metadata["mention_url"] == f"{_WEB}/issues/7#issuecomment-11" + assert "--- Mention (bob, " in second.content + assert "@alice can you look at this?" in second.content + + # The reader has read that record. The next update is an update only. + fake.comments.append(_comment(13, "some-ci[bot]", "Deployed.", 600)) + fake.calls.clear() + src._run = gh._Run(login=_ME, first_look=_START, unread={}) + _, third = await _record(src, _notif(updated_at=620)) + + assert f"{_API}/issues/7/comments?since={_ts(320)}&per_page=100" in fake.calls + assert third.summary.endswith("(update, no new mention)") + assert third.metadata["mention"] == "old" + assert "@alice can you look at this?" not in third.content + + +@pytest.mark.asyncio +async def test_a_read_state_that_is_not_known_counts_as_unread(monkeypatch): + fake = _FakeGitHub() + fake.comments = [_comment(11, "bob", "@alice can you look at this?", 30)] + fake.events = [(32, "alice")] + src = _source(monkeypatch, fake) + await _record(src, _notif(updated_at=52)) + + fake.comments.append(_comment(12, "some-ci[bot]", "Build finished.", 300)) + src._run = gh._Run(login=_ME, first_look=_START, unread=None) + _, second = await _record(src, _notif(updated_at=320)) + + assert second.metadata["mention"] == "new" + + +@pytest.mark.asyncio +async def test_a_listed_again_update_keeps_its_record_after_it_was_read(monkeypatch): + fake = _FakeGitHub() + fake.comments = [_comment(11, "bob", "@alice can you look at this?", 30)] + fake.events = [(32, "alice")] + src = _source(monkeypatch, fake) + _, first = await _record(src, _notif(updated_at=52)) + + fake.comments.append(_comment(12, "some-ci[bot]", "Build finished.", 300)) + src._run = gh._Run(login=_ME, first_look=_START, unread={"t1": first.metadata}) + _, second = await _record(src, _notif(updated_at=320)) + + # The reader read the second record; the same update is listed again. + src._run = gh._Run(login=_ME, first_look=_START, unread={}) + _, again = await _record(src, _notif(updated_at=320)) + + assert again == second + assert again.metadata["mention"] == "new" + + +@pytest.mark.asyncio +async def test_after_a_restart_an_unread_new_mention_is_looked_at_again(monkeypatch): + # No look is remembered and the cursor is past the mention, but the stored + # record that says new has not been read: the look reaches back to it. + fake = _FakeGitHub() + fake.comments = [ + _comment(11, "bob", "@alice can you look at this?", 30), + _comment(12, "some-ci[bot]", "Build finished.", 300), + ] + fake.events = [(32, "alice")] + src = _source(monkeypatch, fake) + src._run = gh._Run( + login=_ME, first_look=_T0 + timedelta(seconds=200), + unread={"t1": {"reason": "mention", "mention": "new", "mention_at": _ts(30)}}, + ) + + _, record = await _record(src, _notif(updated_at=320)) + + assert f"{_API}/issues/7/comments?since={_ts(15)}&per_page=100" in fake.calls + assert record.metadata["mention"] == "new" + assert record.metadata["mention_by"] == "bob" + + +@pytest.mark.asyncio +async def test_after_a_restart_a_read_mention_is_old(monkeypatch): + fake = _FakeGitHub() + fake.comments = [ + _comment(11, "bob", "@alice can you look at this?", 30), + _comment(12, "some-ci[bot]", "Build finished.", 300), + ] + fake.events = [(32, "alice")] + src = _source(monkeypatch, fake) + src._run = gh._Run(login=_ME, first_look=_T0 + timedelta(seconds=200), unread={}) + + _, record = await _record(src, _notif(updated_at=320)) + + assert record.metadata["mention"] == "old" + + +@pytest.mark.asyncio +async def test_a_look_that_failed_is_repeated_from_its_start(monkeypatch): + fake = _FakeGitHub() + fake.comments = [_comment(11, "bob", "@alice can you look at this?", 30)] + fake.events = [(32, "alice")] + fake.graphql_fails = True + src = _source(monkeypatch, fake) + + _, first = await _record(src, _notif(updated_at=52)) + assert "mention" not in first.metadata + + # The next update: the read works again, and the look covers the time the + # failed one could not, so the mention is not lost. + fake.graphql_fails = False + fake.comments.append(_comment(12, "bob", "Any news?", 300)) + _, second = await _record(src, _notif(updated_at=320)) + + assert second.metadata["mention"] == "new" + assert second.metadata["mention_by"] == "bob" + + +@pytest.mark.asyncio +async def test_a_thread_never_looked_at_starts_before_the_cursor(monkeypatch): + fake = _FakeGitHub() + src = _source(monkeypatch, fake) + src._run = gh._Run(login=_ME, first_look=_T0 - gh._FIRST_LOOK_MARGIN) + + await _record(src, _notif()) + + assert f"{_API}/issues/7/comments?since={_ts(-120)}&per_page=100" in fake.calls + + +@pytest.mark.asyncio +async def test_only_the_latest_looks_are_remembered(monkeypatch): + monkeypatch.setattr(gh, "_REMEMBERED_THREADS", 2) + fake = _FakeGitHub() + src = _source(monkeypatch, fake) + + for thread_id in ("t1", "t2", "t3"): + await _record(src, _notif(thread_id=thread_id)) + + assert list(src._looks) == ["t2", "t3"] + + +# --------------------------------------------------------------------------- +# The new comments +# --------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_new_comments_are_listed_by_time_and_shown_newest_first(monkeypatch): + fake = _FakeGitHub() + fake.comments = [_comment(1, "old-timer", "First!", -9_000)] + fake.comments += [_comment(10 + i, f"user{i}", f"comment {i}", 10 + i) for i in range(7)] + fake.comments.append(_comment(30, "some-ci[bot]", "Build finished.", 40)) + fake.comments.append(_comment(31, "deploy-bot", "Deployed.", 41)) + src = _source(monkeypatch, fake) + + extra, record = await _record(src, _notif(reason="assign")) + + # The listing of one thread's comments takes no sort order: it is asked + # for the comments since the start of the look instead. + listing = [c for c in fake.calls if "/issues/7/comments" in c] + assert listing == [f"{_API}/issues/7/comments?since={_ts(0)}&per_page=100"] + assert [c["user"] for c in extra["recent_comments"]] == [ + "user6", "user5", "user4", "user3", "user2", + ] + assert "First!" not in record.content + assert "Build finished." not in record.content + assert "--- New Comments (human, newest first) ---" in record.content + + +@pytest.mark.asyncio +async def test_a_long_listing_of_comments_is_read_page_by_page(monkeypatch): + fake = _FakeGitHub() + fake.comments = [_comment(100 + i, "bob", f"comment {i}", 1 + i) for i in range(150)] + src = _source(monkeypatch, fake) + + extra, _ = await _record(src, _notif(reason="assign", updated_at=400)) + + listing = [c for c in fake.calls if "/issues/7/comments" in c] + assert listing == [ + f"{_API}/issues/7/comments?since={_ts(0)}&per_page=100", + f"{_API}/issues/7/comments?since={_ts(0)}&per_page=100&page=2", + ] + assert extra["recent_comments"][0]["body"] == "comment 149" + + +@pytest.mark.asyncio +async def test_a_listing_longer_than_the_pages_followed_is_not_used(monkeypatch): + fake = _FakeGitHub() + fake.comments = [ + _comment(1000 + i, "bob", f"comment {i}", 1) for i in range(100 * gh._COMMENT_PAGES) + ] + fake.events = [(-9_000_000, "alice")] + src = _source(monkeypatch, fake) + + extra, record = await _record(src, _notif()) + + assert "recent_comments" not in extra + # The mention answer needs no comments; the look is not kept as complete, + # so the next update's look repeats the range. + assert record.metadata["mention"] == "old" + assert src._looks["t1"].end == _START + + +@pytest.mark.asyncio +async def test_the_latest_comment_is_not_repeated_among_the_new_comments(monkeypatch): + fake = _FakeGitHub() + fake.comments = [ + _comment(11, "bob", "First reply.", 20), + _comment(12, "erin", "Second reply.", 30), + ] + src = _source(monkeypatch, fake) + notif = _notif(reason="assign") + notif["subject"]["latest_comment_url"] = f"{_API}/issues/comments/12" + real_get = fake.get + + async def get(url, timeout=30): + if url.endswith("/issues/comments/12"): + return fake.comments[1] + return await real_get(url, timeout) + + monkeypatch.setattr(src, "_gh_api_get", get) + + extra, record = await _record(src, notif) + + assert [c["user"] for c in extra["recent_comments"]] == ["bob"] + assert record.content.count("Second reply.") == 1 + + +# --------------------------------------------------------------------------- +# The login is read in every run that needs it +# --------------------------------------------------------------------------- + +class _FakeProc: + def __init__(self, stdout: bytes, returncode: int = 0): + self._stdout = stdout + self.returncode = returncode + + async def communicate(self): + return self._stdout, b"" + + +def _list_with(monkeypatch, notifications: list[dict]) -> None: + """Make ``gh api notifications?...`` print *notifications*.""" + async def fake_exec(*argv, **kwargs): + assert argv[:2] == ("gh", "api") and argv[2].startswith("notifications?") + return _FakeProc("\n".join(json.dumps(n) for n in notifications).encode()) + + monkeypatch.setattr(gh.asyncio, "create_subprocess_exec", fake_exec) + + +@pytest.mark.asyncio +async def test_the_login_is_read_in_each_run_with_a_mention_thread(monkeypatch): + fake = _FakeGitHub() + fake.events = [(-9_000_000, "alice"), (-8_000_000, "zoe")] + src = _source(monkeypatch, fake) + _list_with(monkeypatch, [_notif()]) + + first = await src.fetch(_ts(0)) + assert fake.calls.count("user") == 1 + assert first.records[0].metadata["mention"] == "old" + assert first.records[0].metadata["mention_at"] == _ts(-9_000_000) + + # gh acts as another account from now on. The next run reads the login + # again and looks for that account's mentions. + fake.login = "zoe" + second = await src.fetch(_ts(0)) + assert fake.calls.count("user") == 2 + assert second.records[0].metadata["mention_at"] == _ts(-8_000_000) + + +@pytest.mark.asyncio +async def test_a_run_without_a_mention_thread_reads_no_login(monkeypatch): + fake = _FakeGitHub() + src = _source(monkeypatch, fake) + unreadable = _notif(thread_id="t2") + unreadable["subject"]["url"] = None # nothing to read mentions from + _list_with(monkeypatch, [_notif(reason="author"), unreadable]) + + await src.fetch(_ts(0)) + + assert "user" not in fake.calls + + +@pytest.mark.asyncio +async def test_a_login_that_cannot_be_read_leaves_the_records_as_they_are(monkeypatch): + fake = _FakeGitHub() + fake.events = [(-9_000_000, "alice")] + fake.login = None + src = _source(monkeypatch, fake) + _list_with(monkeypatch, [_notif()]) + + result = await src.fetch(_ts(0)) + + assert result.records[0].summary.endswith("(mention)") + assert "mention" not in result.records[0].metadata + + +@pytest.mark.asyncio +async def test_another_account_forgets_the_looks(monkeypatch): + fake = _FakeGitHub() + src = _source(monkeypatch, fake) + _list_with(monkeypatch, [_notif()]) + + await src.fetch(_ts(0)) + assert list(src._looks) == ["t1"] + + fake.login = "zoe" + _list_with(monkeypatch, [_notif(thread_id="t9")]) + await src.fetch(_ts(0)) + + assert list(src._looks) == ["t9"] + + +@pytest.mark.asyncio +async def test_a_run_starts_its_first_looks_shortly_before_the_cursor(monkeypatch): + fake = _FakeGitHub() + src = _source(monkeypatch, fake) + _list_with(monkeypatch, [_notif()]) + + await src.fetch(_ts(0)) + + assert f"{_API}/issues/7/comments?since={_ts(-120)}&per_page=100" in fake.calls + + +# --------------------------------------------------------------------------- +# What the readers of the inbox have read +# --------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_a_run_asks_which_records_are_unread(monkeypatch): + asked: list[list[str]] = [] + + async def unread(ids): + asked.append(ids) + return {} + + fake = _FakeGitHub() + src = _source(monkeypatch, fake, unread=unread) + _list_with(monkeypatch, [ + _notif(thread_id="t1"), + _notif(reason="assign", thread_id="t2"), + _notif(reason="author", thread_id="t3"), + ]) + + await src.fetch(_ts(0)) + + assert asked == [["t1", "t2"]] + + +@pytest.mark.asyncio +async def test_a_run_that_cannot_read_the_read_state_treats_records_as_unread(monkeypatch): + async def unread(ids): + raise RuntimeError("database is locked") + + fake = _FakeGitHub() + src = _source(monkeypatch, fake, unread=unread) + _list_with(monkeypatch, [_notif()]) + + await src.fetch(_ts(0)) + + assert src._run.unread is None + assert src._unread_record("t1") == {} + + +def _stored(rid: str, metadata: dict): + from nerve.sources.models import SourceRecord + + return SourceRecord( + id=rid, source="github", record_type="github_notification", + summary=f"[acme/widgets] thread {rid} (mention)", content="text", + timestamp=_ts(0), metadata=metadata, + ) + + +@pytest.mark.asyncio +async def test_storage_says_a_record_is_unread_until_every_live_consumer_read_it(db): + # Nobody reads the source: nothing is unread. + await db.insert_source_messages([_stored("t0", {"mention": "new"})], source="github") + assert await db.unread_source_message_metadata("github", ["t0"]) == {} + + # Two consumers start here; a cursor of another source does not count. + start = await db.get_consumer_cursor("inbox", "github") + assert await db.get_consumer_cursor("digest", "github") == start + await db.set_consumer_cursor("inbox", "gmail", 0) + await db.insert_source_messages( + [_stored("t1", {"mention": "new", "mention_at": _ts(30)}), _stored("t2", {})], + source="github", + ) + + assert await db.unread_source_message_metadata("github", ["t0", "t1", "t2", "missing"]) == { + "t1": {"mention": "new", "mention_at": _ts(30)}, + "t2": {}, + } + assert await db.unread_source_message_metadata("github", []) == {} + assert await db.unread_source_message_metadata("gmail", ["t1"]) == {} + + # One consumer read them. The other has not. + rows = await db.read_source_messages_by_rowid("github", after_seq=start, limit=50) + newest = max(r["rowid"] for r in rows) + await db.set_consumer_cursor("inbox", "github", newest) + assert sorted(await db.unread_source_message_metadata("github", ["t1", "t2"])) == ["t1", "t2"] + + # The other consumer's cursor expires: it starts again at the newest + # record and never reads these. + await db.db.execute( + "UPDATE consumer_cursors SET expires_at = ? WHERE consumer = 'digest'", + ("2000-01-01T00:00:00+00:00",), + ) + await db.db.commit() + assert await db.unread_source_message_metadata("github", ["t1", "t2"]) == {} + + +@pytest.mark.asyncio +async def test_the_registered_source_reads_the_read_state_from_storage(db): + cfg = NerveConfig.from_dict({"sync": {"github": {"enabled": True}}}) + runner = next( + r for r in build_source_runners(cfg, db) if r.source.source_name == "github" + ) + await db.get_consumer_cursor("inbox", "github") + await db.insert_source_messages([_stored("t1", {"mention": "new"})], source="github") + + assert await runner.source._unread(["t1", "t2"]) == {"t1": {"mention": "new"}} + + +@pytest.mark.asyncio +async def test_a_mention_replaced_before_the_reader_came_still_reaches_the_reader(db, monkeypatch): + # The whole path: the source runs twice between two reads of the inbox. + # A human mentions the account, the first run stores a record that says + # new, a bot comment moves the thread, and the second run replaces the + # record. The reader then sees one record of the thread, and it must be + # the mention, not an update. + fake = _FakeGitHub() + src = GitHubSource(unread=lambda ids: db.unread_source_message_metadata("github", ids)) + monkeypatch.setattr(src, "_gh_api_get", fake.get) + monkeypatch.setattr(src, "_gh_graphql", fake.graphql) + runner = SourceRunner(src, db) + await db.set_sync_cursor("github", _ts(0)) + read_until = await db.get_consumer_cursor("inbox", "github") + + async def read_inbox() -> list[dict]: + nonlocal read_until + rows = await db.read_source_messages_by_rowid("github", after_seq=read_until, limit=50) + if rows: + read_until = max(r["rowid"] for r in rows) + await db.set_consumer_cursor("inbox", "github", read_until) + return rows + + fake.comments = [_comment(11, "bob", "@alice can you look at this?", 30)] + fake.events = [(32, "alice")] + _list_with(monkeypatch, [_notif(updated_at=52)]) + await runner.run() + + fake.comments.append(_comment(12, "some-ci[bot]", "Build finished.", 300)) + _list_with(monkeypatch, [_notif(updated_at=320)]) + await runner.run() + + rows = await read_inbox() + assert len(rows) == 1 + assert rows[0]["summary"] == "[acme/widgets] Widget breaks (mention)" + metadata = rows[0]["metadata"] + assert metadata["mention"] == "new" + assert metadata["mention_by"] == "bob" + assert "@alice can you look at this?" in rows[0]["content"] + + # The reader has read the mention. The next update is an update only. + fake.comments.append(_comment(13, "some-ci[bot]", "Deployed.", 600)) + _list_with(monkeypatch, [_notif(updated_at=620)]) + await runner.run() + + rows = await read_inbox() + assert len(rows) == 1 + assert rows[0]["summary"] == "[acme/widgets] Widget breaks (update, no new mention)" + assert rows[0]["metadata"]["mention"] == "old" + + +# --------------------------------------------------------------------------- +# gh api graphql +# --------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_graphql_sends_strings_raw_and_integers_typed(monkeypatch): + seen: list[tuple] = [] + + async def fake_exec(*argv, **kwargs): + seen.append(argv) + return _FakeProc(json.dumps({"data": {"repository": None}}).encode()) + + monkeypatch.setattr(gh.asyncio, "create_subprocess_exec", fake_exec) + + data = await GitHubSource._gh_graphql( + "query { viewer { login } }", {"owner": "@acme", "number": 7}, + ) + + assert data == {"repository": None} + assert seen == [( + "gh", "api", "graphql", "-f", "query=query { viewer { login } }", + "-f", "owner=@acme", "-F", "number=7", + )] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("stdout,returncode", [ + (json.dumps({"data": {"repository": None}, "errors": [{"type": "NOT_FOUND"}]}), 0), + (json.dumps({"data": {"repository": None}}), 1), + ("not json", 0), + (json.dumps([1, 2]), 0), + (json.dumps({"data": None}), 0), +]) +async def test_graphql_failures_return_none(monkeypatch, stdout, returncode): + async def fake_exec(*argv, **kwargs): + return _FakeProc(stdout.encode(), returncode) + + monkeypatch.setattr(gh.asyncio, "create_subprocess_exec", fake_exec) + + assert await GitHubSource._gh_graphql("query { viewer { login } }", {}) is None + + +# --------------------------------------------------------------------------- +# The record +# --------------------------------------------------------------------------- + +def test_a_record_without_a_mention_answer_has_no_mention_keys(): + record = notification_record(_notif(), {"html_url": f"{_WEB}/issues/7"}) + + assert record.summary == "[acme/widgets] Widget breaks (mention)" + assert sorted(record.metadata) == [ + "actors", "ci_branch", "reason", "repo_name", "repo_url", "subject_type", "subject_url", + ] + + +def test_the_author_of_an_old_mention_is_not_an_actor(): + record = notification_record(_notif(), { + "user": "carol", + "mention": {"state": "old", "at": _ts(-400), "by": "bob"}, + }) + + assert record.metadata["actors"] == ["carol"] + + +# --------------------------------------------------------------------------- +# GitHub events source: the username is looked up in every run +# --------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_events_source_looks_the_username_up_again(monkeypatch): + answers = [(b"alice\n", 0), (b"zoe\n", 0), (b"", 1)] + + async def fake_exec(*argv, **kwargs): + assert argv == ("gh", "api", "user", "--jq", ".login") + return _FakeProc(*answers.pop(0)) + + monkeypatch.setattr("nerve.sources.github_events.asyncio.create_subprocess_exec", fake_exec) + src = GitHubEventsSource() + + assert await src._resolve_username() == "alice" + assert await src._resolve_username() == "zoe" + # A failed lookup falls back to the login the last one returned. + assert await src._resolve_username() == "zoe" + + +@pytest.mark.asyncio +async def test_events_source_fails_without_any_username(monkeypatch): + async def fake_exec(*argv, **kwargs): + return _FakeProc(b"", 1) + + monkeypatch.setattr("nerve.sources.github_events.asyncio.create_subprocess_exec", fake_exec) + + with pytest.raises(RuntimeError): + await GitHubEventsSource()._resolve_username() + + +@pytest.mark.asyncio +async def test_events_source_keeps_a_configured_username(monkeypatch): + async def fake_exec(*argv, **kwargs): + raise AssertionError("no lookup for a configured username") + + monkeypatch.setattr("nerve.sources.github_events.asyncio.create_subprocess_exec", fake_exec) + + assert await GitHubEventsSource({"username": "alice"})._resolve_username() == "alice"