Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -292,6 +292,8 @@ NOTIFICATION_POLL_SECONDS=60 # polled rule evaluation interval; minimum
ONCALL_POLL_SECONDS=30 # on-call escalation worker interval; minimum 5, invalid values fall back to 30. Kept separate from NOTIFICATION_POLL_SECONDS so raising rule-evaluation intervals never delays paging. A buffered Wake() channel makes freshly opened pages notify L1 near-instantly regardless of this interval.
OUTBOX_POLL_SECONDS=15 # notification outbox drain interval; minimum 5, invalid values fall back to 15. The outbox (backend/app/outbox, notification_outbox table in the main DB) is the persist-then-send layer for ALL notifications: rule dispatch and the escalator only enqueue (with an adapter-config snapshot) inside their transactions; the drain worker sends with retries (backoff 1m/5m/15m/60m, 5 attempts, then terminal failed + CaptureException). Crash-safe at-least-once: stale 'sending' rows are reclaimed after 5 min, cancelled rows can never resurrect (guarded status transitions), ack/resolve cancels queued page deliveries via outbox.CancelByKey. Cooldown and event-rule dedup record at enqueue commit (the durable promise), and fired_notifications is written at the terminal outcome. /api/health/deep exposes an `outbox` block; `traceway.outbox.*` metrics are emitted when monitoring is on; terminal rows are pruned daily (sent/cancelled 7d, failed 30d).

**GitHub channel round trip.** The `github` channel is the one adapter with a lifecycle: it opens an issue when a rule fires and closes it again when the exception is archived. A `new_error`/`error_regression` delivery is tagged in `dispatch` (`trackGitHubIssue`, gated on `models.IsIssueRuleType` — an issue opened by a latency or metric rule has no exception to archive, so it is never tracked), and `AdapterSend` routes github deliveries to `sendGitHubIssue`, which records the created issue number in `github_issues` (main DB, keyed project + `issue_key` = exception hash + channel, `closed_at` doubles as the state). Every archive path — `/exception-stack-traces/archive` plus the two on-call page resolves that archive — then calls `notifications.CloseGitHubIssuesForArchived` in its transaction, which marks the rows closed and enqueues an `OutboxKindGithubClose` delivery per open issue. The rows are marked closed at enqueue (the durable promise, as with cooldowns) so a second archive cannot queue a second close; the close PATCHes `state=closed` using the **recorded** owner/repo rather than the channel's current config, treats 404/410 as already closed, and posts its explanatory comment after the close so a retry never doubles it. A channel retyped away from github drops the row instead of queueing a send that can only fail; a disabled one still closes, since this is one-shot cleanup of an issue that channel itself opened. Deleting the channel (or the project) cascades the rows away, which is also the only way a tracked issue is abandoned. A regression opens a fresh issue under the same hash, so an `issue_key` carries several rows over time.

# Synthetics (synthetic uptime monitoring: backend/app/synthetics, /synthetics frontend route)
SYNTHETICS_POLL_SECONDS=15 # scheduler tick for due checks; minimum 5, invalid values fall back to 15. The scheduler enqueues due checks into the check_runs queue (main DB, outbox-style guarded claims, advisory lock 824737004) and records expired queued runs as `missed` in telemetry — a probe is never executed late. In-process executors claim http/tcp runs always, browser runs only when mode=embedded.
SYNTHETICS_BROWSER_MODE=off # off | embedded | remote. Browser checks are real @playwright/test specs executed by spawning Node against a harness dir (no npm/npx at runtime; allowlisted env so user scripts never see server secrets). `embedded` requires the :browser image (Dockerfile.browser, DuckDB base + Node + Chromium) and fails fast at startup otherwise; `remote` queues browser runs for traceway-runner binaries that long-poll /api/runners/poll authenticating with SYNTHETICS_RUNNER_SECRET. Hard-blocked in cloud mode (startup panic + 422 at check creation).
Expand Down Expand Up @@ -911,6 +913,7 @@ func (c *ReportController) Report(ctx *gin.Context) {
| `invitations` | Team invitations with token, role, expiry |
| `source_maps` | Uploaded source map files (project, version, storage key) |
| `metric_registry` | Custom metric definitions (type, unit, description) |
| `github_issues` | Issues the GitHub channel opened, so archiving the exception can close them (`closed_at` = state) |
| `dashboards` | Org-owned dashboards (name, JSONB definition with widgets, template provenance) |
| `project_dashboards` | Which projects show a dashboard, and tab order |
| `dashboard_templates` | Marketplace templates (key, category, definition), seeded by migrations |
Expand Down Expand Up @@ -944,7 +947,7 @@ Both embedded builds (SQLite and DuckDB telemetry) are single-instance: the data
**Main DB tables** (`db.DB` — transactional, uses lit with `*sql.Tx`):
- `users`, `organizations`, `organization_users`, `projects`, `invitations`
- `source_maps`, `metric_registry`, `dashboards`, `project_dashboards`, `dashboard_templates`, `starred_dashboard_widgets` (plus the legacy `widget_groups`/`widget_group_widgets`/`starred_widgets`)
- `notification_channels`, `notification_rules`
- `notification_channels`, `notification_rules`, `github_issues`
- `synthetic_checks`, `check_runs`, `check_incidents`, `incident_updates`, `post_mortems`, `post_mortem_events`, `synthetic_runners`, `status_pages`

**Telemetry DB tables** (`db.TelemetryDB` — non-transactional, uses lit with `db.TelemetryDB` directly):
Expand Down
69 changes: 46 additions & 23 deletions backend/app/controllers/exception_stack_trace.controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,9 @@ import (
"github.com/tracewayapp/traceway/backend/app/db"
"github.com/tracewayapp/traceway/backend/app/middleware"
"github.com/tracewayapp/traceway/backend/app/models"
"github.com/tracewayapp/traceway/backend/app/notifications"
"github.com/tracewayapp/traceway/backend/app/oncall"
"github.com/tracewayapp/traceway/backend/app/outbox"
"github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
"github.com/tracewayapp/traceway/backend/app/repositories/transactional"
"github.com/tracewayapp/traceway/backend/app/storage"
Expand Down Expand Up @@ -45,6 +47,13 @@ type ArchiveRequest struct {
ResolvePages bool `json:"resolvePages"`
}

// archiveOutcome is the main-DB follow-up to an archive: the on-call pages it
// resolved and the GitHub issues it queued a close for.
type archiveOutcome struct {
ResolvedPages int
ClosedGithubIssues int
}

type ExceptionDetailRequest struct {
Pagination PaginationParams `json:"pagination"`
}
Expand Down Expand Up @@ -243,36 +252,50 @@ func (e exceptionStackTraceController) ArchiveExceptions(c *gin.Context) {
return
}

resolvedPages := 0
if request.ResolvePages {
userId := middleware.GetUserId(c)
now := time.Now().UTC()
resolvedPages, err = db.ExecuteTransaction(func(tx *sql.Tx) (int, error) {
resolved := 0
for _, hash := range request.Hashes {
pages, err := transactional.PageRepository.FindUnresolvedByIssueHash(tx, projectId, hash)
// The archive itself is telemetry and already landed; what follows is main-DB
// bookkeeping for the same issues, so it runs in one transaction after it.
userId := middleware.GetUserId(c)
now := time.Now().UTC()
outcome, err := db.ExecuteTransaction(func(tx *sql.Tx) (archiveOutcome, error) {
var result archiveOutcome
closed, err := notifications.CloseGitHubIssuesForArchived(tx, projectId, request.Hashes)
if err != nil {
return result, err
}
result.ClosedGithubIssues = closed
if !request.ResolvePages {
return result, nil
}
for _, hash := range request.Hashes {
pages, err := transactional.PageRepository.FindUnresolvedByIssueHash(tx, projectId, hash)
if err != nil {
return result, err
}
for _, page := range pages {
ok, err := oncall.ResolvePage(tx, page.Id, userId, now)
if err != nil {
return 0, err
return result, err
}
for _, page := range pages {
ok, err := oncall.ResolvePage(tx, page.Id, userId, now)
if err != nil {
return 0, err
}
if ok {
resolved++
}
if ok {
result.ResolvedPages++
}
}
return resolved, nil
})
if err != nil {
c.AbortWithError(500, traceway.NewStackTraceErrorf("error resolving pages for archived issues %s: %w", strings.Join(request.Hashes, ","), err))
return
}
return result, nil
})
if err != nil {
c.AbortWithError(500, traceway.NewStackTraceErrorf("error closing out archived issues %s: %w", strings.Join(request.Hashes, ","), err))
return
}
if outcome.ClosedGithubIssues > 0 {
outbox.Wake()
}

c.JSON(http.StatusOK, gin.H{"archived": len(request.Hashes), "resolvedPages": resolvedPages})
c.JSON(http.StatusOK, gin.H{
"archived": len(request.Hashes),
"resolvedPages": outcome.ResolvedPages,
"closedGithubIssues": outcome.ClosedGithubIssues,
})
}

func (e exceptionStackTraceController) UnarchiveExceptions(c *gin.Context) {
Expand Down
22 changes: 20 additions & 2 deletions backend/app/controllers/page.controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@ import (
"github.com/tracewayapp/traceway/backend/app/db"
"github.com/tracewayapp/traceway/backend/app/middleware"
"github.com/tracewayapp/traceway/backend/app/models"
"github.com/tracewayapp/traceway/backend/app/notifications"
"github.com/tracewayapp/traceway/backend/app/oncall"
"github.com/tracewayapp/traceway/backend/app/outbox"
"github.com/tracewayapp/traceway/backend/app/repositories/telemetry"
"github.com/tracewayapp/traceway/backend/app/repositories/transactional"

Expand Down Expand Up @@ -210,6 +212,7 @@ func (c *pageController) Resolve(ctx *gin.Context) {
}

archived := false
closedIssues := 0
if request.ArchiveIssue {
hash := page.IssueHash()
if hash == "" {
Expand All @@ -233,8 +236,15 @@ func (c *pageController) Resolve(ctx *gin.Context) {
return
}
archived = true
if closedIssues, err = notifications.CloseGitHubIssuesForArchived(tx, page.ProjectId, []string{hash}); err != nil {
ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to close GitHub issues for archived issue %s: %w", hash, err))
return
}
if closedIssues > 0 {
middleware.OnCommit(ctx, outbox.Wake)
}
}
ctx.JSON(http.StatusOK, gin.H{"message": "Page resolved", "archivedIssue": archived})
ctx.JSON(http.StatusOK, gin.H{"message": "Page resolved", "archivedIssue": archived, "closedGithubIssues": closedIssues})
}

type bulkAcknowledgeRequest struct {
Expand Down Expand Up @@ -343,6 +353,7 @@ func (c *pageController) BulkResolve(ctx *gin.Context) {
}

archivedIssues := 0
closedIssues := 0
if len(hashSet) > 0 {
// Resolving deliberately has no write gate, but archiving issues is a
// write: enforce the effective project role in-handler. Aborting rolls
Expand All @@ -365,8 +376,15 @@ func (c *pageController) BulkResolve(ctx *gin.Context) {
return
}
archivedIssues = len(hashes)
if closedIssues, err = notifications.CloseGitHubIssuesForArchived(tx, projectId, hashes); err != nil {
ctx.AbortWithError(http.StatusInternalServerError, traceway.NewStackTraceErrorf("failed to close GitHub issues for archived issues: %w", err))
return
}
if closedIssues > 0 {
middleware.OnCommit(ctx, outbox.Wake)
}
}
ctx.JSON(http.StatusOK, gin.H{"resolved": resolved, "archivedIssues": archivedIssues})
ctx.JSON(http.StatusOK, gin.H{"resolved": resolved, "archivedIssues": archivedIssues, "closedGithubIssues": closedIssues})
}

type pagesForIssuesRequest struct {
Expand Down
11 changes: 11 additions & 0 deletions backend/app/migrations/pg/0143_create_github_issues.up.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
CREATE TABLE IF NOT EXISTS github_issues (
id SERIAL PRIMARY KEY,
project_id UUID NOT NULL,
channel_id INT NOT NULL REFERENCES notification_channels(id) ON DELETE CASCADE,
issue_key VARCHAR(200) NOT NULL DEFAULT '',
owner VARCHAR(200) NOT NULL DEFAULT '',
repo VARCHAR(200) NOT NULL DEFAULT '',
issue_number INT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
closed_at TIMESTAMPTZ
)
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
CREATE INDEX IF NOT EXISTS github_issues_open_idx ON github_issues (project_id, issue_key) WHERE closed_at IS NULL
13 changes: 13 additions & 0 deletions backend/app/migrations/sqlite/0077_create_github_issues.up.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
CREATE TABLE IF NOT EXISTS github_issues (
id INTEGER PRIMARY KEY AUTOINCREMENT,
project_id TEXT NOT NULL,
channel_id INTEGER NOT NULL REFERENCES notification_channels(id) ON DELETE CASCADE,
issue_key TEXT NOT NULL DEFAULT '',
owner TEXT NOT NULL DEFAULT '',
repo TEXT NOT NULL DEFAULT '',
issue_number INTEGER NOT NULL,
created_at DATETIME NOT NULL,
closed_at DATETIME
);

CREATE INDEX IF NOT EXISTS github_issues_open_idx ON github_issues (project_id, issue_key) WHERE closed_at IS NULL;
9 changes: 8 additions & 1 deletion backend/app/models/escalation.model.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,18 @@ type Page struct {
UpdatedAt time.Time `json:"updatedAt" lit:"updated_at"`
}

// IsIssueRuleType reports whether a rule fires for one exception group, and so
// carries that group's hash as its dedup token. Pages and GitHub issues both
// use it to decide whether they are issue-linked.
func IsIssueRuleType(ruleType string) bool {
return ruleType == "new_error" || ruleType == "error_regression"
}

// IssueHash returns the exception hash a page was opened for, or "" when the
// page is not issue-linked. New-error and regression rules carry the hash as
// the dedup token after the "ruleId|" prefix (see oncall.pageDedupKey).
func (p *Page) IssueHash() string {
if p.RuleType != "new_error" && p.RuleType != "error_regression" {
if !IsIssueRuleType(p.RuleType) {
return ""
}
_, hash, found := strings.Cut(p.DedupKey, "|")
Expand Down
27 changes: 27 additions & 0 deletions backend/app/models/github_issue.model.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
package models

import (
"time"

"github.com/google/uuid"
)

// GithubIssue records an issue the GitHub channel opened, so archiving the
// exception it tracks can close it again. ClosedAt doubles as the state: an
// issue still open in GitHub has none.
//
// Rows are keyed per channel rather than per rule: several rules can point at
// the same GitHub channel, and the issue belongs to the repository it landed
// in, which is the channel's configuration. A regression opens a second issue
// under the same IssueKey, so a key can carry more than one row over time.
type GithubIssue struct {
Id int `json:"id" lit:"id"`
ProjectId uuid.UUID `json:"projectId" lit:"project_id"`
ChannelId int `json:"channelId" lit:"channel_id"`
IssueKey string `json:"issueKey" lit:"issue_key"`
Owner string `json:"owner" lit:"owner"`
Repo string `json:"repo" lit:"repo"`
IssueNumber int `json:"issueNumber" lit:"issue_number"`
CreatedAt time.Time `json:"createdAt" lit:"created_at"`
ClosedAt *time.Time `json:"closedAt" lit:"closed_at"`
}
1 change: 1 addition & 0 deletions backend/app/models/models.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ func Init(driver lit.Driver) {
lit.RegisterModel[NotificationChannel](driver)
lit.RegisterModel[NotificationRule](driver)
lit.RegisterModel[NotificationRuleWithChannel](driver)
lit.RegisterModel[GithubIssue](driver)
lit.RegisterModel[Team](driver)
lit.RegisterModel[TeamWithCounts](driver)
lit.RegisterModel[TeamProjectRow](driver)
Expand Down
26 changes: 26 additions & 0 deletions backend/app/models/notification_message.model.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,12 @@ type NotificationMessage struct {

Email *NotificationEmail `json:",omitempty"`

// GitHub is the GitHub adapter's payload: on a rule delivery it names what
// the created issue tracks, so archiving that exception can close the issue
// again; on a close delivery it names the issue to close. Only the GitHub
// adapter reads it.
GitHub *NotificationGitHub `json:",omitempty"`

// DedupToken is the stable identity of what fired within the rule
// (exception hash, endpoint, task or metric name; empty for rule-level
// conditions). Page dedup keys are built from it — never from URL, which
Expand All @@ -42,6 +48,26 @@ const (
EmailTemplateTest = "test"
)

// NotificationGitHub is carried by messages a GitHub channel can act on. Like
// the rest of NotificationMessage it is persisted in notification_outbox, so
// the shape is a wire format: add fields, never repurpose them.
type NotificationGitHub struct {
// IssueKey is the exception hash a created issue tracks, and ProjectId the
// project it belongs to. Empty when the firing rule is not issue-shaped:
// nothing would ever archive that issue, so nothing is recorded for it.
IssueKey string `json:",omitempty"`
ProjectId string `json:",omitempty"`
ChannelId int `json:",omitempty"`

// CloseNumber makes the delivery close that issue in Owner/Repo instead of
// creating a new one. Owner and Repo come from the recorded issue rather
// than the channel's current config, so a repository changed since the
// issue was opened still closes the right one.
CloseNumber int `json:",omitempty"`
Owner string `json:",omitempty"`
Repo string `json:",omitempty"`
}

type NotificationEmail struct {
Template string
Exception *EmailException `json:",omitempty"`
Expand Down
3 changes: 3 additions & 0 deletions backend/app/models/outbox.model.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ const (
OutboxKindRule = "rule"
OutboxKindPage = "page"
OutboxKindVerification = "verification"
// OutboxKindGithubClose closes a GitHub issue whose exception was archived.
// It is not a rule fire, so it records no fired_notifications audit row.
OutboxKindGithubClose = "github_close"

OutboxPending = "pending"
OutboxSending = "sending"
Expand Down
Loading