Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 33 additions & 14 deletions PROJECT_DESCRIPTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,10 +78,11 @@ Strict **Controller → Service → Repository** layering. No controller accesse
│ AsyncDataSourceCollectService · SyncJobTracker │
│ GitLocalCollector · RepoService · GitRepositoryService │
│ GitHubClientFactory · GitHubRepositoryService │
│ GitHubCollector · GitHubCommitIngestService │
│ GitHubCommitCollector · GitHubCommitIngestService │
│ GitHubCommitStatsEnrichmentService │
│ GitHubPrCollector · GitHubPullRequestCollector │
│ GitHubPrStatsEnrichmentService · GitHubIssuesCollector │
│ GitHubPullRequestCollector · GitHubPullRequestIngestService │
│ GitHubPullRequestStatsEnrichmentService │
│ GitHubPullRequestQueryService · GitHubIssuesCollector │
│ JiraCollector · IssueService │
│ MetricsService · MetricSnapshotService · MetricsScheduler │
│ MetricBackfillService · MetricBackfillScheduler │
Expand Down Expand Up @@ -121,6 +122,22 @@ Strict **Controller → Service → Repository** layering. No controller accesse
| `messaging/` | 1:1 direct message service and controller, conversation/inbox API |
| `invite/` | Token-based team invitation issuance and redemption |

`github/` runs two collection pipelines, so it is split further. `controller/`, `model/` and
`repository/` sit beside these.

| Subpackage | Classes |
|---|---|
| `github/client/` | `GitHubClientFactory` |
| `github/repo/` | `GitHubRepositoryService` |
| `github/identity/` | `GitHubAccountLookup`, `GitHubAccountLookupImpl`, `GitHubIdentityBackfill`, `GitHubIdentityBackfillService` |
| `github/commit/` | `GitHubCommitCollector`, `GitHubCommitIngestService`, `GitHubCommitStatsEnrichmentService` |
| `github/pullrequest/` | `GitHubPullRequestCollector`, `GitHubPullRequestIngestService`, `GitHubPullRequestStatsEnrichmentService`, `GitHubPullRequestQueryService` |
| `github/issue/` | `GitHubIssuesCollector`, `AsyncIssuesCollector` |
| `github/scheduler/` | `StatsEnrichmentScheduler` |

`commit/` and `pullrequest/` hold the same three roles — collector, ingest service, stats
enrichment service — so the two-phase pattern is legible from the tree.

---

## 3. Backend — Domain Packages
Expand Down Expand Up @@ -430,7 +447,7 @@ CHECK constraints (added V35):

**`DataSourceCollectService`** — Dispatches to collectors based on `DataSourceType`. For GITHUB type it also conditionally collects issues per repo when `repo.isCollectIssues()` is true:
- `GIT_LOCAL` → `GitLocalCollector.collectForRepository()`
- `GITHUB` → Phase 1: `GitHubCollector.collectForRepository()` (commits), Phase 2: `GitHubPrCollector.collectForRepository()` (PRs), Phase 3 (conditional): `GitHubIssuesCollector.collectIssuesForRepo()` if `repo.collectIssues`
- `GITHUB` → Phase 1: `GitHubCommitCollector.collectForRepository()` (commits), Phase 2: `GitHubPullRequestCollector.collectForRepository()` (PRs), Phase 3 (conditional): `GitHubIssuesCollector.collectIssuesForRepo()` if `repo.collectIssues`
- `GITHUB_ISSUES` → `GitHubIssuesCollector.collectIssuesForRepo(cfg, repoFullName)` per repo
- `JIRA` → iterates `JiraProjectService.listTrackedProjects(cfg)`; calls `JiraCollector.collectIssues(project)` per project. Logs a warning if no tracked projects exist.
- Updates `lastSuccessSync` on success.
Expand Down Expand Up @@ -834,21 +851,23 @@ The datasource a subscription belongs to is derived via `repo_id → git_reposit
- `enrichSingle(entity, ...)` — `GET /repos/{owner}/{repo}/commits/{sha}`. On 200: applies `additions`/`deletions`/`filesChanged`, sets `COMPLETE`. On 403 diff-too-large: sets `SKIPPED`. On 404/422: sets `FAILED`. On 5xx: increments attempts, backs off.
- Rate limiting: synchronized token bucket on `MIN_INTERVAL_NS`; 100–300 ms jitter; proactive throttle when `X-RateLimit-Remaining < 100`; exponential backoff on error.

**`GitHubCollector`** — Orchestrates commit collection:
**`GitHubCommitCollector`** — Orchestrates commit collection:
1. `GitHubCommitIngestService.ingestForRepository()` (Phase A).
2. Sort saved by `authorDate` desc.
3. `GitHubCommitStatsEnrichmentService.enrichImmediate()` (Phase B — top 150).
4. Log remaining PENDING count (Phase C handled by scheduler).

**`GitHubPrCollector`** — Orchestrates PR collection:
1. `GitHubPullRequestCollector` (ingest): pages all PRs, upserts by `(repoId, number)`, saves with `statsStatus=PENDING`.
**`GitHubPullRequestCollector`** — Orchestrates PR collection:
1. `GitHubPullRequestIngestService` (ingest): pages all PRs, upserts by `(repoId, number)`, saves with `statsStatus=PENDING`.
2. Sort saved by `createdAt` desc.
3. `GitHubPrStatsEnrichmentService.enrichImmediate()` (Phase B — top 150): fetches reviews (delete+save in `github_pr_reviews`) and detail stats per PR.
3. `GitHubPullRequestStatsEnrichmentService.enrichImmediate()` (Phase B — top 150): fetches reviews (delete+save in `github_pr_reviews`) and detail stats per PR.
4. Phase C handled by scheduler.

**`GitHubPullRequestCollector`** — Ingest-only service. Pages PRs from GitHub API, upserts into `github_pull_requests`. Also used by `GitHubPullRequestController` for paginated listing. Returns `IngestResult(savedEntities, apiBase, token)`.
**`GitHubPullRequestIngestService`** — Ingest-only service. Pages PRs from GitHub API, upserts into `github_pull_requests`. Returns `IngestResult(savedEntities, apiBase, token)`.

**`GitHubPullRequestQueryService`** — Read side for stored PRs. `listPullRequests(repoId, pageable)` — the paginated listing `GitHubPullRequestController` serves.

**`GitHubPrStatsEnrichmentService`** — Enriches PR stats. Same rate-limit pattern as commit enrichment. `enrichSingle(pr, ...)` — `GET /repos/{owner}/{repo}/pulls/{number}` for additions/deletions/changedFiles/commitsCount; `GET .../reviews` for review list (delete+save all reviews then mark PR `COMPLETE`).
**`GitHubPullRequestStatsEnrichmentService`** — Enriches PR stats. Same rate-limit pattern as commit enrichment. `enrichSingle(pr, ...)` — `GET /repos/{owner}/{repo}/pulls/{number}` for additions/deletions/changedFiles/commitsCount; `GET .../reviews` for review list (delete+save all reviews then mark PR `COMPLETE`).

**`GitHubIssuesCollector`** — Fetches all issues (open + closed) via Kohsuke API, maps to `IssueEntity` with `externalId = owner/repo#{number}`. Upserts by `(dataSourceId, externalId)`. Accepts either a `GitRepositoryEntity` or a bare `repoFullName` string.

Expand Down Expand Up @@ -1618,8 +1637,8 @@ POST /datasources/{id}/collect
│ └─▶ reads lastSuccessSync [before collection stamps it → "first run?"]
│ └─▶ DataSourceCollectService.collectForDataSource()
│ ├─ GIT_LOCAL → GitLocalCollector.collectForRepository()
│ ├─ GITHUB → GitHubCollector.collectForRepository() [commits]
│ │ GitHubPrCollector.collectForRepository() [PRs]
│ ├─ GITHUB → GitHubCommitCollector.collectForRepository() [commits]
│ │ GitHubPullRequestCollector.collectForRepository() [PRs]
│ │ GitHubIssuesCollector (if repo.collectIssues) [issues]
│ ├─ GITHUB_ISSUES → GitHubIssuesCollector.collectIssuesForRepo()
│ └─ JIRA → JiraCollector.collectIssues()
Expand All @@ -1633,14 +1652,14 @@ Client polls `GET /datasources/{id}/collect/status` every 3 seconds. `SyncJobTra

```
Phase A — Fast ingest (seconds)
GitHubCommitIngestService / GitHubPullRequestCollector
GitHubCommitIngestService / GitHubPullRequestIngestService
→ pages all items from GitHub list endpoint (100/page)
→ saves with statsStatus=PENDING
→ returns List<entity> sorted newest-first

Phase B — Immediate enrich top 150 (seconds to minutes)
GitHubCommitStatsEnrichmentService.enrichImmediate() ← 2 worker threads
GitHubPrStatsEnrichmentService.enrichImmediate()
GitHubPullRequestStatsEnrichmentService.enrichImmediate()
→ fetches detail/reviews per item, sets statsStatus=COMPLETE
→ rate-limited: ~1.4 req/s, proactive throttle on X-RateLimit-Remaining < 100

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

import com.juliashtal.devanalytics.git.model.GitRepositoryEntity;
import com.juliashtal.devanalytics.git.repository.GitRepositoryEntityRepository;
import com.juliashtal.devanalytics.github.service.GitHubIdentityBackfill;
import com.juliashtal.devanalytics.github.identity.GitHubIdentityBackfill;
import com.juliashtal.devanalytics.jira.service.JiraIdentityBackfill;
import com.juliashtal.devanalytics.metrics.service.MetricBackfillTrigger;
import com.juliashtal.devanalytics.user.model.User;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@
import com.juliashtal.devanalytics.datasource.service.SyncJobTracker;
import com.juliashtal.devanalytics.git.model.GitRepositoryEntity;
import com.juliashtal.devanalytics.git.repository.GitRepositoryEntityRepository;
import com.juliashtal.devanalytics.github.service.GitHubAccountLookup;
import com.juliashtal.devanalytics.github.service.GitHubCollector;
import com.juliashtal.devanalytics.github.service.GitHubIssuesCollector;
import com.juliashtal.devanalytics.github.service.GitHubPrCollector;
import com.juliashtal.devanalytics.github.identity.GitHubAccountLookup;
import com.juliashtal.devanalytics.github.commit.GitHubCommitCollector;
import com.juliashtal.devanalytics.github.issue.GitHubIssuesCollector;
import com.juliashtal.devanalytics.github.pullrequest.GitHubPullRequestCollector;
import com.juliashtal.devanalytics.user.service.AuthorIdentityService;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
Expand All @@ -24,8 +24,8 @@
public class GitHubSourceCollector implements SourceCollector {

private final GitRepositoryEntityRepository gitRepoRepository;
private final GitHubCollector gitHubCollector;
private final GitHubPrCollector prCollector;
private final GitHubCommitCollector commitCollector;
private final GitHubPullRequestCollector prCollector;
private final GitHubIssuesCollector issuesCollector;
private final GitHubAccountLookup accountLookup;
private final AuthorIdentityService authorIdentityService;
Expand All @@ -51,7 +51,7 @@ public int collect(DataSourceConfig cfg, SyncJobTracker.JobState jobState) {
for (var repo : gitRepoRepository.findAllByDataSourceConfig(cfg)) {
try {
if (jobState != null) tracker.setPhase(jobState, "commits", -1);
int commits = gitHubCollector.collectForRepository(repo.getId(), jobState);
int commits = commitCollector.collectForRepository(repo.getId(), jobState);

if (jobState != null) tracker.setPhase(jobState, "pull requests", -1);
int prs = prCollector.collectForRepository(repo.getId(), jobState);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
import com.juliashtal.devanalytics.git.model.dto.RepoDto;
import com.juliashtal.devanalytics.github.model.dto.DiscoveredRepoDto;
import com.juliashtal.devanalytics.github.model.dto.DiscoveryResult;
import com.juliashtal.devanalytics.github.service.GitHubRepositoryService;
import com.juliashtal.devanalytics.github.repo.GitHubRepositoryService;
import com.juliashtal.devanalytics.security.SecurityUtils;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.tags.Tag;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
import com.juliashtal.devanalytics.git.repository.GitRepositoryEntityRepository;
import com.juliashtal.devanalytics.git.repository.UserRepoRegistrationRepository;
import com.juliashtal.devanalytics.git.service.GitRepositoryService;
import com.juliashtal.devanalytics.github.service.GitHubRepositoryService;
import com.juliashtal.devanalytics.github.repo.GitHubRepositoryService;
import com.juliashtal.devanalytics.jira.service.JiraProjectService;
import com.juliashtal.devanalytics.user.model.Role;
import com.juliashtal.devanalytics.user.model.Team;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

import com.juliashtal.devanalytics.git.model.dto.RepoDto;
import com.juliashtal.devanalytics.git.service.RepoService;
import com.juliashtal.devanalytics.github.service.AsyncIssuesCollectService;
import com.juliashtal.devanalytics.github.issue.AsyncIssuesCollector;
import io.swagger.v3.oas.annotations.Operation;
import lombok.RequiredArgsConstructor;
import org.springframework.http.ResponseEntity;
Expand All @@ -26,7 +26,7 @@
public class RepoController {

private final RepoService repoService;
private final AsyncIssuesCollectService asyncIssuesCollectService;
private final AsyncIssuesCollector asyncIssuesCollector;

@Operation(summary = "List repos visible to the current user (own + team), with subscribed flag")
@GetMapping
Expand Down Expand Up @@ -58,6 +58,6 @@ public RepoDto setCollectIssues(
) {
boolean enabled = Boolean.TRUE.equals(body.get("enabled"));
return repoService.setCollectIssues(repoId, enabled,
asyncIssuesCollectService::collectIssuesForRepo);
asyncIssuesCollector::collectIssuesForRepo);
}
}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package com.juliashtal.devanalytics.github.service;
package com.juliashtal.devanalytics.github.client;

import com.juliashtal.devanalytics.datasource.model.DataSourceConfig;
import com.juliashtal.devanalytics.exception.GitHubException;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package com.juliashtal.devanalytics.github.service;
package com.juliashtal.devanalytics.github.commit;

import com.juliashtal.devanalytics.datasource.service.SyncJobTracker;
import com.juliashtal.devanalytics.git.model.GitCommitEntity;
Expand Down Expand Up @@ -35,7 +35,7 @@
@Service
@RequiredArgsConstructor
@Slf4j
public class GitHubCollector {
public class GitHubCommitCollector {

private final GitHubCommitIngestService ingestService;
private final GitHubCommitStatsEnrichmentService enrichmentService;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package com.juliashtal.devanalytics.github.service;
package com.juliashtal.devanalytics.github.commit;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
Expand All @@ -10,6 +10,7 @@
import com.juliashtal.devanalytics.git.model.StatsStatus;
import com.juliashtal.devanalytics.git.repository.GitCommitEntityRepository;
import com.juliashtal.devanalytics.git.repository.GitRepositoryEntityRepository;
import com.juliashtal.devanalytics.github.client.GitHubClientFactory;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package com.juliashtal.devanalytics.github.service;
package com.juliashtal.devanalytics.github.commit;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
import com.juliashtal.devanalytics.git.model.dto.GitRepositoryDto;
import com.juliashtal.devanalytics.git.model.GitRepositoryEntity;
import com.juliashtal.devanalytics.github.model.dto.RegisterGitHubRepoRequest;
import com.juliashtal.devanalytics.github.service.GitHubCollector;
import com.juliashtal.devanalytics.github.service.GitHubRepositoryService;
import com.juliashtal.devanalytics.github.commit.GitHubCommitCollector;
import com.juliashtal.devanalytics.github.repo.GitHubRepositoryService;
import com.juliashtal.devanalytics.security.SecurityUtils;
import io.swagger.v3.oas.annotations.Operation;
import lombok.RequiredArgsConstructor;
Expand All @@ -23,7 +23,7 @@
public class GitHubController {

private final GitHubRepositoryService gitHubRepositoryService;
private final GitHubCollector gitHubCollector;
private final GitHubCommitCollector commitCollector;

@Operation(summary = "Register a GitHub repository under a data source, reusing an existing entity if already registered")
@PostMapping("/repos")
Expand All @@ -40,7 +40,7 @@ public ResponseEntity<GitRepositoryDto> registerRepo(@RequestBody RegisterGitHub
@Operation(summary = "Synchronously collect new commits for a GitHub repository")
@PostMapping("/repos/{repoId}/collect")
public ResponseEntity<String> collect(@PathVariable Long repoId) {
int saved = gitHubCollector.collectForRepository(repoId, null);
int saved = commitCollector.collectForRepository(repoId, null);
return ResponseEntity.ok("Collected " + saved + " commits from GitHub");
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

import com.juliashtal.devanalytics.github.model.dto.GitHubPullRequestDto;

import com.juliashtal.devanalytics.github.service.GitHubPrCollector;
import com.juliashtal.devanalytics.github.service.GitHubPullRequestCollector;
import com.juliashtal.devanalytics.github.pullrequest.GitHubPullRequestCollector;
import com.juliashtal.devanalytics.github.pullrequest.GitHubPullRequestQueryService;
import io.swagger.v3.oas.annotations.Operation;
import lombok.RequiredArgsConstructor;
import org.springframework.data.domain.Page;
Expand All @@ -22,8 +22,8 @@
@RequiredArgsConstructor
public class GitHubPullRequestController {

private final GitHubPrCollector prCollector;
private final GitHubPullRequestCollector prQueryService;
private final GitHubPullRequestCollector prCollector;
private final GitHubPullRequestQueryService prQueryService;

@Operation(summary = "Synchronously collect new and updated pull requests for a GitHub repository")
@PostMapping("/repos/{repoId}/pull-requests/collect")
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package com.juliashtal.devanalytics.github.service;
package com.juliashtal.devanalytics.github.identity;

import com.juliashtal.devanalytics.datasource.model.DataSourceConfig;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
package com.juliashtal.devanalytics.github.service;
package com.juliashtal.devanalytics.github.identity;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.juliashtal.devanalytics.datasource.model.DataSourceConfig;
import com.juliashtal.devanalytics.datasource.model.DataSourceType;
import com.juliashtal.devanalytics.datasource.repository.DataSourceConfigRepository;
import com.juliashtal.devanalytics.exception.GitHubException;
import com.juliashtal.devanalytics.github.client.GitHubClientFactory;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package com.juliashtal.devanalytics.github.service;
package com.juliashtal.devanalytics.github.identity;

import com.juliashtal.devanalytics.git.model.GitRepositoryEntity;

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package com.juliashtal.devanalytics.github.service;
package com.juliashtal.devanalytics.github.identity;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
Expand All @@ -7,7 +7,10 @@
import com.juliashtal.devanalytics.git.model.GitCommitEntity;
import com.juliashtal.devanalytics.git.model.GitRepositoryEntity;
import com.juliashtal.devanalytics.git.repository.GitCommitEntityRepository;
import com.juliashtal.devanalytics.github.client.GitHubClientFactory;
import com.juliashtal.devanalytics.github.issue.GitHubIssuesCollector;
import com.juliashtal.devanalytics.github.model.GitHubPullRequestEntity;
import com.juliashtal.devanalytics.github.pullrequest.GitHubPullRequestStatsEnrichmentService;
import com.juliashtal.devanalytics.github.repository.GitHubPullRequestRepository;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
Expand Down Expand Up @@ -45,7 +48,7 @@ public class GitHubIdentityBackfillService implements GitHubIdentityBackfill {

private final GitCommitEntityRepository commitRepository;
private final GitHubPullRequestRepository prRepository;
private final GitHubPrStatsEnrichmentService prEnrichmentService;
private final GitHubPullRequestStatsEnrichmentService prEnrichmentService;
private final GitHubIssuesCollector issuesCollector;
private final GitHubClientFactory clientFactory;
private final ObjectMapper objectMapper;
Expand Down
Loading
Loading