Feat/parmanent scraping task#224
Conversation
bullmwSchedulerを導入するため
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request migrates the scraping scheduler from a custom time-range implementation to a distributed queue system using BullMQ and Redis. It includes updates to the Docker configuration, the introduction of a new BullMQScheduler, and a refactor of the scraping logic into worker-based tasks. Feedback focuses on improving robustness and maintainability, specifically by handling potential NaN values in port parsing, correcting the implementation of worker concurrency, simplifying complex TypeScript type inferences, and replacing hardcoded values with dynamic logic or constants.
| ); | ||
| } | ||
| }, | ||
| 6 |
There was a problem hiding this comment.
並行処理数6がマジックナンバーになっています。可読性とメンテナンス性を向上させるために、ファイルの先頭で定数として定義することをお勧めします（例：const WORKER_CONCURRENCY = 6;）。これは241行目の他のワーカーにも適用されます。
| 6 | |
| WORKER_CONCURRENCY |
| logger.log(`Faculty: ${department}`, "error"); | ||
| } | ||
| }, | ||
| 6 |
There was a problem hiding this comment.
並行処理数6がマジックナンバーになっています。可読性とメンテナンス性を向上させるために、ファイルの先頭で定数として定義することをお勧めします（例：const WORKER_CONCURRENCY = 6;）。
| 6 | |
| WORKER_CONCURRENCY |
There was a problem hiding this comment.
Pull request overview
This PR migrates syllabus-scrape task scheduling from an in-memory time-range loop to a BullMQ (Redis-backed) queue so scheduled scraping jobs can persist across process restarts. It also changes the Scheduler interface to support named workers and typed task payloads (breaking change).
Changes:
- Introduces
BullMQScheduler(BullMQ + Redis) and wires it intosrc/main.ts. - Reworks
Scheduler/TimeRangeSchedulerto use{ type, payload }tasks plus worker registration. - Adds Redis service + env wiring in
docker-compose.yml, and addsbullmqdependency.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| syllabus-scrape/src/scheduler/timeRangeScheduler.ts | Updates in-memory scheduler to dispatch TaskPayload via registered workers |
| syllabus-scrape/src/scheduler/core.ts | Breaking interface change: addWorker + TaskPayload |
| syllabus-scrape/src/scheduler/bullmqScheduler.ts | New BullMQ-backed scheduler implementation |
| syllabus-scrape/src/main.ts | Switches scraping orchestration to BullMQScheduler workers/queues |
| syllabus-scrape/package.json | Adds bullmq dependency |
| syllabus-scrape/pnpm-lock.yaml | Locks BullMQ + transitive deps |
| docker-compose.yml | Adds Redis service/volume and Redis env vars for syllabus-scrape |
Files not reviewed (1)
- syllabus-scrape/pnpm-lock.yaml: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ): void; | ||
| addTask(task: TaskPayload): void; | ||
| start(): void; | ||
| stop(): void; |
| export class BullMQScheduler implements Scheduler { | ||
| private workers: Record<string, Worker> = {}; | ||
| private queues: Record<string, Queue> = {}; | ||
| private redisConfig: ConnectionOptions; | ||
|
|
||
| constructor(connectionOpts: ConnectionOptions) { | ||
| this.redisConfig = connectionOpts; | ||
| } | ||
|
|
| async (taskPayload) => { | ||
| const { syllabusSearchResult } = taskPayload as { | ||
| syllabusSearchResult: ReturnType< | ||
| typeof scrapeSyllabusSearchResult | ||
| > extends Promise<infer U> | ||
| ? U extends (infer V)[] | ||
| ? V | ||
| : never | ||
| : never; | ||
| }; |
より安全のため Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
読みにくいため
型アサーションを避けるため
予期せぬ動作を防ぐため
平行処理数と勘違いされたため
正常に処理を止めるため
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
変更内容
syllabus-scrapeのスケジューラに登録されたタスクを永続化した。変更の種類
関連Issue
Closes #
変更の詳細
BullMQを使ってタスクスケジューラを実装。
TimeRangeSchedulerのインターフェイスを変更したためTimeRangeSchedulerの実装やmain.tsの実装も変更した。テスト
テスト内容
チェックリスト
スクリーンショット
その他