Lesca - LeetCode Scraper Architecture

A powerful, modular TypeScript-based LeetCode content scraper for creating personal knowledge bases in Obsidian and other markdown-based systems.

⚠️ Legal Notice

Use at your own risk. This tool violates LeetCode's Terms of Service regarding automated access. It is intended strictly for personal, offline use (e.g., studying while disconnected).

Do not redistribute scraped content.
Read LEGAL.md carefully before use.
The authors are not responsible for account bans or other consequences.

✨ Features

Core Functionality

🚀 Comprehensive Scraping: Problems, editorials, discussions, and problem lists
🎯 Type-Safe: Built with strict TypeScript for maximum reliability
📝 Markdown Export: Beautiful, readable Markdown output
💎 Obsidian Ready: Full support with frontmatter and wikilinks
⚡ Smart Rate Limiting: Respectful, configurable API throttling
💾 Intelligent Caching: Speed up re-scraping with tiered cache system
🔧 Highly Configurable: YAML/JSON config with environment variable support
🧩 Plugin System: Extend functionality with custom plugins
🔒 Premium Content: Detects and handles premium-only problems gracefully
🛡️ Robust Error Handling: Automatic retries and backoff for network stability

Developer Experience

🔌 Modular Architecture: Clean separation with swappable components
🧪 Well-Tested: 631 passing tests with 68.43% coverage
📦 Monorepo Structure: Organized, maintainable codebase
🎨 Extensible: Plugin-ready architecture
🔍 Smart Filtering: Filter by difficulty, tags, and limits
🌐 Browser Automation: Playwright-based for dynamic content
📊 Progress Tracking: Resume interrupted batch scrapes

📚 Documentation

For Users

User Guide - Complete usage guide with examples
Installation - Installation instructions
CLI Reference - All commands and options
Configuration - Configuration guide
Examples - Common usage patterns and recipes
Troubleshooting - Common issues and solutions

For Developers

Architecture Review - Design decisions and patterns
Coding Standards - Code style and best practices
Testing Guide - Testing infrastructure and best practices
Agent Guidelines - Guidelines for AI assistants
TypeScript Guide - TypeScript patterns used
Contributing - How to contribute

🚀 Quick Start

Installation

# Clone the repository
git clone https://github.com/yourusername/lesca.git
cd lesca

# Install dependencies
npm install

# Initialize configuration
npm run dev -- init

Set Up Authentication

Run the interactive authentication command:
```
npm run dev -- auth
```
Follow the prompts to provide your cookie file path or paste your cookies.

See User Guide - Authentication Setup for detailed instructions.

Explore Problems

# List easy problems
npm run dev -- list --difficulty Easy

# Search for a specific problem
npm run dev -- search "two sum"

Scrape Your First Problem

# Scrape a single problem
npm run dev -- scrape two-sum

# Scrape multiple problems
npm run dev -- scrape-list --difficulty Medium --limit 10

# Interactive selection
npm run dev -- list --interactive

# Scrape with Obsidian format
npm run dev -- scrape two-sum --format obsidian

Output: Files saved to ./output/ directory by default.

📖 Usage Examples

Scrape Problem Types

# Single problem
npm run dev -- scrape two-sum

# Multiple problems with filtering
npm run dev -- scrape-list \
  --difficulty Medium \
  --tags "dynamic-programming,array" \
  --limit 50

# Editorial (requires browser)
npm run dev -- scrape-editorial two-sum

# Discussions
npm run dev -- scrape-discussions two-sum \
  --category solution \
  --sort most-votes \
  --limit 10

Batch Scraping

# Scrape all Easy problems
npm run dev -- scrape-list --difficulty Easy --concurrency 3

# Resume interrupted scraping
npm run dev -- scrape-list --limit 500 --resume

# Organized by difficulty
npm run dev -- scrape-list --difficulty Hard --output ./hard-problems

# Sort by quality
npm run dev -- list --sort quality --limit 20

Advanced Usage

# Custom configuration
npm run dev -- --config ./custom-config.yaml scrape-list

# Disable caching
npm run dev -- scrape two-sum --no-cache

# Visible browser mode (debugging)
npm run dev -- scrape-editorial two-sum --no-headless

# Manage configuration
npm run dev -- config list
npm run dev -- config get browser.headless

See CLI Reference for all commands and options.

🏗️ Architecture

Monorepo Structure

lesca/
├── packages/
│   ├── core/                # Facade & orchestration layer
│   ├── auth/                # Cookie-based authentication
│   ├── api-client/          # GraphQL client with rate limiting
│   ├── browser-automation/  # Playwright driver
│   ├── scrapers/            # Scraping strategies (Problem, List, Editorial, Discussion)
│   ├── converters/          # HTML to Markdown/Obsidian
│   ├── storage/             # Filesystem & SQLite adapters
│   └── cli/                 # CLI application
├── shared/
│   ├── types/               # Shared TypeScript types
│   ├── config/              # Configuration system
│   └── utils/               # Caching, logging, utilities
├── docs/                    # User & developer documentation
└── examples/                # Example configurations

Design Patterns

Facade Pattern: LeetCodeScraper orchestrates all components
Strategy Pattern: Pluggable scrapers, storage, converters
Singleton Pattern: Configuration manager
Pipeline Pattern: Composable content processing
Builder Pattern: Progressive configuration building

See Architecture Review for detailed analysis.

🧪 Testing

# Run unit tests (fast - runs on every PR)
npm test
# or
npm run test:unit

# Run integration tests (slower - runs on release)
npm run test:integration

# Run all tests (unit + integration)
npm run test:all

# Run with coverage report
npm run test:coverage

# Run tests in watch mode
npm run test:ui

# Check coverage thresholds
npm run check-coverage

# Run performance benchmarks
npm run benchmark

Test Infrastructure

Unit Tests: Fast tests that run on every PR (< 30s target)
Integration Tests: End-to-end tests that run on release
Benchmarks: Performance tracking for critical paths
Factories: Dynamic test data generation
Fixtures: Static test data for consistent testing

⚙️ Configuration

Lesca uses a flexible configuration system supporting YAML/JSON files and environment variables.

Basic Configuration

# lesca.config.yaml
auth:
  cookiePath: ~/.lesca/cookies.json

api:
  rateLimit:
    requestsPerMinute: 30
    minDelay: 2000

storage:
  path: ./output

output:
  format: obsidian # or 'markdown'
  frontmatter: true

scraping:
  concurrency: 3
  batchSize: 10

cache:
  enabled: true
  ttl:
    problem: 604800000 # 7 days

See Configuration Guide for all options.

🛠️ Development

Prerequisites

Node.js >= 18.0.0
npm >= 9.0.0

Setup

# Install dependencies
npm install

# Run in development mode
npm run dev -- scrape two-sum

# Build all packages
npm run build

# Type checking
npm run typecheck

# Linting
npm run lint
npm run lint:fix

Available Scripts

Command	Description
`npm run dev`	Development mode with TypeScript watch
`npm run build`	Build all packages
`npm test`	Run unit tests (fast)
`npm run test:unit`	Run unit tests only
`npm run test:integration`	Run integration tests only
`npm run test:all`	Run all tests (unit + integration)
`npm run test:coverage`	Generate coverage report
`npm run check-coverage`	Validate coverage thresholds
`npm run benchmark`	Run performance benchmarks
`npm run lint`	Lint codebase
`npm run lint:fix`	Fix linting issues
`npm run format`	Format code with Prettier
`npm run typecheck`	Type check without emitting

Testing Guidelines

See Coding Standards for testing best practices.

🗺️ Roadmap

Current Status: v0.1.0 - MVP Complete ✅

✅ Core scraping functionality
✅ Configuration system
✅ CLI with all major commands
✅ Browser automation
✅ Caching system
✅ Comprehensive testing (539 tests)
✅ User documentation

Upcoming Releases

v0.2.0 - Production Ready

CI/CD pipeline (GitHub Actions)
npm package publication
Binary releases (Windows, macOS, Linux)
Docker image

v0.3.0 - Enhanced Features

Quality scoring (Wilson score algorithm)
SQLite storage adapter
Enhanced filtering options
Performance optimizations

v1.0.0 - Stable Release

Plugin system
Web UI (optional)
Advanced analytics
Cloud deployment options

See Roadmap for detailed plans.

📝 License

MIT License - see LICENSE file for details.

🙏 Acknowledgments

LeetCode for providing an excellent platform and GraphQL API
TypeScript and Node.js communities for amazing tools
Playwright for robust browser automation
Vitest for fast, modern testing
All contributors and users of this project

🤝 Contributing

Contributions are welcome! Please read CONTRIBUTING.md before submitting PRs.

Quick Contribution Guide

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Follow Coding Standards
Write tests for new features
Ensure all tests pass (npm test)
Commit changes (git commit -m 'Add amazing feature')
Push to branch (git push origin feature/amazing-feature)
Open a Pull Request

🐛 Bug Reports

Found a bug? Please open an issue with:

Lesca version (npm run dev -- --version)
Node.js version (node --version)
Operating system
Steps to reproduce
Expected vs. actual behavior

See Troubleshooting Guide for common issues.

💬 Support

Documentation: Start with User Guide

Name		Name	Last commit message	Last commit date
Latest commit History 133 Commits
.claude		.claude
.github		.github
.husky		.husky
.roo		.roo
.zencoder/rules		.zencoder/rules
docs		docs
examples		examples
packages		packages
scripts		scripts
shared		shared
tests		tests
.eslintrc.cjs		.eslintrc.cjs
.gitignore		.gitignore
.prettierrc		.prettierrc
.roomodes		.roomodes
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
LEGAL.md		LEGAL.md
PERFORMANCE.md		PERFORMANCE.md
README.md		README.md
SECURITY.md		SECURITY.md
package.json		package.json
requirements.txt		requirements.txt
test-config-new.yaml		test-config-new.yaml
test-graphql.ts		test-graphql.ts
tsconfig.build.json		tsconfig.build.json
tsconfig.json		tsconfig.json
tsup.config.ts		tsup.config.ts
vitest.config.ts		vitest.config.ts
vitest.integration.config.ts		vitest.integration.config.ts
vitest.unit.config.ts		vitest.unit.config.ts

Folders and files

Latest commit

History

Repository files navigation