This document lists potential new features for lsweb, ordered by implementation difficulty (easiest to hardest).
-
Enhanced Progress Bars
- Add ETA and speed indicators to progress bars
- Color-coded progress bars based on speed
- File size display in human-readable format
- Overall progress bar for batch downloads
-
Output Filtering and Sorting
- Sort links by file size, type, or timestamp
- Filter by file types (e.g., only
.pdfor.zipfiles) - Option to only show links with direct downloads
-
Download to Specific Directory
- Allow specifying output directory for downloads
- Create directory structure if it doesn't exist
- Preserve directory structure from URLs
-
Advanced Filtering
- Multiple regex patterns with AND/OR logic
- Content-type filtering
- Size-based filtering before downloading
-
Export/Import Capabilities
- Export link lists to various formats (CSV, Markdown)
- Import URLs from text files
- Generate shell scripts for downloading with other tools
-
Enhanced GitHub Integration
- Filter by specific GitHub release versions (latest, specific tag)
- Better error handling for API rate limits
- Include additional GitHub metadata (release notes, timestamps)
-
Proxy Support
- Configure HTTP/SOCKS proxies
- Rotate between multiple proxies for large downloads
- Add proxy health checking
-
Resume Interrupted Downloads
- Support for HTTP Range requests
- Track partially downloaded files
- Option to resume all interrupted downloads
-
Basic Authentication Support
- Basic auth for accessing protected sites
- Cookie-based auth with cookie jar storage
- Command-line credential input
-
Caching Mechanism
- Cache parsed URLs for faster repeat access
- Store previous download lists for comparison
- Cache HTTP responses with proper invalidation
-
Checksum Verification
- Automatically find and verify checksums when available
- Support multiple hash algorithms (MD5, SHA1, SHA256)
- Generate checksums for downloaded files
-
Content Processing
- Extract links from more formats (PDF, docx)
- Recursive crawling with depth control
- Extract metadata like titles along with links
-
Web Interface
- Simple local web UI for interactive use
- Real-time progress visualization
- Drag-and-drop file selection
- Mobile-friendly design
-
Performance Improvements
- Distributed downloading across machines
- Smart throttling to avoid overwhelming servers
- Streaming content processing for large files
-
Integration with Other Tools
- Webhooks to notify on completion
- JSON API for programmatic access
- Plugin system for custom link extraction
-
Daemon Mode
- Run lsweb as a background service/daemon
- Auto-start capability on system boot
- Logging and rotation system for long-running instances
- Graceful shutdown and restart capabilities
-
Client-Daemon Communication
- File/directory polling interface for job submission
- Unix socket communication protocol
- TCP/IP interface with optional TLS
- Command-line client to interact with daemon
-
OAuth Integration
- Support for various OAuth providers
- Token management and refresh
- Secure credential storage
-
Advanced Download Management
- Priority-based download queue
- Bandwidth scheduling and throttling
- Smart retry with exponential backoff
-
Plugin System
- Extensible architecture for custom site handlers
- Developer documentation for plugins
- Plugin marketplace or registry
-
Advanced Daemon Features
- Multi-client support with authentication
- RESTful API for remote management
- Job queuing and scheduling system
- Resource usage monitoring and alerting
- Cluster mode for distributed downloading across multiple machines