Skip to content

Latest commit

 

History

History
124 lines (97 loc) · 4.05 KB

requested-features.md

File metadata and controls

124 lines (97 loc) · 4.05 KB

Requested Features for lsweb

This document lists potential new features for lsweb, ordered by implementation difficulty (easiest to hardest).

Easy to Implement

  1. Enhanced Progress Bars

    • Add ETA and speed indicators to progress bars
    • Color-coded progress bars based on speed
    • File size display in human-readable format
    • Overall progress bar for batch downloads
  2. Output Filtering and Sorting

    • Sort links by file size, type, or timestamp
    • Filter by file types (e.g., only .pdf or .zip files)
    • Option to only show links with direct downloads
  3. Download to Specific Directory

    • Allow specifying output directory for downloads
    • Create directory structure if it doesn't exist
    • Preserve directory structure from URLs
  4. Advanced Filtering

    • Multiple regex patterns with AND/OR logic
    • Content-type filtering
    • Size-based filtering before downloading
  5. Export/Import Capabilities

    • Export link lists to various formats (CSV, Markdown)
    • Import URLs from text files
    • Generate shell scripts for downloading with other tools

Moderate Difficulty

  1. Enhanced GitHub Integration

    • Filter by specific GitHub release versions (latest, specific tag)
    • Better error handling for API rate limits
    • Include additional GitHub metadata (release notes, timestamps)
  2. Proxy Support

    • Configure HTTP/SOCKS proxies
    • Rotate between multiple proxies for large downloads
    • Add proxy health checking
  3. Resume Interrupted Downloads

    • Support for HTTP Range requests
    • Track partially downloaded files
    • Option to resume all interrupted downloads
  4. Basic Authentication Support

    • Basic auth for accessing protected sites
    • Cookie-based auth with cookie jar storage
    • Command-line credential input
  5. Caching Mechanism

    • Cache parsed URLs for faster repeat access
    • Store previous download lists for comparison
    • Cache HTTP responses with proper invalidation

Hard to Implement

  1. Checksum Verification

    • Automatically find and verify checksums when available
    • Support multiple hash algorithms (MD5, SHA1, SHA256)
    • Generate checksums for downloaded files
  2. Content Processing

    • Extract links from more formats (PDF, docx)
    • Recursive crawling with depth control
    • Extract metadata like titles along with links
  3. Web Interface

    • Simple local web UI for interactive use
    • Real-time progress visualization
    • Drag-and-drop file selection
    • Mobile-friendly design
  4. Performance Improvements

    • Distributed downloading across machines
    • Smart throttling to avoid overwhelming servers
    • Streaming content processing for large files
  5. Integration with Other Tools

    • Webhooks to notify on completion
    • JSON API for programmatic access
    • Plugin system for custom link extraction

Hard to Implement (continued)

  1. Daemon Mode

    • Run lsweb as a background service/daemon
    • Auto-start capability on system boot
    • Logging and rotation system for long-running instances
    • Graceful shutdown and restart capabilities
  2. Client-Daemon Communication

    • File/directory polling interface for job submission
    • Unix socket communication protocol
    • TCP/IP interface with optional TLS
    • Command-line client to interact with daemon

Very Challenging

  1. OAuth Integration

    • Support for various OAuth providers
    • Token management and refresh
    • Secure credential storage
  2. Advanced Download Management

    • Priority-based download queue
    • Bandwidth scheduling and throttling
    • Smart retry with exponential backoff
  3. Plugin System

    • Extensible architecture for custom site handlers
    • Developer documentation for plugins
    • Plugin marketplace or registry
  4. Advanced Daemon Features

    • Multi-client support with authentication
    • RESTful API for remote management
    • Job queuing and scheduling system
    • Resource usage monitoring and alerting
    • Cluster mode for distributed downloading across multiple machines