Contributing to Quicrawl - Guide for LLMs

This document provides guidance for Large Language Models (LLMs) on how to contribute to the Quicrawl project while maintaining code quality, consistency, and architectural integrity.

Project Overview

Quicrawl is a versatile Go-based web crawler that provides both HTTP API and CLI interfaces. It's designed with clean architecture principles, separating concerns between service logic, HTTP handlers, and CLI commands.

Core Architecture

quicrawl/
├── cmd/                    # Entry points
│   ├── cli/main.go        # CLI application entry
│   └── server/main.go     # HTTP server entry
├── internal/              # Private application code
│   ├── service/           # Core business logic
│   ├── http/              # HTTP handlers and routing
│   └── cli/               # CLI commands and interface
├── main.go                # Main entry point
└── go.mod                 # Go module definition

Code Style and Standards

1. Go Conventions

Follow Standard Go Practices:

Use gofmt for consistent formatting
Follow effective Go naming conventions (CamelCase for public, camelCase for private)
Write clear, self-documenting code with appropriate comments
Use meaningful variable and function names

Example of Good Style:

// CrawlResult represents the result of a website crawl
type CrawlResult struct {
    URL        string              `json:"url"`
    HTML       string              `json:"html"`
    StatusCode int                 `json:"status_code"`
    Headers    map[string][]string `json:"headers"`
    Timestamp  time.Time           `json:"timestamp"`
    Error      string              `json:"error,omitempty"`
}

2. Error Handling

Consistent Error Patterns:

Always handle errors explicitly
Provide meaningful error messages with context
Use structured error responses in HTTP handlers
Include error details in result structures when appropriate

Example:

result, err := s.service.CrawlWebsite(ctx, url)
if err != nil {
    return c.JSON(http.StatusInternalServerError, map[string]string{
        "error": err.Error(),
    })
}

3. Documentation Standards

Comment Requirements:

Public functions and types must have godoc comments
Complex logic should include inline comments
API endpoints should be clearly documented
CLI commands should have helpful descriptions

Example:

// CrawlWebsite crawls a website and returns the HTML content.
// It validates the URL, makes an HTTP request, and processes the response.
// Returns a CrawlResult containing the HTML content and metadata.
func (s *Service) CrawlWebsite(ctx context.Context, targetURL string) (*CrawlResult, error) {
    // Implementation...
}

Project Structure Guidelines

1. Service Layer (`internal/service/`)

Purpose: Contains core business logic and domain models.

Guidelines:

Keep business logic independent of HTTP/CLI concerns
Use dependency injection for external dependencies
Implement proper context handling for cancellation
Follow single responsibility principle

When Adding New Features:

Add new methods to the Service struct
Create appropriate result structures
Handle errors gracefully with meaningful messages
Consider performance implications (timeouts, concurrency)

2. HTTP Layer (`internal/http/`)

Purpose: Handles HTTP requests, routing, and responses.

Guidelines:

Keep handlers thin - delegate to service layer
Use consistent request/response structures
Implement proper HTTP status codes
Validate input parameters

Request Structure Pattern:

var request struct {
    URL       string `json:"url"`
    Parameter bool   `json:"parameter,omitempty"`
}

3. CLI Layer (`internal/cli/`)

Purpose: Provides command-line interface using Cobra.

Guidelines:

Use Cobra command structure consistently
Provide helpful descriptions and examples
Implement proper flag handling
Support file output where appropriate

Command Pattern:

cmd := &cobra.Command{
    Use:   "command [args]",
    Short: "Brief description",
    Long:  `Detailed description with examples.`,
    Args:  cobra.ExactArgs(1),
    RunE: func(cmd *cobra.Command, args []string) error {
        // Implementation
    },
}

Feature Development Guidelines

1. Adding New Endpoints

Process:

Add method to service layer with proper error handling
Create HTTP handler that delegates to service
Add CLI command if applicable
Update routing in SetupRoutes
Test the implementation

Example Workflow:

// 1. Service method
func (s *Service) NewFeature(ctx context.Context, param string) (*Result, error) {
    // Implementation
}

// 2. HTTP handler
func (h *Handler) NewFeature(c echo.Context) error {
    var request struct {
        Param string `json:"param"`
    }
    // Handle request, call service, return response
}

// 3. Add route
api.POST("/new-feature", h.NewFeature)

2. Maintaining Backward Compatibility

Critical Rules:

Never change existing API response structures without versioning
Add new fields as omitempty when possible
Preserve existing CLI command behavior
Use feature flags for new functionality

Example of Safe Addition:

type ExistingResult struct {
    URL     string `json:"url"`
    Content string `json:"content"`
    // Safe to add new optional fields
    NewField string `json:"new_field,omitempty"`
}

3. Configuration Management

Current Pattern:

Configuration is handled in the service constructor
Use sensible defaults
Make limits configurable where appropriate

Example:

config := &Config{
    MaxResponseSize: 10 * 1024 * 1024, // 10MB
    MaxLinksToCrawl: 20,
    MaxConcurrency:  10,
    MaxHTMLSize:     1024 * 1024, // 1MB
}

Testing Guidelines

1. Test Structure

Requirements:

Write unit tests for service layer logic
Test error conditions and edge cases
Use table-driven tests for multiple scenarios
Mock external dependencies

Example Test Pattern:

func TestCrawlWebsite(t *testing.T) {
    tests := []struct {
        name     string
        url      string
        expected *CrawlResult
        wantErr  bool
    }{
        {
            name: "valid URL",
            url:  "https://example.com",
            // expected result
        },
        // more test cases
    }
    
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            // test implementation
        })
    }
}

2. Integration Testing

Guidelines:

Test HTTP endpoints with actual requests
Verify CLI commands work correctly
Test error scenarios and edge cases
Ensure proper JSON serialization/deserialization

Performance Considerations

1. Concurrency

Current Patterns:

Use goroutines with semaphores for controlled concurrency
Implement proper context cancellation
Handle timeouts appropriately

Example:

semaphore := make(chan struct{}, maxConcurrency)
var wg sync.WaitGroup

for _, item := range items {
    wg.Add(1)
    go func(item string) {
        defer wg.Done()
        semaphore <- struct{}{} // Acquire
        defer func() { <-semaphore }() // Release
        // Process item
    }(item)
}
wg.Wait()

2. Memory Management

Guidelines:

Implement size limits for responses
Use streaming for large content when possible
Clean up resources properly
Consider memory usage in concurrent operations

Security Considerations

1. Input Validation

Requirements:

Validate all URLs before processing
Sanitize user input
Implement rate limiting where appropriate
Handle malicious content gracefully

2. Resource Limits

Current Protections:

Maximum response size limits
Timeout constraints
Concurrency limits
HTML size restrictions

Common Contribution Patterns

1. Adding Output Formats

Pattern:

Create new result structure
Add formatting method to service
Add parameter to existing endpoints
Update CLI with new flag
Maintain backward compatibility

2. Extending Crawling Capabilities

Guidelines:

Add new methods to service layer
Implement proper error handling
Consider resource usage implications
Provide both HTTP and CLI interfaces

3. Improving Error Handling

Approach:

Enhance error messages with context
Add structured error responses
Implement retry logic where appropriate
Provide debugging information

Code Review Checklist

Before contributing, ensure:

Code follows Go conventions and project style
All public functions have godoc comments
Error handling is consistent and meaningful
Backward compatibility is maintained
Both HTTP and CLI interfaces are updated (if applicable)
Resource limits and timeouts are respected
Code compiles without warnings
Basic functionality testing is performed

Example Contribution: Adding a New Feature

Here's a complete example of adding a new feature following project patterns:

// 1. Add to service layer
func (s *Service) ExtractMetadata(ctx context.Context, url string) (*MetadataResult, error) {
    // Implementation with proper error handling
}

// 2. Add result structure
type MetadataResult struct {
    URL         string            `json:"url"`
    Title       string            `json:"title"`
    Description string            `json:"description"`
    Keywords    []string          `json:"keywords"`
    Timestamp   time.Time         `json:"timestamp"`
    Error       string            `json:"error,omitempty"`
}

// 3. Add HTTP handler
func (h *Handler) ExtractMetadata(c echo.Context) error {
    var request struct {
        URL string `json:"url"`
    }
    // Standard request handling pattern
}

// 4. Add CLI command
func (c *CLI) addMetadataCommand() {
    // Standard CLI command pattern
}

// 5. Update routes
api.POST("/extract-metadata", h.ExtractMetadata)

Getting Started with Contributions

Understand the Codebase: Study the existing patterns and architecture
Start Small: Begin with minor improvements or bug fixes
Follow Patterns: Use existing code as templates for new features
Test Thoroughly: Ensure your changes don't break existing functionality
Document Changes: Update relevant documentation and comments

Resources

Go Documentation: https://golang.org/doc/
Echo Framework: https://echo.labstack.com/
Cobra CLI: https://github.com/spf13/cobra
Project Repository: https://github.com/ja1code/quicrawl

Questions and Support

When contributing to this project:

Study the existing codebase patterns
Follow the established architectural principles
Maintain consistency with existing code style
Ensure backward compatibility
Test your changes thoroughly

Remember: The goal is to enhance the project while maintaining its reliability, performance, and usability for both API and CLI users.

FilesExpand file tree

CONTRIBUTING_FOR_LLMS.md

Latest commit

History

CONTRIBUTING_FOR_LLMS.md

File metadata and controls

Contributing to Quicrawl - Guide for LLMs

Project Overview

Core Architecture

Code Style and Standards

1. Go Conventions

2. Error Handling

3. Documentation Standards

Project Structure Guidelines

1. Service Layer (internal/service/)

2. HTTP Layer (internal/http/)

3. CLI Layer (internal/cli/)

Feature Development Guidelines

1. Adding New Endpoints

2. Maintaining Backward Compatibility

3. Configuration Management

Testing Guidelines

1. Test Structure

2. Integration Testing

Performance Considerations

1. Concurrency

2. Memory Management

Security Considerations

1. Input Validation

2. Resource Limits

Common Contribution Patterns

1. Adding Output Formats

2. Extending Crawling Capabilities

3. Improving Error Handling

Code Review Checklist

Example Contribution: Adding a New Feature

Getting Started with Contributions

Resources

Questions and Support

1. Service Layer (`internal/service/`)

2. HTTP Layer (`internal/http/`)

3. CLI Layer (`internal/cli/`)