Skip to content

TODO_2024_09_25.md

Latest commit

 

History

History
112 lines (91 loc) · 3.92 KB

TODO_2024_09_25.md

File metadata and controls

112 lines (91 loc) · 3.92 KB

Development Plan - September 25, 2024

Today's Achievements (Sept 24)

  • ✅ Added support for multiple JOIN conditions with AND
  • ✅ Implemented detailed execution plan phase tracking
  • ✅ Massive WIN: 6x GROUP BY performance improvement by reusing ArithmeticEvaluator
  • ✅ Released v1.54.0 with these improvements

Tomorrow's Focus Areas

1. 🔧 Clean Up Hardcoded Keywords in Parser

Priority: HIGH File: src/sql/recursive_parser.rs

Current Issues:

  • Keywords are hardcoded with string comparisons throughout
  • Multiple to_uppercase() calls creating unnecessary allocations
  • Pattern matching like trimmed.to_uppercase().ends_with(" AND") (lines 1880-1885)
  • Inconsistent case handling

Tasks:

  • Audit all hardcoded keyword checks in recursive_parser.rs
  • Create a centralized keyword token system or use existing Token enum better
  • Replace string comparisons with token comparisons where possible
  • Eliminate redundant to_uppercase() calls
  • Consider creating a Keywords module with constants

Example Problem Areas:

// Current (inefficient):
if trimmed.to_uppercase().ends_with(" AND") ||
   trimmed.to_uppercase().ends_with(" OR") {

// Better:
if matches!(last_token, Token::And | Token::Or) {

2. 🌐 Extract WEB CTE into Dedicated Module

Priority: HIGH Current Location: Mixed into general CTE handling

Tasks:

  • Create new module: src/sql/web_cte/mod.rs
  • Move WebCTESpec and related types
  • Extract WEB CTE parsing logic from recursive_parser
  • Extract WEB CTE execution from query_engine
  • Improve error handling for network operations
  • Add retry logic and timeout configuration
  • Create comprehensive tests for WEB CTE

Module Structure:

src/sql/web_cte/
├── mod.rs          # Public API
├── parser.rs       # WEB CTE specific parsing
├── executor.rs     # HTTP request execution
├── cache.rs        # Caching logic
└── tests.rs        # Unit tests

Future Enhancements (if time permits):

  • Support for more protocols (WebSocket, GraphQL)
  • Better authentication mechanisms
  • Streaming support for large responses
  • Parallel fetching for multiple WEB CTEs

3. 🚀 Additional Optimization Opportunities

Priority: MEDIUM (if time permits)

Quick Wins to Look For:

  • Check for similar patterns to ArithmeticEvaluator creation
  • Look for repeated allocations in hot paths
  • Special fast path for simple column references in GROUP BY
  • Cache compiled regular expressions

Parser Performance:

  • Reduce temporary string allocations
  • Pre-size vectors where possible
  • Consider string interning for common identifiers

4. 📝 Documentation Updates

Priority: LOW (can be done alongside)

  • Update CLAUDE.md with new execution plan features
  • Document the WEB CTE module architecture
  • Add performance tuning guide based on today's findings

Success Metrics

  • Parser should have fewer string allocations and to_uppercase() calls
  • WEB CTE code should be in its own module with clear interfaces
  • Tests should continue to pass
  • Benchmarks should show no performance regression

Notes

  • Start with keyword cleanup as it's foundational
  • WEB CTE extraction is well-defined and good for code organization
  • Keep an eye out for more "easy wins" like today's GROUP BY optimization
  • Consider creating benchmarks for parser performance

Questions to Consider

  1. Should we create a lexer keyword cache?
  2. Should WEB CTEs support connection pooling?
  3. Is it worth creating a query optimization framework?

Code Quality Goals

  • Reduce code duplication
  • Improve separation of concerns
  • Make the codebase more maintainable
  • Keep performance improvements from today

Ready to tackle these improvements tomorrow! The keyword cleanup will make the parser cleaner and more efficient, while the WEB CTE extraction will improve modularity.