Skip to content

Refactor: Lexer string reading code duplication #149

Description

@clank-hayen

Problem

In `lexer/lexer.cpp`, there are four nearly identical string reading functions:

  1. read_string() (lines 106-138): Double-quoted with escapes
  2. read_single_quoted() (lines 145-177): Single-quoted with escapes
  3. read_raw_string() (lines 183-199): Raw double-quoted
  4. read_raw_single_quoted() (lines 205-221): Raw single-quoted

Functions 1 and 2 differ only in the quote character ('"' vs '\'').
Functions 3 and 4 differ only in the quote character.

Suggested Solution

Consolidate into two functions with a quote parameter:

Token read_string_literal(char quote_char, bool handle_escapes) {
    int start_line = line;
    advance();  // skip opening quote
    string value;

    while (current() != quote_char && current() != '\0') {
        if (handle_escapes && current() == '\\\\') {
            advance();
            switch (current()) {
                case 'n': value += '\\n'; break;
                case 't': value += '\\t'; break;
                case 'r': value += '\\r'; break;
                case '\\\\': value += '\\\\'; break;
                case '\"': value += '\"'; break;
                case '\\'': value += '\\''; break;
                default: value += current(); break;
            }
            advance();
        } else {
            value += current();
            advance();
        }
    }

    if (current() == '\0') {
        throw runtime_error("Unterminated string literal at line " + to_string(start_line));
    }

    advance();  // skip closing quote
    return {TokenType::STRING, value, start_line};
}

Usage:

// In tokenize():
if (current() == '\"') {
    tokens.push_back(read_string_literal('\"', true));
}
if (current() == '\\'') {
    tokens.push_back(read_string_literal('\\'', true));
}
// For raw strings: read_string_literal(quote, false)

Impact

  • Reduces ~80 lines to ~30 lines
  • Single place to add new escape sequences
  • Consistent behavior across string types

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions