Is it a good idea that let the `tokenize` has only one step?

[PeterlitsZo]

Now the tokenize has two steps:

• Parse string and output PartialTokens.
• Parse PartialTokens and output Tokens.

But I can let it have only one step. Now we can do a +/* ignore me please */= 1, but I do not think it is a good idea.

[ISibboI]

That's a good point. Ideally there would be a `PartialToken::Comment` that carries no data, such that those cases can be avoided in the second stage of the parser.

…

On Sat, 4 Apr 2026, 9.41 peterlits zo, ***@***.***> wrote: *PeterlitsZo* created an issue (ISibboI/evalexpr#198) <#198> Now the tokenize has two steps: - Parse string and output PartialTokens. - Parse PartialTokens and output Tokens. But I can let it have only one step. Now we can do a +/* ignore me please */= 1, but I do not think it is a good idea. — Reply to this email directly, view it on GitHub <#198>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASATXX5KIGD5CKLL477QNL4UCVANAVCNFSM6AAAAACXMNPPUCVHI2DSMVQWIX3LMV43ASLTON2WKOZUGIYDGOBYGI4TMOA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>