I originally created this to create tokenz, but I'm finding myself more and more using this to walk through text without generating tokenz.
Because of this I might want to name this project as if it walks through text, not just a tokenizer library.
Because of that I'm not entirely sure I should rename this to walk-, but the one thing preventing me from doing so is that I have a package called tokenz-html. I like the idea of having different tokenizers based on this tokenz library.
Maybe I could rename it to a name that implies it deals with strings.
I originally created this to create tokenz, but I'm finding myself more and more using this to walk through text without generating tokenz.
Because of this I might want to name this project as if it walks through text, not just a tokenizer library.
Because of that I'm not entirely sure I should rename this to walk-, but the one thing preventing me from doing so is that I have a package called tokenz-html. I like the idea of having different tokenizers based on this tokenz library.
Maybe I could rename it to a name that implies it deals with strings.