Prepare v0.2.0 release

CHANGELOG.md

@@ -5,6 +5,22 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.2.0] - 2026-07-24
+
+### Added
+
+- `comments: true` option for `ShellWords.split/2` and `ShellWords.split!/2`.
+  An unquoted, unescaped `#` starts a comment only at the start of a word;
+  `echo hello#world` remains one word.
+- Compatibility tests for supported behavior shared with Python `shlex` and
+  Ruby `Shellwords`.
+
+### Changed
+
+- Documented that invalid UTF-8 bytes are preserved by `split/2`.
+- Documented that backslash-newline is preserved literally rather than
+  treated as line continuation.
+
 ## [0.1.0] - 2026-07-05
 
 ### Added
@@ -22,4 +38,5 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - StreamData property tests for the `split(join(argv)) == {:ok, argv}`
   round-trip guarantee.
 
+[0.2.0]: https://github.com/ivan-podgurskiy/shell_words/releases/tag/v0.2.0
 [0.1.0]: https://github.com/ivan-podgurskiy/shell_words/releases/tag/v0.1.0

README.md

@@ -1,6 +1,8 @@
 # ShellWords
 
 [![CI](https://github.com/ivan-podgurskiy/shell_words/actions/workflows/ci.yml/badge.svg)](https://github.com/ivan-podgurskiy/shell_words/actions/workflows/ci.yml)
+[![Hex.pm](https://img.shields.io/hexpm/v/shell_words.svg)](https://hex.pm/packages/shell_words)
+[![HexDocs](https://img.shields.io/badge/hex-docs-blue.svg)](https://hexdocs.pm/shell_words)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
 
 POSIX-like shell word splitting, escaping, and joining for Elixir — the
@@ -22,7 +24,7 @@ Add `shell_words` to your dependencies in `mix.exs`:
 ```elixir
 def deps do
   [
-    {:shell_words, "~> 0.1.0"}
+    {:shell_words, "~> 0.2.0"}
   ]
 end
 ```
@@ -52,6 +54,17 @@ backslash semantics (`\` escapes `$`, `` ` ``, `"`, `\`; any other
 backslash sequence passes through literally); backslash escaping outside
 quotes; empty quoted words; adjacent segment concatenation.
 
+Comments are disabled by default. Pass `comments: true` to treat an
+unquoted, unescaped `#` as a comment starter only at the start of a word:
+
+```elixir
+ShellWords.split("echo hello # comment", comments: true)
+# {:ok, ["echo", "hello"]}
+
+ShellWords.split("echo hello#world", comments: true)
+# {:ok, ["echo", "hello#world"]}
+```
+
 ### split!/2
 
 Same as `split/2` but returns the list directly and raises
@@ -112,10 +125,9 @@ or PowerShell escaping.
 
 No command execution, pipes, redirects, variable expansion (`$HOME`),
 command substitution (`$(...)`), globbing (`*.txt`), tilde expansion,
-heredocs, or Bash-specific syntax. Comments are not parsed in v0.1.0: `#`
-is an ordinary character (a `comments:` option may be added in a future version). Backslash-newline
-line continuation is not supported. Behavior on invalid UTF-8 is
-unspecified.
+heredocs, or Bash-specific syntax. Backslash-newline line continuation is
+not supported; an escaped newline is preserved literally. `split/2`
+preserves invalid UTF-8 bytes instead of validating or rewriting them.
 
 ## Comparison with Python shlex and Ruby Shellwords
 

ROADMAP.md

@@ -3,9 +3,9 @@
 Planned direction for `shell_words`. Dates are intentions, not promises;
 scope may shift based on feedback and real-world usage.
 
-## v0.1.0 — Initial release (current)
+## v0.1.0 — Initial release
 
-Status: complete, pending publication to Hex.
+Status: complete.
 
 - `split/2` and `split!/2` with POSIX-like word rules: whitespace
   separation, single quotes, double quotes with POSIX backslash semantics,
@@ -17,21 +17,20 @@ Status: complete, pending publication to Hex.
 - Property-tested round-trip guarantee:
   `split(join(argv)) == {:ok, argv}` for arbitrary valid-UTF-8 argv.
 
-## v0.2.0 — Comments and edge-case decisions
+## v0.2.0 — Comments and edge-case decisions (current)
 
+- Status: complete, pending publication to Hex.
 - `comments: true` option for `split/2` and `split!/2`. Semantics: an
   unquoted, unescaped `#` starts a comment only at the start of a word
   (start of input or immediately after unquoted whitespace); the comment
   runs to the next newline or end of input. `echo hello#world` does NOT
   start a comment; `echo hello #world` does. Default stays
   `comments: false`.
-- Decide and document invalid-UTF-8 handling: explicit error vs byte
-  pass-through (currently unspecified).
-- Decide and document backslash-newline continuation (currently a literal
-  newline outside quotes mid-input, `:trailing_escape` at end of input,
-  literal pass-through inside double quotes).
-- Compatibility test suite comparing outputs against Python `shlex` and
-  Ruby `Shellwords`.
+- Invalid UTF-8 handling documented as byte pass-through for `split/2`.
+- Backslash-newline behavior documented as literal preservation, not line
+  continuation.
+- Compatibility test suite compares supported outputs against Python `shlex`
+  and Ruby `Shellwords`.
 
 ## v1.0.0 — Stable API
 

lib/shell_words.ex

@@ -21,7 +21,7 @@ defmodule ShellWords do
 
   This is not a shell interpreter. There is no command execution, pipe or
   redirect handling, variable expansion, command substitution, globbing,
-  tilde expansion, or comment parsing. `#` is an ordinary character.
+  or tilde expansion. Comment parsing is opt-in with `comments: true`.
 
   Only POSIX-like shells are targeted; there is no Windows `cmd.exe` or
   PowerShell escaping.
@@ -44,9 +44,9 @@ defmodule ShellWords do
 
   Word separators are exactly ASCII space, tab, newline, and carriage
   return; Unicode whitespace is an ordinary character. Backslash-newline is
-  not line continuation: the newline is kept as a literal character. Error
-  positions are 0-based byte offsets. Behavior on invalid UTF-8 input is
-  unspecified and may change.
+  not line continuation: the newline is kept as a literal character. Invalid
+  UTF-8 bytes are preserved by `split/2`. Error positions are 0-based byte
+  offsets.
   """
 
   alias ShellWords.ParseError
@@ -60,8 +60,13 @@ defmodule ShellWords do
 
   Returns `{:ok, words}` or `{:error, %ShellWords.ParseError{}}`.
 
-  No options are supported yet; the `opts` argument exists to keep the arity
-  stable for future releases. Unknown options raise `ArgumentError`.
+  Supported options:
+
+    * `:comments` — when `true`, an unquoted, unescaped `#` starts a comment
+      only at the start of a word. The comment runs to the next newline or
+      the end of input. Defaults to `false`.
+
+  Unknown options raise `ArgumentError`.
 
   ## Examples
 
@@ -71,13 +76,17 @@ defmodule ShellWords do
       iex> ShellWords.split("")
       {:ok, []}
 
+      iex> ShellWords.split("echo hello # comment", comments: true)
+      {:ok, ["echo", "hello"]}
+
   """
   @spec split(String.t(), keyword()) :: {:ok, [String.t()]} | {:error, ParseError.t()}
   def split(input, opts \\ [])
 
   def split(input, opts) when is_binary(input) and is_list(opts) do
-    validate_opts!(opts)
-    bare(input, 0, "", [], false)
+    comments? = validate_opts!(opts)
+
+    bare(input, 0, "", [], false, comments?)
   end
 
   @doc """
@@ -154,9 +163,19 @@ defmodule ShellWords do
   defp safe_bare_word?(<<c, rest::binary>>) when c in @safe_chars, do: safe_bare_word?(rest)
   defp safe_bare_word?(_), do: false
 
-  defp validate_opts!([]), do: :ok
+  defp validate_opts!(opts), do: parse_opts(opts, false)
+
+  defp parse_opts([], comments?), do: comments?
+
+  defp parse_opts([{:comments, comments?} | opts], _comments?) when is_boolean(comments?) do
+    parse_opts(opts, comments?)
+  end
 
-  defp validate_opts!([opt | _]) do
+  defp parse_opts([{:comments, value} | _opts], _comments?) do
+    raise ArgumentError, "expected :comments to be a boolean, got: #{inspect(value)}"
+  end
+
+  defp parse_opts([opt | _opts], _comments?) do
     raise ArgumentError, "unknown option: #{inspect(opt)}"
   end
 
@@ -170,76 +189,86 @@ defmodule ShellWords do
   # State: bare — outside any quotes.
   # word is the word built so far; started? distinguishes "no word open"
   # from "word open but currently empty" (needed for '' and "").
-  defp bare(<<>>, _pos, word, acc, started?) do
+  defp bare(<<>>, _pos, word, acc, started?, _comments?) do
     {:ok, Enum.reverse(finish_word(word, acc, started?))}
   end
 
-  defp bare(<<"\\">>, pos, _word, _acc, _started?) do
+  defp bare(<<"\\">>, pos, _word, _acc, _started?, _comments?) do
     {:error, ParseError.new(:trailing_escape, pos)}
   end
 
-  defp bare(<<"\\", c, rest::binary>>, pos, word, acc, _started?) do
-    bare(rest, pos + 2, <<word::binary, c>>, acc, true)
+  defp bare(<<"\\", c, rest::binary>>, pos, word, acc, _started?, comments?) do
+    bare(rest, pos + 2, <<word::binary, c>>, acc, true, comments?)
+  end
+
+  defp bare(<<"'", rest::binary>>, pos, word, acc, _started?, comments?) do
+    single(rest, pos + 1, pos, word, acc, comments?)
   end
 
-  defp bare(<<"'", rest::binary>>, pos, word, acc, _started?) do
-    single(rest, pos + 1, pos, word, acc)
+  defp bare(<<"\"", rest::binary>>, pos, word, acc, _started?, comments?) do
+    double(rest, pos + 1, pos, word, acc, comments?)
   end
 
-  defp bare(<<"\"", rest::binary>>, pos, word, acc, _started?) do
-    double(rest, pos + 1, pos, word, acc)
+  defp bare(<<"#", rest::binary>>, pos, word, acc, false, true) do
+    {rest, pos} = skip_comment(rest, pos + 1)
+
+    bare(rest, pos, word, acc, false, true)
   end
 
-  defp bare(<<c, rest::binary>>, pos, word, acc, started?) when c in @whitespace do
-    bare(rest, pos + 1, "", finish_word(word, acc, started?), false)
+  defp bare(<<c, rest::binary>>, pos, word, acc, started?, comments?) when c in @whitespace do
+    bare(rest, pos + 1, "", finish_word(word, acc, started?), false, comments?)
   end
 
-  defp bare(<<c, rest::binary>>, pos, word, acc, _started?) do
-    bare(rest, pos + 1, <<word::binary, c>>, acc, true)
+  defp bare(<<c, rest::binary>>, pos, word, acc, _started?, comments?) do
+    bare(rest, pos + 1, <<word::binary, c>>, acc, true, comments?)
   end
 
   # State: single — inside single quotes. Everything is literal until the
   # closing quote; open_pos is the offset of the opening quote.
-  defp single(<<>>, _pos, open_pos, _word, _acc) do
+  defp single(<<>>, _pos, open_pos, _word, _acc, _comments?) do
     {:error, ParseError.new(:unterminated_single_quote, open_pos)}
   end
 
-  defp single(<<"'", rest::binary>>, pos, _open_pos, word, acc) do
-    bare(rest, pos + 1, word, acc, true)
+  defp single(<<"'", rest::binary>>, pos, _open_pos, word, acc, comments?) do
+    bare(rest, pos + 1, word, acc, true, comments?)
   end
 
-  defp single(<<c, rest::binary>>, pos, open_pos, word, acc) do
-    single(rest, pos + 1, open_pos, <<word::binary, c>>, acc)
+  defp single(<<c, rest::binary>>, pos, open_pos, word, acc, comments?) do
+    single(rest, pos + 1, open_pos, <<word::binary, c>>, acc, comments?)
   end
 
   # State: double — inside double quotes. Backslash escapes only
   # @dquote_escapable; any other backslash sequence passes through with the
   # backslash preserved. open_pos is the offset of the opening quote.
-  defp double(<<>>, _pos, open_pos, _word, _acc) do
+  defp double(<<>>, _pos, open_pos, _word, _acc, _comments?) do
     {:error, ParseError.new(:unterminated_double_quote, open_pos)}
   end
 
-  defp double(<<"\\">>, _pos, open_pos, _word, _acc) do
+  defp double(<<"\\">>, _pos, open_pos, _word, _acc, _comments?) do
     {:error, ParseError.new(:unterminated_double_quote, open_pos)}
   end
 
-  defp double(<<"\\", c, rest::binary>>, pos, open_pos, word, acc)
+  defp double(<<"\\", c, rest::binary>>, pos, open_pos, word, acc, comments?)
        when c in @dquote_escapable do
-    double(rest, pos + 2, open_pos, <<word::binary, c>>, acc)
+    double(rest, pos + 2, open_pos, <<word::binary, c>>, acc, comments?)
   end
 
-  defp double(<<"\\", c, rest::binary>>, pos, open_pos, word, acc) do
-    double(rest, pos + 2, open_pos, <<word::binary, ?\\, c>>, acc)
+  defp double(<<"\\", c, rest::binary>>, pos, open_pos, word, acc, comments?) do
+    double(rest, pos + 2, open_pos, <<word::binary, ?\\, c>>, acc, comments?)
   end
 
-  defp double(<<"\"", rest::binary>>, pos, _open_pos, word, acc) do
-    bare(rest, pos + 1, word, acc, true)
+  defp double(<<"\"", rest::binary>>, pos, _open_pos, word, acc, comments?) do
+    bare(rest, pos + 1, word, acc, true, comments?)
   end
 
-  defp double(<<c, rest::binary>>, pos, open_pos, word, acc) do
-    double(rest, pos + 1, open_pos, <<word::binary, c>>, acc)
+  defp double(<<c, rest::binary>>, pos, open_pos, word, acc, comments?) do
+    double(rest, pos + 1, open_pos, <<word::binary, c>>, acc, comments?)
   end
 
+  defp skip_comment(<<>>, pos), do: {<<>>, pos}
+  defp skip_comment(<<"\n", rest::binary>>, pos), do: {rest, pos + 1}
+  defp skip_comment(<<_c, rest::binary>>, pos), do: skip_comment(rest, pos + 1)
+
   defp finish_word(_word, acc, false), do: acc
   defp finish_word(word, acc, true), do: [word | acc]
 end

mix.exs

@@ -1,7 +1,7 @@
 defmodule ShellWords.MixProject do
   use Mix.Project
 
-  @version "0.1.0"
+  @version "0.2.0"
   @source_url "https://github.com/ivan-podgurskiy/shell_words"
 
   def project do