diff --git a/CHANGELOG.md b/CHANGELOG.md index d873b1f..fde31ce 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,22 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.2.0] - 2026-07-24 + +### Added + +- `comments: true` option for `ShellWords.split/2` and `ShellWords.split!/2`. + An unquoted, unescaped `#` starts a comment only at the start of a word; + `echo hello#world` remains one word. +- Compatibility tests for supported behavior shared with Python `shlex` and + Ruby `Shellwords`. + +### Changed + +- Documented that invalid UTF-8 bytes are preserved by `split/2`. +- Documented that backslash-newline is preserved literally rather than + treated as line continuation. + ## [0.1.0] - 2026-07-05 ### Added @@ -22,4 +38,5 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - StreamData property tests for the `split(join(argv)) == {:ok, argv}` round-trip guarantee. +[0.2.0]: https://github.com/ivan-podgurskiy/shell_words/releases/tag/v0.2.0 [0.1.0]: https://github.com/ivan-podgurskiy/shell_words/releases/tag/v0.1.0 diff --git a/README.md b/README.md index 500f355..4294075 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ # ShellWords [![CI](https://github.com/ivan-podgurskiy/shell_words/actions/workflows/ci.yml/badge.svg)](https://github.com/ivan-podgurskiy/shell_words/actions/workflows/ci.yml) +[![Hex.pm](https://img.shields.io/hexpm/v/shell_words.svg)](https://hex.pm/packages/shell_words) +[![HexDocs](https://img.shields.io/badge/hex-docs-blue.svg)](https://hexdocs.pm/shell_words) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) POSIX-like shell word splitting, escaping, and joining for Elixir — the @@ -22,7 +24,7 @@ Add `shell_words` to your dependencies in `mix.exs`: ```elixir def deps do [ - {:shell_words, "~> 0.1.0"} + {:shell_words, "~> 0.2.0"} ] end ``` @@ -52,6 +54,17 @@ backslash semantics (`\` escapes `$`, `` ` ``, `"`, `\`; any other backslash sequence passes through literally); backslash escaping outside quotes; empty quoted words; adjacent segment concatenation. +Comments are disabled by default. Pass `comments: true` to treat an +unquoted, unescaped `#` as a comment starter only at the start of a word: + +```elixir +ShellWords.split("echo hello # comment", comments: true) +# {:ok, ["echo", "hello"]} + +ShellWords.split("echo hello#world", comments: true) +# {:ok, ["echo", "hello#world"]} +``` + ### split!/2 Same as `split/2` but returns the list directly and raises @@ -112,10 +125,9 @@ or PowerShell escaping. No command execution, pipes, redirects, variable expansion (`$HOME`), command substitution (`$(...)`), globbing (`*.txt`), tilde expansion, -heredocs, or Bash-specific syntax. Comments are not parsed in v0.1.0: `#` -is an ordinary character (a `comments:` option may be added in a future version). Backslash-newline -line continuation is not supported. Behavior on invalid UTF-8 is -unspecified. +heredocs, or Bash-specific syntax. Backslash-newline line continuation is +not supported; an escaped newline is preserved literally. `split/2` +preserves invalid UTF-8 bytes instead of validating or rewriting them. ## Comparison with Python shlex and Ruby Shellwords diff --git a/ROADMAP.md b/ROADMAP.md index 5451907..04e38f8 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -3,9 +3,9 @@ Planned direction for `shell_words`. Dates are intentions, not promises; scope may shift based on feedback and real-world usage. -## v0.1.0 — Initial release (current) +## v0.1.0 — Initial release -Status: complete, pending publication to Hex. +Status: complete. - `split/2` and `split!/2` with POSIX-like word rules: whitespace separation, single quotes, double quotes with POSIX backslash semantics, @@ -17,21 +17,20 @@ Status: complete, pending publication to Hex. - Property-tested round-trip guarantee: `split(join(argv)) == {:ok, argv}` for arbitrary valid-UTF-8 argv. -## v0.2.0 — Comments and edge-case decisions +## v0.2.0 — Comments and edge-case decisions (current) +- Status: complete, pending publication to Hex. - `comments: true` option for `split/2` and `split!/2`. Semantics: an unquoted, unescaped `#` starts a comment only at the start of a word (start of input or immediately after unquoted whitespace); the comment runs to the next newline or end of input. `echo hello#world` does NOT start a comment; `echo hello #world` does. Default stays `comments: false`. -- Decide and document invalid-UTF-8 handling: explicit error vs byte - pass-through (currently unspecified). -- Decide and document backslash-newline continuation (currently a literal - newline outside quotes mid-input, `:trailing_escape` at end of input, - literal pass-through inside double quotes). -- Compatibility test suite comparing outputs against Python `shlex` and - Ruby `Shellwords`. +- Invalid UTF-8 handling documented as byte pass-through for `split/2`. +- Backslash-newline behavior documented as literal preservation, not line + continuation. +- Compatibility test suite compares supported outputs against Python `shlex` + and Ruby `Shellwords`. ## v1.0.0 — Stable API diff --git a/lib/shell_words.ex b/lib/shell_words.ex index 9d7545b..954a53b 100644 --- a/lib/shell_words.ex +++ b/lib/shell_words.ex @@ -21,7 +21,7 @@ defmodule ShellWords do This is not a shell interpreter. There is no command execution, pipe or redirect handling, variable expansion, command substitution, globbing, - tilde expansion, or comment parsing. `#` is an ordinary character. + or tilde expansion. Comment parsing is opt-in with `comments: true`. Only POSIX-like shells are targeted; there is no Windows `cmd.exe` or PowerShell escaping. @@ -44,9 +44,9 @@ defmodule ShellWords do Word separators are exactly ASCII space, tab, newline, and carriage return; Unicode whitespace is an ordinary character. Backslash-newline is - not line continuation: the newline is kept as a literal character. Error - positions are 0-based byte offsets. Behavior on invalid UTF-8 input is - unspecified and may change. + not line continuation: the newline is kept as a literal character. Invalid + UTF-8 bytes are preserved by `split/2`. Error positions are 0-based byte + offsets. """ alias ShellWords.ParseError @@ -60,8 +60,13 @@ defmodule ShellWords do Returns `{:ok, words}` or `{:error, %ShellWords.ParseError{}}`. - No options are supported yet; the `opts` argument exists to keep the arity - stable for future releases. Unknown options raise `ArgumentError`. + Supported options: + + * `:comments` — when `true`, an unquoted, unescaped `#` starts a comment + only at the start of a word. The comment runs to the next newline or + the end of input. Defaults to `false`. + + Unknown options raise `ArgumentError`. ## Examples @@ -71,13 +76,17 @@ defmodule ShellWords do iex> ShellWords.split("") {:ok, []} + iex> ShellWords.split("echo hello # comment", comments: true) + {:ok, ["echo", "hello"]} + """ @spec split(String.t(), keyword()) :: {:ok, [String.t()]} | {:error, ParseError.t()} def split(input, opts \\ []) def split(input, opts) when is_binary(input) and is_list(opts) do - validate_opts!(opts) - bare(input, 0, "", [], false) + comments? = validate_opts!(opts) + + bare(input, 0, "", [], false, comments?) end @doc """ @@ -154,9 +163,19 @@ defmodule ShellWords do defp safe_bare_word?(<>) when c in @safe_chars, do: safe_bare_word?(rest) defp safe_bare_word?(_), do: false - defp validate_opts!([]), do: :ok + defp validate_opts!(opts), do: parse_opts(opts, false) + + defp parse_opts([], comments?), do: comments? + + defp parse_opts([{:comments, comments?} | opts], _comments?) when is_boolean(comments?) do + parse_opts(opts, comments?) + end - defp validate_opts!([opt | _]) do + defp parse_opts([{:comments, value} | _opts], _comments?) do + raise ArgumentError, "expected :comments to be a boolean, got: #{inspect(value)}" + end + + defp parse_opts([opt | _opts], _comments?) do raise ArgumentError, "unknown option: #{inspect(opt)}" end @@ -170,76 +189,86 @@ defmodule ShellWords do # State: bare — outside any quotes. # word is the word built so far; started? distinguishes "no word open" # from "word open but currently empty" (needed for '' and ""). - defp bare(<<>>, _pos, word, acc, started?) do + defp bare(<<>>, _pos, word, acc, started?, _comments?) do {:ok, Enum.reverse(finish_word(word, acc, started?))} end - defp bare(<<"\\">>, pos, _word, _acc, _started?) do + defp bare(<<"\\">>, pos, _word, _acc, _started?, _comments?) do {:error, ParseError.new(:trailing_escape, pos)} end - defp bare(<<"\\", c, rest::binary>>, pos, word, acc, _started?) do - bare(rest, pos + 2, <>, acc, true) + defp bare(<<"\\", c, rest::binary>>, pos, word, acc, _started?, comments?) do + bare(rest, pos + 2, <>, acc, true, comments?) + end + + defp bare(<<"'", rest::binary>>, pos, word, acc, _started?, comments?) do + single(rest, pos + 1, pos, word, acc, comments?) end - defp bare(<<"'", rest::binary>>, pos, word, acc, _started?) do - single(rest, pos + 1, pos, word, acc) + defp bare(<<"\"", rest::binary>>, pos, word, acc, _started?, comments?) do + double(rest, pos + 1, pos, word, acc, comments?) end - defp bare(<<"\"", rest::binary>>, pos, word, acc, _started?) do - double(rest, pos + 1, pos, word, acc) + defp bare(<<"#", rest::binary>>, pos, word, acc, false, true) do + {rest, pos} = skip_comment(rest, pos + 1) + + bare(rest, pos, word, acc, false, true) end - defp bare(<>, pos, word, acc, started?) when c in @whitespace do - bare(rest, pos + 1, "", finish_word(word, acc, started?), false) + defp bare(<>, pos, word, acc, started?, comments?) when c in @whitespace do + bare(rest, pos + 1, "", finish_word(word, acc, started?), false, comments?) end - defp bare(<>, pos, word, acc, _started?) do - bare(rest, pos + 1, <>, acc, true) + defp bare(<>, pos, word, acc, _started?, comments?) do + bare(rest, pos + 1, <>, acc, true, comments?) end # State: single — inside single quotes. Everything is literal until the # closing quote; open_pos is the offset of the opening quote. - defp single(<<>>, _pos, open_pos, _word, _acc) do + defp single(<<>>, _pos, open_pos, _word, _acc, _comments?) do {:error, ParseError.new(:unterminated_single_quote, open_pos)} end - defp single(<<"'", rest::binary>>, pos, _open_pos, word, acc) do - bare(rest, pos + 1, word, acc, true) + defp single(<<"'", rest::binary>>, pos, _open_pos, word, acc, comments?) do + bare(rest, pos + 1, word, acc, true, comments?) end - defp single(<>, pos, open_pos, word, acc) do - single(rest, pos + 1, open_pos, <>, acc) + defp single(<>, pos, open_pos, word, acc, comments?) do + single(rest, pos + 1, open_pos, <>, acc, comments?) end # State: double — inside double quotes. Backslash escapes only # @dquote_escapable; any other backslash sequence passes through with the # backslash preserved. open_pos is the offset of the opening quote. - defp double(<<>>, _pos, open_pos, _word, _acc) do + defp double(<<>>, _pos, open_pos, _word, _acc, _comments?) do {:error, ParseError.new(:unterminated_double_quote, open_pos)} end - defp double(<<"\\">>, _pos, open_pos, _word, _acc) do + defp double(<<"\\">>, _pos, open_pos, _word, _acc, _comments?) do {:error, ParseError.new(:unterminated_double_quote, open_pos)} end - defp double(<<"\\", c, rest::binary>>, pos, open_pos, word, acc) + defp double(<<"\\", c, rest::binary>>, pos, open_pos, word, acc, comments?) when c in @dquote_escapable do - double(rest, pos + 2, open_pos, <>, acc) + double(rest, pos + 2, open_pos, <>, acc, comments?) end - defp double(<<"\\", c, rest::binary>>, pos, open_pos, word, acc) do - double(rest, pos + 2, open_pos, <>, acc) + defp double(<<"\\", c, rest::binary>>, pos, open_pos, word, acc, comments?) do + double(rest, pos + 2, open_pos, <>, acc, comments?) end - defp double(<<"\"", rest::binary>>, pos, _open_pos, word, acc) do - bare(rest, pos + 1, word, acc, true) + defp double(<<"\"", rest::binary>>, pos, _open_pos, word, acc, comments?) do + bare(rest, pos + 1, word, acc, true, comments?) end - defp double(<>, pos, open_pos, word, acc) do - double(rest, pos + 1, open_pos, <>, acc) + defp double(<>, pos, open_pos, word, acc, comments?) do + double(rest, pos + 1, open_pos, <>, acc, comments?) end + defp skip_comment(<<>>, pos), do: {<<>>, pos} + defp skip_comment(<<"\n", rest::binary>>, pos), do: {rest, pos + 1} + defp skip_comment(<<_c, rest::binary>>, pos), do: skip_comment(rest, pos + 1) + defp finish_word(_word, acc, false), do: acc defp finish_word(word, acc, true), do: [word | acc] end diff --git a/mix.exs b/mix.exs index 7f6faed..c173e83 100644 --- a/mix.exs +++ b/mix.exs @@ -1,7 +1,7 @@ defmodule ShellWords.MixProject do use Mix.Project - @version "0.1.0" + @version "0.2.0" @source_url "https://github.com/ivan-podgurskiy/shell_words" def project do diff --git a/test/shell_words_compatibility_test.exs b/test/shell_words_compatibility_test.exs new file mode 100644 index 0000000..e9b22da --- /dev/null +++ b/test/shell_words_compatibility_test.exs @@ -0,0 +1,92 @@ +defmodule ShellWordsCompatibilityTest do + use ExUnit.Case, async: true + + @split_cases [ + {"simple words", "echo hello", ["echo", "hello"]}, + {"quoted whitespace", ~S(echo "hello world"), ["echo", "hello world"]}, + {"single quotes", ~S(echo 'hello world'), ["echo", "hello world"]}, + {"escaped whitespace", ~S(echo hello\ world), ["echo", "hello world"]}, + {"adjacent segments", ~S(echo foo"bar"'baz'), ["echo", "foobarbaz"]}, + {"escaped hash", ~S(echo \#), ["echo", "#"]} + ] + + @comment_cases [ + {"whole-line comment", "# comment\nnext word", ["next", "word"]}, + {"word-start comment", "echo hello # comment", ["echo", "hello"]} + ] + + describe "Python shlex compatibility" do + @tag :external_tool + test "split/2 matches shlex.split for supported non-comment cases" do + require_tool!("python3") + + for {_name, input, expected} <- @split_cases do + assert_python_split(input, false, expected) + assert ShellWords.split(input) == {:ok, expected} + end + end + + @tag :external_tool + test "comments: true matches shlex.split comments behavior" do + require_tool!("python3") + + for {_name, input, expected} <- @comment_cases do + assert_python_split(input, true, expected) + assert ShellWords.split(input, comments: true) == {:ok, expected} + end + end + end + + describe "Ruby Shellwords compatibility" do + @tag :external_tool + test "split/2 matches Shellwords.split for supported non-comment cases" do + require_tool!("ruby") + + for {_name, input, expected} <- @split_cases do + assert_ruby_split(input, expected) + assert ShellWords.split(input) == {:ok, expected} + end + end + end + + defp require_tool!(name) do + if System.find_executable(name) == nil do + flunk("#{name} is required for this compatibility test") + end + end + + defp assert_python_split(input, comments?, expected) do + python_comments? = if comments?, do: "True", else: "False" + + script = """ + import shlex + import sys + + actual = shlex.split(sys.argv[1], comments=#{python_comments?}) + expected = sys.argv[2:] + + if actual != expected: + print(f"expected {expected!r}, got {actual!r}", file=sys.stderr) + sys.exit(1) + """ + + assert {"", 0} = + System.cmd("python3", ["-c", script, input | expected], stderr_to_stdout: true) + end + + defp assert_ruby_split(input, expected) do + script = """ + require 'shellwords' + + actual = Shellwords.split(ARGV.fetch(0)) + expected = ARGV.drop(1) + + unless actual == expected + warn "expected \#{expected.inspect}, got \#{actual.inspect}" + exit 1 + end + """ + + assert {"", 0} = System.cmd("ruby", ["-e", script, input | expected], stderr_to_stdout: true) + end +end diff --git a/test/shell_words_test.exs b/test/shell_words_test.exs index 58f6f62..59e3830 100644 --- a/test/shell_words_test.exs +++ b/test/shell_words_test.exs @@ -99,20 +99,54 @@ defmodule ShellWordsTest do assert ShellWords.split("caf\u00E9 \u00FCber") == {:ok, ["caf\u00E9", "\u00FCber"]} end - test "# is an ordinary character" do + test "# is an ordinary character by default" do assert ShellWords.split("echo hello # comment") == {:ok, ["echo", "hello", "#", "comment"]} end + test "comments: true starts comments only at the start of a word" do + assert ShellWords.split("echo hello # comment", comments: true) == + {:ok, ["echo", "hello"]} + + assert ShellWords.split("echo hello#world", comments: true) == + {:ok, ["echo", "hello#world"]} + + assert ShellWords.split("# comment\nnext word", comments: true) == + {:ok, ["next", "word"]} + + assert ShellWords.split("echo\t# comment\nnext", comments: true) == + {:ok, ["echo", "next"]} + end + + test "quoted and escaped # characters do not start comments" do + assert ShellWords.split(~S(echo "# not a comment"), comments: true) == + {:ok, ["echo", "# not a comment"]} + + assert ShellWords.split(~S(echo '# not a comment'), comments: true) == + {:ok, ["echo", "# not a comment"]} + + assert ShellWords.split(~S(echo \#), comments: true) == + {:ok, ["echo", "#"]} + end + + test "comments: false preserves the v0.1.0 behavior explicitly" do + assert ShellWords.split("echo hello # comment", comments: false) == + {:ok, ["echo", "hello", "#", "comment"]} + end + test "unknown options raise ArgumentError" do assert_raise ArgumentError, fn -> - ShellWords.split("echo", comments: true) + ShellWords.split("echo", unknown: true) end end test "empty options list is accepted" do assert ShellWords.split("echo", []) == {:ok, ["echo"]} end + + test "invalid UTF-8 bytes are preserved by split/2" do + assert ShellWords.split(<<255, ?a, ?\s, ?b>>) == {:ok, [<<255, ?a>>, "b"]} + end end describe "split/2 — single quotes" do @@ -244,7 +278,7 @@ defmodule ShellWordsTest do test "propagates option validation" do assert_raise ArgumentError, fn -> - ShellWords.split!("echo", comments: true) + ShellWords.split!("echo", unknown: true) end end end