Shell Compatibility Reports from Oils

[andychu]

Thank you for fixing issue #538 !

I created a compatibility report, with several shells including Brush:

• https://pages.oils.pub/spec-compat/2025-06-15/renamed-tmp/spec/compat/TOP.html

Summary:

• According to our assertions, brush is passing 1528 tests. It's the most bash-compatible shell, other than OSH!
• The columns are sortable
  • https://pages.oils.pub/spec-compat/2025-06-15/renamed-tmp/spec/compat/DELTA-bash.html#t:DELTA-bash=6a
  • So you can see there that the biggest differences of brush vs. bash are alias, nameref, trap, errexit, brace-expansion, etc.
• See all the notes and caveats on the page

Let me know what you think!

I'd like to get some feedback, and then I'll probably reach out to the authors of other active shells (e.g. I just found out that ksh93 is still being developed!)

---

For some context, the Oils project is swinging back to OSH, from YSH, probably for the next year or so. We're making a push for compatibility with distro builds, e.g. Alpine Linux and Nix ...

As mentioned, I think Brush's goals are somewhat complementary to that, i.e. if you are more interested in the interactive shell than scripting. I think it would be nice to reduce duplication somehow, although certainly it's not straightforward to do that!

[reubeno]

This is great, @andychu! Thanks for putting together the report. I'll need some time to review it in greater detail 🙂. Having this sort of objective assessment of bash compatibility is awesome.

And I do generally agree with your sentiments on finding the right sort of complement. I'm also not sure where exactly that is right now but I'm eager to continue the conversation.

[andychu]

OK great, yes we should think of some ways to collaborate ... It's been hard to find contributors, and I probably made the mistake of trying to do too much myself at the beginning.

NLnet really helped with that though -- we've had grants since 2022, and will probably have another one in the very near future. NLnet is interested in a better foundation for Nix (which uses a TON of bash -- more bash features than other distros!)

---

I suppose the tests are supposed to give a feeling for "How much work on bash compatibility is left?"

And I was surprised that Brush passed so many tests! When I first looked, I saw 34K lines of code, and <600 commits, so I didn't think it would pass that many tests. And it passes more than ksh and mksh, which are very old and mature codebases, so that is impressive!

I poked around manually and it works well (although I have the classic low contrast issue on my Debian machine)

---

On the other hand, my experience in this area is that "once you do the first 90%, then you have to do the other 90%" :-) I'll be interested in what you think after looking at some of the failures.

I actually did a ton of work on the interactive shell all the way back in 2019, and made OSH run git-completion.bash, and other parts of bash-completion

But that was a bit painful, so I haven't pushed as hard on OSH in awhile. I worked on YSH and the translation to C++, which was also hard ("three languages takes longer than one" :-) )

There were plenty of OSH bug reports from users to keep busy. i.e. they isolated test cases for us, so I haven't done "app-driven" work as much

Some contributors are now doing app-driven work, and that has been really helpful. I watched this video on the Ladybird browser and realized that this is a fundamental difference:

https://youtu.be/9YM7pDMLvr4?t=1767

i.e. some people like doing app-driven improvement, and some people like doing spec-driven improvement (e.g. making all those spec tests pass)

Also another thing from that video I liked is the focus on tools to help contributors. Basically I think we have to be a bit "meta", and make it clear:

1. how much compatibility work there is left
2. give fast feedback on whether a given change helps. Right now it's hard to make the connection between a commit and being able to run a new shell program

---

I still think the liboils idea (win32 and Rust) is good, but it is probably secondary to both compatibility work, and tools to help contributors make things more compatible.

I can imagine allocating 10% of a grant to it though

---

But I also think there are some advantages to the way Oils works now, aside from liboils:

1. The edit-run cycle

For Python, it's ~50ms. For brush I did an incremental debug build in about 4 seconds? That is fast enough for most projects, but personally I get value out of the sub-second response, for fixing all these compatibility bugs.

2. We have precise location info and precise error messages - this took awhile to work out, but is settled now.

3. We seem to have less binary code than brush and sush (at 2.5 MB, will be < 2MB soon), and all our code is self-contained in the repo. It's a tradeoff because we reuse less, but we get to control it.

I had a discussion with a nushell contributor about the tradeoffs of reuse ... e.g. reusing Rust's portability to Windows can cause some issues for shells

---

After reviewing the test results, I'd be curious if you have any interest in dipping your toes into OSH development ... I think a lot of stuff on https://brush.sh/ is possible, or should be possible, within OSH. So maybe OSH is something you can personally use (?)

I always find it more motivating to work on things I'm going to use. That is one reason the interactive shell is less fancy -- because I like a minimalist UI. (I'd like it to be closer to zsh than bash though, and not as fancy as fish)

There is also more low-level compiler- and runtime work to do, if that is more interesting to you

---

I'd also be interested in a report on OSH compatibility from your perspective. I think our tests are pretty good, but other test suites may improve our picture of what's left! OSH is definitely "converging", and I want to use this next NLnet grant to be more precise about that.

These spec tests and the Alpine Linux app tests should give everyone a good picture. We're also going to write about Oils more -- here's a recent post from a contributor Melvin, who is awesome - https://melvinw.com/posts/2025-06-11-oils-for-unix-overview.html

I have this #bug-post-mortem idea too, since we've been fixing a lot of hairy bugs. This is probably the most elaborate example of that - https://lobste.rs/s/hru0ib/how_lose_control_your_shell (related to a bug I hit running the Starship prompt, which I don't use !)

[reubeno]

I suppose the tests are supposed to give a feeling for "How much work on bash compatibility is left?"

Indeed. I'd love to find a way for our CI processes to run the oils compat tests. Do you have a recommended way to do that? We already have some precedent with running the bash-completion test suite and some other external tests as part of outer-loop testing during PR checks.

give fast feedback on whether a given change helps. Right now it's hard to make the connection between a commit and being able to run a new shell program

That's a great insight! I definitely find it motivating to see a new popular tool / script / completion start working once a compatibility issue is resolved :) I've started labeling bugs with some indication of what popular tool may get (partially) unblocked by it, but I haven't been very systematic about it.

We seem to have less binary code than brush and sush (at 2.5 MB, will be < 2MB soon), and all our code is self-contained in the repo. It's a tradeoff because we reuse less, but we get to control it.

I do wish the binary size of brush was smaller. I did some initial investigations with cargo bloat and similar tooling, and there's some low-hanging fruit we could get by tweaking compiler flags, using some unstable rustc options to remove features from the std library, etc. As you hint, there's also a fair bit of the code size that's coming from reused generic crates (e.g., fancy-regex, clap, etc.). We also made an intentional trade-off to keep the implementation of brush simpler through crate reuse, so a fair bit of this is intentional.

I always find it more motivating to work on things I'm going to use.

Ditto! brush is my daily driver as a shell 😄

I have this #bug-post-mortem idea too, since we've been fixing a lot of hairy bugs.

I'd certainly be interested to read a "bug-of-the-month" (or release) post, particularly if it shines light on an otherwise under-illuminated part of the shell world.

[andychu]

Yes I think we should turn the spec tests into something more cross-project.... I actually had that discussion back in 2019 with the author of Smoosh (https://github.com/mgree/smoosh) , but at that time it wasn't worth the effort (i.e. OSH was immature)

I looked, and there is probably a way to do it with Github Actions, although I'm trying to minimize dependencies on clouds and proprietary APIs. Our CI runs on multiple clouds now

The CI set up is basically the same as the contributor setup, except it's baked into a Docker image. You compile a bunch of shells from scratch - https://github.com/oils-for-unix/oils/wiki/Contributing

Right now the expanded test/spec-compat report is just running on my machine, but there should be a container!

---

A primary goal of Oils was always to produce better POSIX spec that is updated with things that many shells agree on

• bash and ksh agree on many things that are not in POSIX -
• busybox ash has been adopting bash features
• now I noticed that brush, sush, and OSH also agree on many of those things !!

So really I think the goal should be agreement on semantics, so that users have a stable languages to write! Not just "whatever bash does"

e.g. I find it hard to believe that the difference between echo ${a[@]:0} and echo ${a[@]:0:} was an intentional design decision in bash:

https://www.oilshell.org/blog/2024/11/release-0.23.0.html#more-osh-compatibility-slicing

---

I also tried to revive bin/osh_parse.py a bit, which is the parser as a library ... If that works on Windows, then it can give an idea of how it will work in Rust. There are some tools like resholve built on Oils in Python - https://github.com/abathur/resholve - but using Rust makes sense too

Historically there have been many Rust users interested in Oils, e.g. the Ion shell was influenced by Oils way back in 2017

---

I definitely see that if you got this far, it is probably not going to be fun to change directions ... If we have the common spec tests, we can assess how much work there is, and how much is duplicated.

That takes awhile -- I actually went back and marked some tests failing for OSH, since there were assertions from 2016-2018 that were a bit biased

So yeah I think it will be good to iron out all these differences ... I will also contact the authors of sush and toysh

(some of our tests are from toybox, and they had some very good corner cases!)

---

A funny thing is that I started in 2016 contributing to toybox, because I want to be more collaborative

But memory safety was a big reason I stopped contributing to that project!

e.g. http://lists.landley.net/pipermail/toybox-landley.net/2016-March/016106.html

http://lists.landley.net/pipermail/toybox-landley.net/2016-March/016179.html

Also, I really think we have to move past bash!

[reubeno]

So really I think the goal should be agreement on semantics, so that users have a stable languages to write! Not just "whatever bash does"

I strongly agree!

Yes I think we should turn the spec tests into something more cross-project...

To me, this goes hand-in-hand with the semantics agreement. The core POSIX shell language definition has a spec, and it comes with (licensed) conformance tests, independent of implementation.

If it's not already supported, I'd love to see (and, if applicable, contribute to) a way for the tests you've built up over time to be self-contained, runnable against an arbitrary input executable, and produce a machine-parseable summary of the test results (e.g., JSON or similar?). Is that doable today? That would be sufficient, say, for the brush project to wire up an additional CI job that runs the tests with the new build of brush, summarize the results, diff against known project-specific test baselines (to ensure we're not regressing, but not penalize for tests we've never seen pass), etc. It's also something I'd run as part of local testing when making changes. We've done this in the past with bash-completion's tests via pytest.

(The multi-cross-shell comparison, CI integration / GitHub actions, etc. is definitely great too, but in my mind that can all be at a higher layer or run more centrally.)

I definitely see that if you got this far, it is probably not going to be fun to change directions ... If we have the common spec tests, we can assess how much work there is, and how much is duplicated.

I appreciate your acknowledgment of the human side of things 😄. I think our projects have overlapping but not-identical goals. (I still absolutely owe this project a full write-up of what my aspirations have been / are / may be!) Starting with shared compatibility tests sounds like a great step.

A funny thing is that I started in 2016 contributing to toybox, because I want to be more collaborative
But memory safety was a big reason I stopped contributing to that project!

We're definitely on the same page regarding memory safety!

simply add brush assertions. e.g. some tests might fail for an innocuous reason like status=1 versus status=2

I'm assuming you've seen brush's compatibility tests? Early on, we made the decision not to encode actual expectations in the tests and rely 100% on using bash as a runtime oracle. (Technically speaking, our tests could use a different shell as an oracle but in practice we go with bash.) With that in mind, even status=1 vs. status=2 is something we'd consider an undesired behavioral difference.

Channeling your comments above, though, we recognize that this ^^^ means we've got a moving target as bash behavior changes across versions. And I'd certainly agree that the "oracle" isn't always consistent and has many behaviors that weren't necessarily intentional. (The phrase "bug for bug compatibility" comes to mind.)

(Aside: we've experimented with using fuzz testing to identify "incompatible" behaviors between brush and bash. It was actually quite revealing for arithmetic evaluation. We had the fuzzer framework generate arithmetic expressions in AST-ish form, serialized them to strings, ran them through both brush and bash and reported a fault when the observable behavior differs. It caught some interesting inconsistencies on overflow/underflow. We haven't gone back to this approach recently but it's mainly because we hadn't yet set up a good sandboxed testing environment in which we'd feel comfortable allowing a fuzz-tester harness execute truly arbitrary shell scripts.)

[andychu]

The spec tests are basically in that shape -- the framework is in Python and is invoked with shell, and you can run it with arbitrary shells

Then it outputs TSV files, and the TSV files are turned into HTML

https://github.com/oils-for-unix/oils/blob/master/test/spec-compat.sh

If you go to a results page like this: https://pages.oils.pub/spec-compat/2025-06-19/renamed-tmp/spec/compat/alias.html

You can derive the .result.tsv URL https://pages.oils.pub/spec-compat/2025-06-19/renamed-tmp/spec/compat/alias.result.tsv

I will try to make that a bit more obvious

---

It's generally easy to run the spec tests locally -- contributors do it -- and it's documented here - https://github.com/oils-for-unix/oils/wiki/Contributing

Although the particular test/spec-compat.sh "driver" is something I just hacked up quickly, based on another driver. It could be parameterized a bit more

---

I do think it's a good idea to share more of the spec tests, so there isn't duplicate effort

And I think users should have a well-defined language to target. I was thinking we should name it "Bashix" or something :-) Based on the observation that there are MANY things that shells agree on, that are not in POSIX

I noticed that when working on Oils, and now I see it very clearly in this report!

We can start with the Oils spec tests, and perhaps add other suites. Oils also has "oracle" tests which we call test/gold -- in that case bash is the oracle

I'll contact the other shell authors and see what they think. AFAIK there are only 4 shells that aim at bash compatibility: OSH, brush, sush, and toysh!

[andychu]

Another way to see the TSV files is to clone the pages repo - https://github.com/oils-for-unix/oils-for-unix.github.io

And then look in spec-compat

---

And yes I'd be interested in reading about the goals of Brush!

I always intended OSH as an "executable spec" -- a language definition

And I've been encouraging people to build different interactive experiences on top of OSH !

I may have mentioned the "headless mode"

https://www.oilshell.org/blog/2023/12/screencasts.html#headless-protocol-oils-web_shell

https://github.com/oils-for-unix/oils/wiki/Headless-Shell-Protocol

[andychu]

Oh also as mentioned, I can think of 2 ways to make the reports more neutral

1. simply add brush assertions. e.g. some tests might fail for an innocuous reason like status=1 versus status=2
2. "majority agreement" - have sh_spec.py export the status and stdout, and then see which shells agree with the majority

There are some possible problems with #2, but I think it could be better

But I am not sure how much bias there is ... as mentioned, I removed one file, and then I also just marked a few more OSH as failing (but I haven't re-published the report)

[andychu]

The author of toysh was also interested in running Oils spec tests

So I went ahead and created the "Bashix" repo with discussions and a wiki:

https://github.com/bashix-spec/bashix?tab=readme-ov-file

https://github.com/bashix-spec/bashix/discussions

I look forward to hearing your thoughts!

I contacted the author of sush too, but haven't heard back yet