Phase 3 Plan E: FS write path + console fd + shell + utilities

README.md

@@ -70,12 +70,21 @@ and `build.zig.zon` pins the minimum Zig version (0.16.0).
 | `zig build kernel-fork` | Build the Phase 3.C `kernel-fork.elf` (same kernel objects + embedded `init.elf` + `hello.elf`) |
 | `zig build kernel-fs` | Build the Phase 3.D `kernel-fs.elf` (FS-mode kernel; loads `/bin/init` from disk) |
 | `zig build kernel-fs-init` | Build `fs_init.elf` (the on-disk `/bin/init` payload baked into `fs.img`) |
+| `zig build kernel-init-shell` | Build the Phase 3.E `init_shell.elf` (on-disk `/bin/init` for `shell-fs.img`; loops fork-exec-sh-wait) |
+| `zig build kernel-sh` | Build the Phase 3.E `sh.elf` (line/token/redirect/builtins/fork+exec) |
+| `zig build kernel-ls` | Build the Phase 3.E `ls.elf` |
+| `zig build kernel-cat` | Build the Phase 3.E `cat.elf` |
+| `zig build kernel-echo` | Build the Phase 3.E `echo.elf` |
+| `zig build kernel-mkdir` | Build the Phase 3.E `mkdir.elf` |
+| `zig build kernel-rm` | Build the Phase 3.E `rm.elf` |
 | `zig build mkfs` | Build the host-side `mkfs` tool (lays out a 4 MB image: superblock + bitmap + inode table + data blocks) |
 | `zig build fs-img` | Stage `userland/fs/` + `fs_init.elf` and run `mkfs` to produce `zig-out/fs.img` |
+| `zig build shell-fs-img` | Stage `userland/shell-fs/` + every Phase 3.E userland binary and run `mkfs` to produce `zig-out/shell-fs.img` (init_shell at `/bin/init`) |
 | `zig build e2e-kernel` | Run `ccc kernel.elf` and assert stdout matches `hello from u-mode\nticks observed: N\n` with N > 0 (Phase 2 §Definition of done) |
 | `zig build e2e-multiproc-stub` | Run `ccc kernel-multi.elf` and assert stdout contains both `hello from u-mode\n` and `[2] hello from u-mode\n`, plus a `ticks observed: N\n` trailer (Plan 3.B milestone) |
 | `zig build e2e-fork` | Boot `kernel-fork.elf`; `init` forks `/bin/hello`; parent reaps; emulator returns 0 (Plan 3.C milestone) |
 | `zig build e2e-fs` | Boot `kernel-fs.elf` against `fs.img`; on-disk `/bin/init` opens `/etc/motd`, reads it, writes to fd 1, exits 0 (Plan 3.D milestone) |
+| `zig build e2e-shell` | Boot `kernel-fs.elf` against `shell-fs.img` with `--input tests/e2e/shell_input.txt`; assert prompt+command echo for the canonical `ls /bin / echo / cat / rm / exit` session and a clean halt (Plan 3.E milestone) |
 | `zig build qemu-diff-kernel` | Diff the kernel.elf trace against `qemu-system-riscv32` (debug aid; needs QEMU installed) |
 | `zig build plic-block-test` | Build the Phase 3.A integration test ELF (asm-only S-mode program) |
 | `zig build e2e-plic-block` | Build a 4 MB test image, run `ccc --disk … plic_block_test.elf`, assert exit 0 (Plan 3.A milestone: full CMD → IRQ → trap → claim path) |
@@ -128,11 +137,11 @@ to "GitHub Actions" in repo settings (one-time manual step).
 
 ## Status
 
-**Phase 3 Plan D done — bufcache + block driver + FS read path.** Plan 3.A
-merged: PLIC, simple block device, UART RX, `--disk` and `--input` flags,
-real `wfi` idle. Plan 3.B merged: free-list page allocator, `ptable[NPROC=16]`,
-round-robin scheduler with `swtch`, kernel-side ELF32 loader,
-`getpid`/`sbrk`/`yield` syscalls, second embedded user ELF,
+**Phase 3 Plan E done — FS write path + console fd + shell + utilities.**
+Plan 3.A merged: PLIC, simple block device, UART RX, `--disk` and `--input`
+flags, real `wfi` idle. Plan 3.B merged: free-list page allocator,
+`ptable[NPROC=16]`, round-robin scheduler with `swtch`, kernel-side ELF32
+loader, `getpid`/`sbrk`/`yield` syscalls, second embedded user ELF,
 `e2e-multiproc-stub` running PID 1 + PID 2. Plan 3.C merged: `fork` (full
 address-space copy), `execve` (in-place AS rebuild + System-V argv tail),
 `wait4` (sleep on self until zombie child), `exit` (reparent + zombie + wake
@@ -148,7 +157,21 @@ directory tree. `proc.exec` now resolves the path via `namei` + `readi` into
 a kernel scratch buffer (FS-mode), or via the embedded-blob lookup (single
 / multi / fork modes) — selected at compile time per kernel variant.
 `e2e-fs` runs `kernel-fs.elf` against `fs.img`: the on-disk `/bin/init` opens
-`/etc/motd`, reads it, writes the contents to fd 1, exits 0.
+`/etc/motd`, reads it, writes the contents to fd 1, exits 0. Plan 3.E
+merged: FS write path (`writei` with `bmap` lazy alloc, `iupdate`, `ialloc`,
+`itrunc`, `iput`-on-zero truncate, real `dirlink` + `dirunlink`,
+`fs/fsops.zig` glue), console as fd 0/1/2 with cooked-mode line discipline
+(echo, backspace, `^U`, `^C`, `^D`, line completion), UART RX delivered
+through PLIC IRQ #10 → `uart.isr` → `console.feedByte`, scheduler now
+`wfi`s in its idle window so `cpu.idleSpin` paces `--input` byte delivery
+to match interactive cadence. New syscalls: `mkdirat` (#34), `unlinkat`
+(#35); `openat` extended with `O_CREAT`/`O_TRUNC`/`O_APPEND`; `write` now
+routes any fd through `file.write`. User stdlib lands at
+`src/kernel/user/lib/` (`start.S`, `usys.S`, `ulib.zig`, `uprintf.zig`),
+fed by an `addUserBinary` build helper. Userland: `init` (init_shell:
+fork-exec-sh-wait), `sh` (line/token/redirect/builtins/fork+exec), `ls`,
+`cat`, `echo`, `mkdir`, `rm`. `e2e-shell` runs the canonical scripted
+session against `kernel-fs.elf` + `shell-fs.img`.
 
 **Phase 1 — RISC-V CPU emulator — complete.**
 
@@ -257,7 +280,53 @@ space. The on-disk `init` reads `/etc/motd` and writes it to UART:
     hello from phase 3
     ticks observed: 4
 
-Next: Plan 3.E — file write path + console line discipline + shell.
+Plan 3.E (FS write path + console fd + shell + utilities) is merged. The
+filesystem grew a write path: `inode.writei` with `bmap`'s lazy allocation
+(`for_write` flag), `iupdate` (in-memory → on-disk inode flush), `ialloc`
+(scan inode table + initial `iupdate`), `itrunc` (free direct + indirect
+blocks; called from `iput` when ref+nlink hit zero), `dirlink` (real impl
+with empty-slot scan), `dirunlink`, plus an `fs/fsops.zig` create/unlink
+glue. `openat` gained `O_CREAT` / `O_TRUNC` / `O_APPEND`; new syscalls
+`mkdirat` (#34) + `unlinkat` (#35). `console.zig` lands as the fd 0/1/2
+backing — cooked-mode line discipline (per-byte echo, backspace, `^U`
+line-kill, `^C` foreground-proc kill via `proc.kill`, `^D` EOF, `\n` line
+commit + sleeper wakeup), Raw mode arm wired but only exercised by 3.F's
+editor. UART RX is now alive: PLIC source 10 → `uart.isr` reads RBR until
+empty, feeding each byte to `console.feedByte`. The scheduler's idle path
+now executes `wfi` (so `cpu.idleSpin` runs and the emulator's `rx_pump`
+paces `--input` bytes one-per-iteration to interleave with cooked-mode
+echo); `s_kernel_trap_dispatch` advances `sepc` past the `wfi` so the
+SIE window can actually close. A small user stdlib lands at
+`src/kernel/user/lib/` (RV32 `_start` parsing argc/argv, 19 `ecall` stubs,
+`mem*`/`str*` + `O_*` constants, a minimal `printf`). The
+`addUserBinary` build helper packs 7 new userland binaries — `init_shell`
+(loops fork-exec-sh-wait, exits cleanly when sh exits 0), `sh`
+(line/token/redirect/builtins/fork+exec), `ls`, `cat`, `echo`, `mkdir`,
+`rm`. `mkfs.zig` learned `--init` (override `/bin/init`) and walks every
+top-level subdir of `--root` (so `/tmp/` empty-dir staging carries
+through). `shell-fs.img` is the parallel image that bakes init_shell as
+`/bin/init` and ships every utility under `/bin/`. The Phase 3.E milestone
+runs the scripted session through `--input`:
+
+    $ zig build kernel-fs shell-fs-img && zig build run -- --input tests/e2e/shell_input.txt --disk zig-out/shell-fs.img zig-out/bin/kernel-fs.elf
+    $ ls /bin
+    .
+    ..
+    cat
+    init
+    echo
+    sh
+    mkdir
+    ls
+    rm
+    $ echo hi > /tmp/x
+    $ cat /tmp/x
+    hi
+    $ rm /tmp/x
+    $ exit
+    ticks observed: 6
+
+Next: Plan 3.F — `edit` userland + raw-mode editor + `e2e-persist`.
 
 ## Layout
 
@@ -297,27 +366,46 @@ src/
     uart.zig          # kernel-side UART driver
     plic.zig          # kernel-side PLIC driver (setPriority/enable/setThreshold/claim/complete)
     block.zig         # kernel-side block driver (single-outstanding submit + sleep on req; isr wakes)
-    file.zig          # NFILE=64 file table + read/lseek/fstat (single Inode type in 3.D; Console in 3.E)
+    file.zig          # NFILE=64 file table + read/write/lseek/fstat — 3.E adds Console-typed entries for fd 0/1/2
+    console.zig       # 3.E: cooked-mode line discipline (echo + backspace + ^C/^U/^D + \n commit) + Raw arm; backs fd 0/1/2
     fs/
       layout.zig      # on-disk constants (BLOCK_SIZE, NBLOCKS, NINODES, SuperBlock, DiskInode, DirEntry)
       bufcache.zig    # NBUF=16 LRU buffer cache with sleep-on-busy + bget/brelse/bread/bwrite
-      balloc.zig      # block bitmap (alloc/free; write-side reserved for 3.E)
-      inode.zig       # NINODE=32 in-memory inode cache + iget/iput/ilock/iunlock + bmap + readi
-      dir.zig         # DirEntry record + dirlookup + dirlink stub (3.E)
+      balloc.zig      # block bitmap (alloc/free; write-side wired in 3.E)
+      inode.zig       # NINODE=32 in-memory inode cache + iget/iput/ilock/iunlock + bmap (lazy alloc on for_write) + readi/writei + iupdate + ialloc + itrunc
+      dir.zig         # DirEntry record + dirlookup + dirlink + dirunlink (3.E)
       path.zig        # namei + nameiparent (root for absolute, cur.cwd for relative)
-    mkfs.zig          # host-side tool: walks --root + --bin into a 4 MB image (super + bitmap + inodes + data)
+      fsops.zig       # 3.E: create + unlink glue used by sysOpenat (O_CREAT) / sysMkdirat / sysUnlinkat
+    mkfs.zig          # host-side tool: walks --root subdirs + --bin into a 4 MB image; --init overrides /bin/init
     linker.ld         # kernel.elf load layout
     user/
       userprog.zig    # PID 1 user payload (embedded into kernel.elf)
       userprog2.zig   # PID 2 user payload (embedded into kernel-multi.elf)
       init.zig        # init userland for kernel-fork.elf (fork+exec+wait)
       hello.zig       # hello userland for kernel-fork.elf (write+exit)
       fs_init.zig     # on-disk /bin/init for kernel-fs.elf (open /etc/motd, read, write fd 1, exit)
+      init_shell.zig  # 3.E: on-disk /bin/init for shell-fs.img (loops fork-exec-sh-wait; exits cleanly on sh status 0)
+      sh.zig          # 3.E: shell — line read, token split, redirect (< > >>), builtins (cd / pwd / exit), fork+exec
+      ls.zig          # 3.E: directory listing + Stat dispatch
+      cat.zig         # 3.E: read fd or args, write fd 1
+      echo.zig        # 3.E: print joined args + \n
+      mkdir.zig       # 3.E: mkdirat for each arg
+      rm.zig          # 3.E: unlinkat for each arg
+      lib/
+        start.S       # 3.E: RV32 _start — parses argc/argv from sp tail, calls main, ecall exit
+        usys.S        # 3.E: 19 syscall stubs (li a7; ecall; ret)
+        ulib.zig      # 3.E: mem*/str* + syscall externs + Stat / O_* constants
+        uprintf.zig   # 3.E: minimal printf(fd, fmt, args)
       user_linker.ld  # user-side linker script
     userland/
       fs/
         etc/
           motd        # staged content for fs.img: "hello from phase 3\n"
+      shell-fs/        # 3.E: staging tree for shell-fs.img (init_shell + utilities go to /bin via mkfs)
+        etc/
+          motd        # same 19-byte content as userland/fs/etc/motd
+        tmp/
+          .gitkeep    # carrier for empty /tmp/ in git; mkfs skips dot-files
 demo/
   web_main.zig        # freestanding wasm entry — runStart/runStep/setMtimeNs/pushInput/consumeOutput, fixed 2 MB ELF buffer (programs fetched at runtime, not embedded)
 programs/
@@ -332,7 +420,10 @@ tests/
     multiproc.zig     # Plan 3.B verifier (PID 1 + PID 2 interleaving)
     fork.zig          # Plan 3.C verifier (fork/exec/wait/exit)
     fs.zig            # Plan 3.D verifier (init opens /etc/motd, writes contents to fd 1)
+    shell.zig         # Plan 3.E verifier (scripted ls/echo/cat/rm/exit session)
+    shell_input.txt   # 51-byte canonical session piped via --input
     snake.zig         # snake e2e verifier (deterministic input → GAME OVER)
+    snake_input.txt   # snake e2e input fixture
   fixtures/           # tiny hand-crafted ELF used only by elf.zig tests
   riscv-tests/        # upstream submodule: riscv-software-src/riscv-tests
   riscv-tests-shim/   # weak handlers + riscv_test.h overrides for the shared test env

build.zig

@@ -445,6 +445,47 @@ pub fn build(b: *std.Build) void {
     const kernel_fs_init_step = b.step("kernel-fs-init", "Build the Phase 3.D fs_init.elf");
     kernel_fs_init_step.dependOn(&install_kernel_fs_init_elf.step);
 
+    const init_shell_exe = addUserBinary(
+        b,
+        "init_shell",
+        "src/kernel/user/init_shell.zig",
+        rv_target,
+        .ReleaseSmall,
+    );
+    const install_init_shell = b.addInstallFile(init_shell_exe.getEmittedBin(), "init_shell.elf");
+    const kernel_init_shell_step = b.step("kernel-init-shell", "Build init_shell.elf (Phase 3.E /bin/init)");
+    kernel_init_shell_step.dependOn(&install_init_shell.step);
+
+    const echo_exe = addUserBinary(b, "echo", "src/kernel/user/echo.zig", rv_target, .ReleaseSmall);
+    const install_echo = b.addInstallFile(echo_exe.getEmittedBin(), "echo.elf");
+    const kernel_echo_step = b.step("kernel-echo", "Build echo.elf (Phase 3.E)");
+    kernel_echo_step.dependOn(&install_echo.step);
+
+    const cat_exe = addUserBinary(b, "cat", "src/kernel/user/cat.zig", rv_target, .ReleaseSmall);
+    const install_cat = b.addInstallFile(cat_exe.getEmittedBin(), "cat.elf");
+    const kernel_cat_step = b.step("kernel-cat", "Build cat.elf (Phase 3.E)");
+    kernel_cat_step.dependOn(&install_cat.step);
+
+    const ls_exe = addUserBinary(b, "ls", "src/kernel/user/ls.zig", rv_target, .ReleaseSmall);
+    const install_ls = b.addInstallFile(ls_exe.getEmittedBin(), "ls.elf");
+    const kernel_ls_step = b.step("kernel-ls", "Build ls.elf (Phase 3.E)");
+    kernel_ls_step.dependOn(&install_ls.step);
+
+    const mkdir_exe = addUserBinary(b, "mkdir", "src/kernel/user/mkdir.zig", rv_target, .ReleaseSmall);
+    const install_mkdir = b.addInstallFile(mkdir_exe.getEmittedBin(), "mkdir.elf");
+    const kernel_mkdir_step = b.step("kernel-mkdir", "Build mkdir.elf (Phase 3.E)");
+    kernel_mkdir_step.dependOn(&install_mkdir.step);
+
+    const rm_exe = addUserBinary(b, "rm", "src/kernel/user/rm.zig", rv_target, .ReleaseSmall);
+    const install_rm = b.addInstallFile(rm_exe.getEmittedBin(), "rm.elf");
+    const kernel_rm_step = b.step("kernel-rm", "Build rm.elf (Phase 3.E)");
+    kernel_rm_step.dependOn(&install_rm.step);
+
+    const sh_exe = addUserBinary(b, "sh", "src/kernel/user/sh.zig", rv_target, .ReleaseSmall);
+    const install_sh = b.addInstallFile(sh_exe.getEmittedBin(), "sh.elf");
+    const kernel_sh_step = b.step("kernel-sh", "Build sh.elf (Phase 3.E)");
+    kernel_sh_step.dependOn(&install_sh.step);
+
     // Phase 3.D: mkfs host tool.
     const mkfs_exe = b.addExecutable(.{
         .name = "mkfs",
@@ -475,6 +516,30 @@ pub fn build(b: *std.Build) void {
     const fs_img_step = b.step("fs-img", "Build fs.img from staged userland + mkfs");
     fs_img_step.dependOn(&install_fs_img.step);
 
+    // Phase 3.E: shell-fs.img — install init_shell as /bin/init plus the
+    // six utility binaries (sh, ls, cat, echo, mkdir, rm). The shell-fs/
+    // staging tree carries /etc/motd and the empty /tmp/ directory.
+    const shell_fs_bin_stage = b.addWriteFiles();
+    _ = shell_fs_bin_stage.addCopyFile(init_shell_exe.getEmittedBin(), "init");
+    _ = shell_fs_bin_stage.addCopyFile(sh_exe.getEmittedBin(), "sh");
+    _ = shell_fs_bin_stage.addCopyFile(ls_exe.getEmittedBin(), "ls");
+    _ = shell_fs_bin_stage.addCopyFile(cat_exe.getEmittedBin(), "cat");
+    _ = shell_fs_bin_stage.addCopyFile(echo_exe.getEmittedBin(), "echo");
+    _ = shell_fs_bin_stage.addCopyFile(mkdir_exe.getEmittedBin(), "mkdir");
+    _ = shell_fs_bin_stage.addCopyFile(rm_exe.getEmittedBin(), "rm");
+
+    const shell_fs_img_run = b.addRunArtifact(mkfs_exe);
+    shell_fs_img_run.addArg("--root");
+    shell_fs_img_run.addDirectoryArg(b.path("src/kernel/userland/shell-fs"));
+    shell_fs_img_run.addArg("--bin");
+    shell_fs_img_run.addDirectoryArg(shell_fs_bin_stage.getDirectory());
+    shell_fs_img_run.addArg("--out");
+    const shell_fs_img = shell_fs_img_run.addOutputFileArg("shell-fs.img");
+
+    const install_shell_fs_img = b.addInstallFile(shell_fs_img, "shell-fs.img");
+    const shell_fs_img_step = b.step("shell-fs-img", "Build shell-fs.img with all Phase 3.E binaries");
+    shell_fs_img_step.dependOn(&install_shell_fs_img.step);
+
     const multi_boot_config_stub_dir = b.addWriteFiles();
     const multi_boot_config_zig = multi_boot_config_stub_dir.add(
         "boot_config.zig",
@@ -754,6 +819,24 @@ pub fn build(b: *std.Build) void {
     const e2e_fs_step = b.step("e2e-fs", "Run the Phase 3.D fs-read e2e test (init opens /etc/motd)");
     e2e_fs_step.dependOn(&e2e_fs_run.step);
 
+    const shell_e2e_exe = b.addExecutable(.{
+        .name = "e2e-shell",
+        .root_module = b.createModule(.{
+            .root_source_file = b.path("tests/e2e/shell.zig"),
+            .target = b.graph.host,
+            .optimize = .Debug,
+        }),
+    });
+    const shell_e2e_run = b.addRunArtifact(shell_e2e_exe);
+    shell_e2e_run.step.dependOn(b.getInstallStep());
+    shell_e2e_run.step.dependOn(shell_fs_img_step);
+    shell_e2e_run.addFileArg(exe.getEmittedBin());
+    shell_e2e_run.addFileArg(shell_fs_img);
+    shell_e2e_run.addFileArg(kernel_fs_elf.getEmittedBin());
+    shell_e2e_run.addFileArg(b.path("tests/e2e/shell_input.txt"));
+    const e2e_shell_step = b.step("e2e-shell", "Run the Phase 3.E shell e2e test");
+    e2e_shell_step.dependOn(&shell_e2e_run.step);
+
     // qemu-diff-kernel: debug-only trace diff against QEMU. Requires
     // qemu-system-riscv32 on PATH; not run by CI.
     const qemu_diff_kernel_cmd = b.addSystemCommand(&.{
@@ -1086,8 +1169,8 @@ pub fn build(b: *std.Build) void {
             },
         }),
     });
-    wasm_exe.entry = .disabled;        // we call our own export, not _start
-    wasm_exe.rdynamic = true;          // expose `export fn` symbols
+    wasm_exe.entry = .disabled; // we call our own export, not _start
+    wasm_exe.rdynamic = true; // expose `export fn` symbols
 
     const install_wasm = b.addInstallArtifact(wasm_exe, .{
         .dest_dir = .{ .override = .{ .custom = "web" } },
@@ -1103,3 +1186,48 @@ pub fn build(b: *std.Build) void {
     wasm_step.dependOn(&install_web_hello.step);
     wasm_step.dependOn(&install_web_snake.step);
 }
+
+/// Build a user binary by linking start.S + usys.S + ulib.zig + uprintf.zig +
+/// the binary's main.zig against user_linker.ld. Returns the executable so
+/// callers can wire its install step into a build target.
+fn addUserBinary(
+    b: *std.Build,
+    name: []const u8,
+    main_src: []const u8,
+    rv_target: std.Build.ResolvedTarget,
+    optimize: std.builtin.OptimizeMode,
+) *std.Build.Step.Compile {
+    // Compile the binary's main.zig as an object (its `@import("lib/ulib.zig")` etc.
+    // pulls ulib + uprintf in transitively).
+    const main_obj = b.addObject(.{
+        .name = name,
+        .root_module = b.createModule(.{
+            .root_source_file = b.path(main_src),
+            .target = rv_target,
+            .optimize = optimize,
+            // Strip debug sections — fs.img has 4 MB of data blocks total;
+            // 7 binaries × ~150 KB debug each won't fit. The kernel ELF
+            // loader only loads PT_LOAD segments anyway.
+            .strip = true,
+            .single_threaded = true,
+        }),
+    });
+
+    const exe = b.addExecutable(.{
+        .name = b.fmt("{s}.elf", .{name}),
+        .root_module = b.createModule(.{
+            .root_source_file = null,
+            .target = rv_target,
+            .optimize = optimize,
+            .strip = true,
+            .single_threaded = true,
+        }),
+    });
+    exe.root_module.addObject(main_obj);
+    exe.root_module.addAssemblyFile(b.path("src/kernel/user/lib/start.S"));
+    exe.root_module.addAssemblyFile(b.path("src/kernel/user/lib/usys.S"));
+    exe.setLinkerScript(b.path("src/kernel/user/user_linker.ld"));
+    exe.entry = .{ .symbol_name = "_start" };
+
+    return exe;
+}