macOS: stdout/stderr output and exit code not captured from VM execution

[Luis-yl]

Description

On macOS (Apple Silicon), ERA's VM execution completes successfully but stdout/stderr output files are empty and exit codes are always 0 regardless of actual command result.

Environment

• OS: macOS 14.x (Darwin 24.6.0) on Apple Silicon (M-series)
• krunvm version: 0.2.4
• libkrun version: 1.16.0 (via Homebrew slp/krun tap)
• ERA: Built from source (latest main branch)

Steps to Reproduce

1. Set up ERA on macOS following the setup guide
2. Run the integration test:

AGENT_STATE_DIR="/Volumes/krunvm/agent-state" \
KRUNVM_DATA_DIR="/Volumes/krunvm/agent-state/krunvm" \
DYLD_LIBRARY_PATH="/opt/homebrew/lib" \
go test -v -run TestVMWorkflow -timeout 120s

3. Observe the test failure

Expected Behavior

• stdout.log should contain "Hello from VM"
• Exit code should reflect the actual command result

Actual Behavior

=== RUN   TestVMWorkflow
    integration_test.go:70: Created VM: python-1764678044018167000
    integration_test.go:94: Expected output to contain 'Hello from VM', got: 
    integration_test.go:97: Command executed successfully, output: 
--- FAIL: TestVMWorkflow (4.63s)

The output files at $AGENT_STATE_DIR/vms/<vm-id>/out/stdout.log and stderr.log are 0 bytes.

Additionally, exit codes are not propagated:

• python -c "import sys; sys.exit(42)" returns exit_code=0
• python -c "raise ValueError()" returns exit_code=0

Root Cause Analysis

This appears to be related to known krunvm limitations:

1. krunvm Issue #21: Exit codes not returned - identified as a libkrun limitation requiring interface design for host-guest information transport.

2. krunvm Issue #22: The -v verbosity option is not yet implemented.

3. libkrun has krun_set_console_output() API (since v1.8.0) that can redirect console output to a file, but krunvm CLI does not expose this API.

Attempted Workarounds

1. AGENT_ENABLE_GUEST_VOLUMES=1: Causes exit code 255 on macOS due to volume mapping issues (as documented in README: "disabled by default to avoid macOS volume-mapping issues")

2. Direct krunvm test:

krunvm start <vm-id> -- python3 -c 'print("Hello")'
# No output printed, exit code 0

Impact

On macOS, ERA is essentially "fire and forget" - execution happens but there's no way to:

• Verify success or failure
• Capture command output
• Get actual exit codes

This makes ERA unsuitable for use cases requiring output validation on macOS.

Possible Solutions

1. Use krunkit instead of krunvm: krunkit may expose more libkrun APIs
2. Modify krunvm to support --console-output: Add CLI flag to call krun_set_console_output()
3. Implement output via shared volumes: Write output to /out/ from within VM (requires fixing guest volumes on macOS)
4. Document macOS limitation: If this is a fundamental limitation, clearly document it

Questions

1. Is output capture expected to work on macOS, or is this a known limitation?
2. Are the integration tests primarily run on Linux?
3. Is there a recommended workaround for macOS users who need output capture?

Thank you for creating ERA - it's a great project! Looking forward to understanding if there's a path forward for macOS support.

[BinSquare]

The native dev environment for this project is in linux.

1. Not known, we'll start testing on mac platforms to capture any platform bugs here.
2. Yes, linux is primary target.
3. I'm working with the maintainer of krunvm to fix some of the critical bugs on macOS as we're observing here: Fixing mapped volumes on MacOS by replacing deprecated method with krun_add_virtiofs. containers/krunvm#74 (comment).

[Luis-yl]

Looking forward to your fix. Thank you

[BinSquare]

This is fixed, you need to update to the newest krunvm version which has my fix merged: containers/krunvm#74 (comment)

---

Alternatively, I have been working on the next version of this with minimal dependencies in https://smolmachines.com/

Open source repo: https://github.com/smol-machines/smolvm