-
Notifications
You must be signed in to change notification settings - Fork 12
test(cli): pin grandchild output capture and exit propagation #320
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -789,6 +789,45 @@ mod redacted_env { | |
| } | ||
| } | ||
|
|
||
| // ============================================================ | ||
| // Group 13: Self-asserting tasks (openjd-specifications conformance shape) | ||
| // ============================================================ | ||
|
|
||
| /// The conformance suite's single-task job fixtures assert their own output: the | ||
| /// `onRun` action spawns the case's command as a child, reproduces its output, and | ||
| /// exits non-zero when that output does not match. That only holds if we capture a | ||
| /// grandchild's output and propagate the action's exit status, so pin both | ||
| /// directions here instead of relying on the external suite to catch a regression. | ||
| mod self_asserting_task { | ||
| use super::*; | ||
|
|
||
| #[test] | ||
| fn test_grandchild_output_captured_and_assertion_passes() { | ||
| let template = templates_dir().join("self_asserting_task.yaml"); | ||
| let (code, stdout, stderr) = run_cli(&["run", template.to_str().unwrap()]); | ||
| assert_eq!(code, 0, "should succeed. stderr: {stderr}"); | ||
| // Printed by the grandchild and echoed by the action. Missing means output | ||
| // from a process we did not spawn ourselves was dropped. | ||
| assert!(stdout.contains("OUTPUT:EXPECTED_VALUE"), "stdout: {stdout}"); | ||
| assert!(!stdout.contains("ASSERT_FAILED"), "stdout: {stdout}"); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This assertion can never fire. The fixture writes |
||
| } | ||
|
|
||
| #[test] | ||
| fn test_assertion_failure_fails_the_run() { | ||
| let template = templates_dir().join("self_asserting_task.yaml"); | ||
| let (code, stdout, stderr) = run_cli(&[ | ||
| "run", | ||
| template.to_str().unwrap(), | ||
| "-p", | ||
| "Printed=WRONG_VALUE", | ||
| ]); | ||
| let output = format!("{stdout}{stderr}"); | ||
| assert_ne!(code, 0, "should fail. stdout: {stdout} stderr: {stderr}"); | ||
| assert!(output.contains("ASSERT_FAILED"), "output: {output}"); | ||
| assert!(output.contains("OUTPUT:WRONG_VALUE"), "output: {output}"); | ||
| } | ||
| } | ||
|
|
||
| // ============================================================ | ||
| // Group 6: Run Command (test_run_command.py) | ||
| // ============================================================ | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| specificationVersion: "jobtemplate-2023-09" | ||
| name: Self Asserting Task | ||
| description: > | ||
| An onRun action that spawns its own child process, reproduces that child's | ||
| output, and turns a comparison of it into the task's exit status. This is the | ||
| shape the openjd-specifications conformance suite uses for its single-task job | ||
| fixtures, so losing grandchild output or dropping the action's exit status | ||
| silently weakens that whole suite. | ||
|
|
||
| parameterDefinitions: | ||
| - name: Printed | ||
| type: STRING | ||
| default: "EXPECTED_VALUE" | ||
|
|
||
| steps: | ||
| - name: Assert | ||
| script: | ||
| actions: | ||
| onRun: | ||
| command: python | ||
| args: ["{{Task.File.Assert}}"] | ||
| embeddedFiles: | ||
| - name: Assert | ||
| type: TEXT | ||
| data: | | ||
| import subprocess | ||
| import sys | ||
|
|
||
| COMMAND = [sys.executable, "-c", "print(r'OUTPUT:{{Param.Printed}}')"] | ||
| EXPECTED = ["OUTPUT:EXPECTED_VALUE"] | ||
|
|
||
| completed = subprocess.run(COMMAND, capture_output=True, text=True) | ||
| sys.stdout.write(completed.stdout) | ||
| sys.stderr.write(completed.stderr) | ||
| output = completed.stdout + completed.stderr | ||
|
|
||
| missing = [line for line in EXPECTED if line not in output] | ||
| for _ in missing: | ||
| sys.stderr.write("ASSERT_FAILED: expected output line not found\n") | ||
| sys.exit(1 if missing else 0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The grandchild-output property this module says it pins is not actually exercised.
In
self_asserting_task.yamlthe action script runssubprocess.run(COMMAND, capture_output=True, ...)and then re-writescompleted.stdout/completed.stderrto its own stdout/stderr. The grandchild's pipes are read and closed by the intermediate Python process — openjd only ever sees the direct child's fds. So this test would still pass if openjd dropped grandchild output entirely; it only re-verifies direct-child capture, which is already covered elsewhere in this file.To actually pin the stated invariant, the grandchild has to inherit the action's stdout/stderr rather than have them captured, e.g. drop
capture_output=Trueand check the returncode instead:and then assert on openjd's captured stdout from the Rust side. As written, the second test (
test_assertion_failure_fails_the_run) also largely duplicates the existingtest_task_fails_still_exits_envexit-status coverage.