Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/publish-failproofai-sdk.yml
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,12 @@ jobs:
# dispatched, so a writer could delete them on a branch. The `environment:` above
# plus its two out-of-file settings (see the header) are what actually hold.
- name: Authorize actor and branch
# The `build` job runs every step inside the package directory, and this one
# runs BEFORE `actions/checkout` — so that directory does not exist yet and the
# runner cannot even start bash in it ("No such file or directory"), failing the
# guard open-endedly rather than on its own verdict. The workspace root exists
# before checkout; the package directory is created by it.
working-directory: ${{ github.workspace }}
env:
# Space-separated GitHub logins, compared case-insensitively — the same
# shape and the same semantics as publish.yml's STABLE_RELEASE_ACTORS.
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/publish-fp-cloud-cli.yml
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,12 @@ jobs:
# dispatched, so a writer could delete them on a branch. The `environment:` above
# plus its two out-of-file settings (see the header) are what actually hold.
- name: Authorize actor and branch
# The `build` job runs every step inside the package directory, and this one
# runs BEFORE `actions/checkout` — so that directory does not exist yet and the
# runner cannot even start bash in it ("No such file or directory"), failing the
# guard open-endedly rather than on its own verdict. The workspace root exists
# before checkout; the package directory is created by it.
working-directory: ${{ github.workspace }}
env:
# Space-separated GitHub logins, compared case-insensitively — the same
# shape and the same semantics as publish.yml's STABLE_RELEASE_ACTORS.
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@

### Fixes

- **Both PyPI publish workflows died on their own authorization step, before it could decide anything.** The `build` job runs every step inside the package directory (`fp-cloud-cli` / `sdk/python`) via a job-level `defaults.run.working-directory`, and the actor/branch guard is deliberately the FIRST step — ahead of `actions/checkout`, so an unauthorized dispatch stops before the runner does any work on its behalf. But the package directory arrives *with* the checkout, so the runner could not start bash in it at all: `An error occurred trying to start process '/usr/bin/bash' with working directory .../fp-cloud-cli. No such file or directory`, and the release stopped for a reason that has nothing to do with who dispatched it. Both workflows failed this way on the first real dispatch after the packages were open-sourced, taking `publish`, `bump` and `release` down as skipped. The guard now runs at `${{ github.workspace }}`, which exists before checkout. Nothing else in either file was affected — every other pre-checkout step sits in a job with no directory default. `__tests__/ci/workflow-step-working-directory.test.ts` fails on any `run:` step that inherits a repo-relative directory before the checkout that creates it, which is the half of this nothing else can see: publish workflows only run on `workflow_dispatch`, so no PR, no push and no schedule would ever have executed the step. (#753)

- **`docs/stars.js` shipped with no test, and it is the kind of code whose failure looks exactly like success.** A broken selector, a renamed API field or a navbar re-render that wins the race all leave a plausible number on the page, because `docs.json` still carries a hand-written one — and today's real count (1,492) formats to exactly the baked-in `⭐ 1.5k`, so even reading the live navbar cannot tell a working fetch from a dead one. `__tests__/docs/stars.test.ts` drives the script the way a browser does — eval it into a DOM, stub `fetch`, read the label back — over the real navbar markup captured from `mintlify dev`: the endpoint it calls, the replacement, the rounding boundaries the comments claim (9,950 is "10k" not "9.9k"; 999,999 is "1.0M" not "1000k"), the four failure paths that must leave the baked-in label untouched, the screen-reader `github` span it must never rewrite, the re-apply after a simulated SPA re-render, and the sessionStorage cache that keeps a second page load off an API that rate-limits at 60 requests/hour. Negative-controlled: renaming `stargazers_count` in the script reddens 12 of the 19. (#702)

- **Nothing tied a Python package to the two places that have to know about it, and both stay quiet when one is missing.** Dependabot resolves per directory — `fp-cloud-cli`'s entry cannot see `sdk/python`'s tree — so an unregistered package simply never gets an update PR, which is indistinguishable from a tree that is already current; and osv-scanner scans the lockfiles named on its command line and discovers none on its own, so a lockfile missing from those args is reported clean because it was never read. Both packages are wired today, and `failproofai-sdk-workflows.test.ts` asserts the SDK's dependabot entry as one of that package's own invariants, but nothing covered `fp-cloud-cli` or would notice a third package. `__tests__/ci/python-package-registration.test.ts` derives the package list from the `uv.lock` files on disk rather than restating it, and asserts each has a dependabot entry for its own directory, declares the `uv` ecosystem (`pip` resolves `pyproject.toml` and ignores the lockfile, so it would open PRs that leave `uv sync --locked` pinned) and is named in the osv-scanner run. (#702)
Expand Down
207 changes: 207 additions & 0 deletions __tests__/ci/workflow-step-working-directory.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
// @vitest-environment node
/**
* A `run:` step that executes before `actions/checkout` in a job whose DEFAULT
* working directory is a repo-relative path cannot start at all.
*
* The runner creates the workspace root before the job begins, but everything
* under it arrives with the checkout — so bash is asked to start in a directory
* that does not exist yet and the step dies with "An error occurred trying to
* start process '/usr/bin/bash' with working directory ... No such file or
* directory". The step never runs, so its own logic is irrelevant: both PyPI
* publish workflows put their actor/branch authorization first, deliberately
* ahead of the checkout, and both `build` jobs died there on the first real
* dispatch after the packages were open-sourced. A guard that cannot execute
* fails the release for the wrong reason, and hides whatever verdict it would
* have reached behind an infrastructure error.
*
* The failure is invisible to review — the step and the `defaults:` block are
* dozens of lines apart, usually in different hunks — and invisible to the rest
* of CI, because publish workflows only ever run on `workflow_dispatch`.
*
* Scope: only INHERITED directories. A step that names its own
* `working-directory` is an author stating that the path is there, next to
* whatever put it there — in this repo, `download-artifact` output directories
* (`release-assets`, `tree`), created by the step immediately above.
*/
import { describe, it, expect } from "vitest";
import { readFileSync, readdirSync } from "node:fs";
import { resolve } from "node:path";
import { parse } from "yaml";

const WORKFLOWS = resolve(process.cwd(), ".github/workflows");

interface Step {
name?: string;
uses?: string;
run?: string;
"working-directory"?: string;
}

interface Defaults {
run?: { "working-directory"?: string };
}

interface Job {
defaults?: Defaults;
steps?: Step[];
}

interface Workflow {
defaults?: Defaults;
jobs?: Record<string, Job>;
}

/**
* Both spellings GitHub accepts. Filtering on `.yml` alone would skip a `.yaml`
* workflow silently — the same shape of failure this file exists to catch.
*/
function isWorkflowFile(name: string): boolean {
return name.endsWith(".yml") || name.endsWith(".yaml");
}

/**
* A directory that is already there when the job starts: the workspace root, or
* an absolute path outside the repo. Anything else relative is created BY the
* checkout.
*/
function existsBeforeCheckout(dir: string): boolean {
return (
dir === "${{ github.workspace }}" ||
dir === "." ||
dir.startsWith("/") ||
dir.startsWith("${{ runner.temp }}")
);
}

/**
* The directory a step inherits when it names none of its own: the job's
* default, else the workflow-level one, which jobs inherit unless they override
* it.
*/
function inheritedDirectory(
job: Job,
workflowDefault?: string,
): string | undefined {
return job.defaults?.run?.["working-directory"] ?? workflowDefault;
}

/** Names of the `run:` steps in `job` that would start in a directory that does not exist yet. */
function stepsThatCannotStart(job: Job, workflowDefault?: string): string[] {
const inherited = inheritedDirectory(job, workflowDefault);
if (inherited === undefined || existsBeforeCheckout(inherited)) return [];

const steps = job.steps ?? [];
const checkout = steps.findIndex((s) =>
(s.uses ?? "").startsWith("actions/checkout"),
);
// No checkout at all means every step in the job runs before one.
const preCheckout = checkout === -1 ? steps : steps.slice(0, checkout);

return preCheckout
.filter((s) => s.run !== undefined && s["working-directory"] === undefined)
.map((s) => s.name ?? "(unnamed step)");
}

interface DiscoveredJob {
file: string;
name: string;
job: Job;
workflowDefault?: string;
}

const JOBS: DiscoveredJob[] = readdirSync(WORKFLOWS)
.filter(isWorkflowFile)
.flatMap((file) => {
const workflow = parse(
readFileSync(resolve(WORKFLOWS, file), "utf8"),
) as Workflow;
const workflowDefault = workflow.defaults?.run?.["working-directory"];
return Object.entries(workflow.jobs ?? {}).map(([name, job]) => ({
file,
name,
job,
workflowDefault,
}));
});

describe("workflow steps that run before checkout", () => {
it("reads every workflow, so the check below is not vacuous", () => {
expect(JOBS.length).toBeGreaterThan(20);
expect(
JOBS.some(({ job }) => job.defaults?.run?.["working-directory"]),
).toBe(true);
});

it("discovers both workflow file extensions", () => {
expect(isWorkflowFile("ci.yml")).toBe(true);
expect(isWorkflowFile("ci.yaml")).toBe(true);
expect(isWorkflowFile("README.md")).toBe(false);
});

it("never inherits a directory the checkout has not created yet", () => {
const broken = JOBS.flatMap(({ file, name, job, workflowDefault }) =>
stepsThatCannotStart(job, workflowDefault).map(
(step) =>
`${file} → ${name} → ${step} inherits "${inheritedDirectory(job, workflowDefault)}" before actions/checkout`,
),
);

expect(broken).toEqual([]);
});

// The detector itself, against the exact job shape that failed. Without this,
// a typo in the traversal above would report a clean sweep of nothing.
it("catches the shape that broke both publish workflows", () => {
const job: Job = {
defaults: { run: { "working-directory": "fp-cloud-cli" } },
steps: [
{ name: "Authorize actor and branch", run: "exit 0" },
{ uses: "actions/checkout@v7.0.1" },
{ name: "Test", run: "pytest" },
],
};

expect(stepsThatCannotStart(job)).toEqual(["Authorize actor and branch"]);

// ...and clears it once the step names a directory that already exists.
job.steps![0]["working-directory"] = "${{ github.workspace }}";
expect(stepsThatCannotStart(job)).toEqual([]);
});

// A workflow-level `defaults:` block reaches every job that does not override
// it, so a step can inherit the directory from a file no reviewer of the job
// is looking at.
it("catches a workflow-level default a job does not override", () => {
const job: Job = {
steps: [
{ name: "Authorize actor and branch", run: "exit 0" },
{ uses: "actions/checkout@v7.0.1" },
],
};

expect(stepsThatCannotStart(job, "fp-cloud-cli")).toEqual([
"Authorize actor and branch",
]);
expect(stepsThatCannotStart(job, "${{ github.workspace }}")).toEqual([]);
});

it("lets the job default win over the workflow default, both ways", () => {
const steps: Step[] = [
{ name: "Authorize actor and branch", run: "exit 0" },
{ uses: "actions/checkout@v7.0.1" },
];
const jobAtRoot: Job = {
defaults: { run: { "working-directory": "${{ github.workspace }}" } },
steps,
};
const jobInPackage: Job = {
defaults: { run: { "working-directory": "sdk/python" } },
steps,
};

expect(stepsThatCannotStart(jobAtRoot, "sdk/python")).toEqual([]);
expect(
stepsThatCannotStart(jobInPackage, "${{ github.workspace }}"),
).toEqual(["Authorize actor and branch"]);
});
});