Add external mode for GitHub Action by matheusmra · Pull Request #297 · webtech-network/autograder

matheusmra · 2026-04-21T18:17:27Z

This pull request introduces support for an "external mode" in the Autograder GitHub Action, allowing grading configuration and result submission to be handled via the Autograder Cloud API instead of repository-local files. The changes add new action inputs and environment variables, document the external mode workflow, and update both the abstract exporter interface and the export step to support richer context. Documentation is significantly expanded to cover usage, configuration, and error handling for external mode.

External Mode Integration

Added new inputs to action.yml for external mode: execution-mode, grading-config-id, autograder-cloud-url, autograder-cloud-token, and submission-language, with corresponding environment variable exports. [1] [2]
Comprehensive documentation for external mode added to docs/github_action/README.md, docs/github_action/configuration.md, and docs/API.md, including usage instructions, workflow examples, and error handling. [1] [2] [3] [4] [5] [6] [7] [8] [9]

Exporter Interface and Pipeline Updates

Enhanced the Exporter abstract class with a new export_with_context method, allowing exporters to access the full PipelineExecution for richer exports; default implementation maintains backward compatibility. [1] [2] [3]
Updated ExportStep to call export_with_context instead of the legacy export method, enabling external mode exporters to access all grading context. [1] [2]

Behavioral Notes and Validation

In external mode, include-feedback is taken from the cloud config, not the action input; submission-language is validated against the config's supported languages. All required secrets and inputs for external mode are documented, with validation and error handling described. [1] [2] [3]

These changes make it possible to centrally manage grading configurations and results using Autograder Cloud, improving scalability for organizations with many assignments or courses.

Introduce an "external" execution mode allowing the Action to fetch grading configs from Autograder Cloud and post results back. Adds CloudClient and CloudExporter to handle config fetch and result submission (with retry/4xx handling), new docs and rollout guide, and workflow/action inputs for execution-mode, grading-config-id, autograder-cloud-url, autograder-cloud-token, and submission-language. Exporter API extended with export_with_context and ExporterStep now calls export_with_context so exporters can submit full pipeline data; GithubActionService gains pipeline builder and failure-submission helpers for cloud mode. Entrypoint and main updated to validate and forward external-mode args. Unit tests for cloud client/exporter and main/service wiring added.

…hub-action-inputs-mode-plumbing-and-compatibility

Copilot

Pull request overview

Adds an “external” execution mode to the Autograder GitHub Action so it can fetch grading configuration and submit results via the Autograder Cloud API instead of relying on repository-local .github/autograder/*.json files. This extends the exporter interface to support richer context and introduces new cloud client/exporter components, with expanded docs and tests.

Changes:

Add new GitHub Action inputs/env vars and entrypoint validation for execution-mode=external plus cloud connection parameters.
Introduce CloudClient + CloudExporter and update the pipeline export path to call export_with_context(PipelineExecution).
Expand documentation and add unit/web tests for external mode integration.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
`action.yml`	Adds external-mode inputs and exports them as env vars.
`github_action/entrypoint.sh`	Validates required external-mode env vars and forwards new CLI args.
`github_action/main.py`	Adds CLI flags + routing logic for external mode and failure submission path.
`github_action/github_action_service.py`	Builds pipelines from cloud config, wires `CloudExporter`, adds failure submission helper.
`github_action/cloud_client.py`	New HTTP client with retry/backoff for cloud config fetch + result submission.
`github_action/cloud_exporter.py`	New exporter that builds external-results payloads from `PipelineExecution`.
`autograder/models/abstract/exporter.py`	Adds `export_with_context()` default implementation for backward compatibility.
`autograder/steps/export_step.py`	Switches export step to call `export_with_context()`.
`docs/github_action/README.md`	Documents repo vs external mode behavior and new components.
`docs/github_action/configuration.md`	Adds external-mode workflow example + input/secret reference.
`docs/github_action/external-mode.md`	New deep-dive doc for external mode call sequence and troubleshooting.
`docs/guides/external-mode-rollout.md`	New rollout/migration/ops checklist for instructors/operators.
`docs/API.md`	Adds an “External Mode Integration” section describing API interaction.
`tests/unit/github_action/test_main.py`	Adds arg parsing/routing/failure-path tests for external mode.
`tests/unit/github_action/test_github_action_service.py`	Adds tests for `autograder_pipeline_from_cloud()` and failure submission.
`tests/unit/github_action/test_cloud_client.py`	New tests for cloud client URL building + retry/error handling behavior.
`tests/unit/github_action/test_cloud_exporter.py`	New tests for payload structure and graceful degradation.
`tests/web/test_integration_auth.py`	Formatting/comment cleanup around auth integration tests.
`tests/web/test_external_results.py`	Formatting/comment cleanup around external-results endpoint tests.

Copilot · 2026-04-21T18:23:49Z

+    grading_config_id="cfg-123",
+    autograder_cloud_url="https://cloud.example.com",
+    autograder_cloud_token="cloud-tok",
+    include_feedback=None,
+):
+    """Build a sys.argv list for external execution mode."""
+    argv = [
+        "entrypoint",
+        "--github-token", github_token,
+        "--template-preset", template_preset,
+        "--student-name", student_name,
+        "--feedback-type", feedback_type,
+        "--app-token", app_token,
+        "--execution-mode", "external",
+        "--grading-config-id", grading_config_id,
+        "--autograder-cloud-url", autograder_cloud_url,
+        "--autograder-cloud-token", autograder_cloud_token,


The helper defaults grading_config_id to values like "cfg-123", but the cloud config-by-id endpoint expects an integer ID. Using numeric-looking defaults (e.g. "123") in argv builders will keep the CLI tests aligned with the real API contract and with the docs’ examples.

Copilot · 2026-04-21T18:23:49Z

        external_user_id = pipeline_exec.submission.user_id
        score = pipeline_exec.get_grade_step_result().final_score
-        feedback: Optional[str] = pipeline_exec.get_feedback()

        logger.info("Exporting result: external_user_id=%s, score=%.2f", external_user_id, score)
-        self._exporter_service.export(external_user_id, score, feedback)
+        self._exporter_service.export_with_context(pipeline_exec)


ExporterStep still calls pipeline_exec.get_grade_step_result() to compute score purely for logging. That will raise if the GRADE step didn’t produce data, preventing exporters (like CloudExporter) from handling missing grade results gracefully inside export_with_context(). Consider either (a) not precomputing score here, or (b) guarding it (e.g., log without score / log score only when available) and letting export_with_context() own score extraction.

Copilot · 2026-04-21T18:23:49Z

+        except Exception as exc:  # noqa: BLE001
+            logger.warning("Failed to submit failure payload to cloud: %s", exc)


except Exception here will trigger pylint’s broad-exception-caught (enabled in this repo) and can fail CI. If the intent is to swallow only network / cloud submission errors, catch the specific exceptions raised by CloudExporter/CloudClient (e.g., CloudClientError/CloudConnectionError/URLError) or add an explicit # pylint: disable=broad-exception-caught with a short justification.

Suggested change

except Exception as exc: # noqa: BLE001

logger.warning("Failed to submit failure payload to cloud: %s", exc)

except Exception as exc: # noqa: BLE001 # pylint: disable=broad-exception-caught

logger.warning(

"Failed to submit failure payload to cloud: %s", exc

)

Copilot · 2026-04-21T18:23:50Z

+from github_action.cloud_client import CloudClient
+
+if TYPE_CHECKING:
+    from autograder.models.pipeline_execution import PipelineExecution, PipelineStatus


PipelineStatus is imported under TYPE_CHECKING but never used, and this repo enables pylint’s unused-import. Removing PipelineStatus from the import (or using it) will avoid a CI lint failure.

Suggested change

from autograder.models.pipeline_execution import PipelineExecution, PipelineStatus

from autograder.models.pipeline_execution import PipelineExecution

Copilot · 2026-04-21T18:23:50Z

+        if not args.grading_config_id:
+            raise ValueError(
+                "grading-config-id is required when execution-mode is 'external'."
+            )


grading-config-id is treated as a free-form string, but the API route it targets is typed as an integer (GET /api/v1/configs/id/{config_id} where config_id: int). Consider validating/coercing this argument to an int in __parser_values (or making the argparse type=int) so users get a clear local error instead of a 422/4xx from the cloud.

Suggested change

)

)

try:

args.grading_config_id = int(args.grading_config_id)

except (TypeError, ValueError) as exc:

raise ValueError(

"grading-config-id must be an integer when execution-mode is 'external'."

) from exc

Copilot · 2026-04-21T18:23:50Z

+    "github_repository": "org/repo",
+    "github_sha": "abc123",
+    "github_run_id": "999",
+    "github_actor": "student1",
+    "github_ref": "refs/heads/main"


The example submission_metadata keys here (github_repository, github_sha, etc.) don’t match what the GitHub Action currently sends in external mode (CloudExporter uses repository, commit_sha, run_id, actor, ref). Since submission_metadata is persisted as-is, it would be good to align the docs with the actual payload shape to avoid confusion for API consumers.

Suggested change

"github_repository": "org/repo",

"github_sha": "abc123",

"github_run_id": "999",

"github_actor": "student1",

"github_ref": "refs/heads/main"

"repository": "org/repo",

"commit_sha": "abc123",

"run_id": "999",

"actor": "student1",

"ref": "refs/heads/main"

Copilot · 2026-04-21T18:23:51Z

+            client.get_grading_config("cfg-42")
+
+        assert len(captured_req) == 1
+        assert captured_req[0].full_url == "https://cloud.example.com/api/v1/configs/id/cfg-42"
+


These tests use non-numeric config IDs like cfg-42, but the actual endpoint is GET /api/v1/configs/id/{config_id} with config_id typed as an integer. Using numeric IDs in tests (e.g. 42) will better reflect real behavior and catch type/validation issues earlier.

Copilot · 2026-04-21T18:23:51Z

+            result = service.autograder_pipeline_from_cloud(
+                grading_config_id="cfg-7",
+                cloud_url="https://cloud.example.com",
+                cloud_token="tok",
+                feedback_mode="default",
+                student_name="alice",
+                submission_language=language,
+            )


grading_config_id is passed as values like "cfg-7" in these tests, but the cloud endpoint expects an integer config ID (/api/v1/configs/id/{config_id} where config_id: int). Updating the test inputs to use numeric IDs will keep them consistent with the API and the external-mode docs.

matheusmra · 2026-04-21T18:34:46Z

@ArthurCRodrigues

ArthurCRodrigues · 2026-04-21T19:05:54Z

@matheusmra did you test it with actual repos?

…plumbing-and-compatibility

Copilot AI review requested due to automatic review settings April 21, 2026 18:17

matheusmra linked an issue Apr 21, 2026 that may be closed by this pull request

[External Mode] GitHub Action inputs, mode plumbing, and compatibility #285

Closed

Copilot started reviewing on behalf of matheusmra April 21, 2026 18:17 View session

Merge remote-tracking branch 'origin/main' into 285-external-mode-git…

c425791

…hub-action-inputs-mode-plumbing-and-compatibility

Copilot AI reviewed Apr 21, 2026

View reviewed changes

fixing tests

f616ba5

matheusmra requested a review from ArthurCRodrigues April 21, 2026 18:34

matheusmra added 4 commits April 21, 2026 17:28

fix: pass resolved Language to Submission in external mode

6a457fb

feat: add locale support to external mode

1ca9016

fix: pass locale to Submission in external mode

3bb81f6

fix unit test

0e42a9e

ArthurCRodrigues approved these changes Apr 21, 2026

View reviewed changes

Merge branch 'main' into 285-external-mode-github-action-inputs-mode-…

9e1e11d

…plumbing-and-compatibility

ArthurCRodrigues merged commit 39c93d6 into main Apr 21, 2026
3 checks passed

ArthurCRodrigues deleted the 285-external-mode-github-action-inputs-mode-plumbing-and-compatibility branch April 21, 2026 21:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add external mode for GitHub Action#297

Add external mode for GitHub Action#297
ArthurCRodrigues merged 8 commits intomainfrom
285-external-mode-github-action-inputs-mode-plumbing-and-compatibility

matheusmra commented Apr 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 21, 2026

Uh oh!

Copilot AI Apr 21, 2026

Uh oh!

Copilot AI Apr 21, 2026

Uh oh!

Copilot AI Apr 21, 2026

Uh oh!

Copilot AI Apr 21, 2026

Uh oh!

Copilot AI Apr 21, 2026

Uh oh!

Copilot AI Apr 21, 2026

Uh oh!

Copilot AI Apr 21, 2026

Uh oh!

matheusmra commented Apr 21, 2026

Uh oh!

ArthurCRodrigues commented Apr 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		except Exception as exc: # noqa: BLE001
		logger.warning("Failed to submit failure payload to cloud: %s", exc)

	from autograder.models.pipeline_execution import PipelineExecution, PipelineStatus
	from autograder.models.pipeline_execution import PipelineExecution

-            )
+            )
+        try:
+            args.grading_config_id = int(args.grading_config_id)
+        except (TypeError, ValueError) as exc:
+            raise ValueError(
+                "grading-config-id must be an integer when execution-mode is 'external'."
+            ) from exc

Conversation

matheusmra commented Apr 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

matheusmra commented Apr 21, 2026

Uh oh!

ArthurCRodrigues commented Apr 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants