Summary
The current input_output template mixes execution-based tests with static analysis (AST) tests (forbidden_import and forbidden_keyword). These static analysis tests should be migrated to a dedicated static_analysis template, which doesn't require a sandbox.
Additionally, the autograder pipeline currently only supports running a single template per assignment. It needs to be refactored to support multiple templates (accepting a list of template names), so teachers can use both input_output and static_analysis tests in the same assignment without duplicate grading runs.
Acceptance Criteria
- Create a new
static_analysis template containing AST and static analysis tests.
- Remove those tests from
input_output.
- Update
PipelineExecution and TemplateLoaderStep to support a list of templates.
- Update
CriteriaTreeService to lookup tests across all loaded templates.
- Update
SandboxStep and GradeStep to check if any of the loaded templates require a sandbox.
- Ensure API and database layers do not break (e.g. support comma-separated string of templates).
- Ensure all unit tests pass with the new architecture.
Summary
The current
input_outputtemplate mixes execution-based tests with static analysis (AST) tests (forbidden_importandforbidden_keyword). These static analysis tests should be migrated to a dedicatedstatic_analysistemplate, which doesn't require a sandbox.Additionally, the autograder pipeline currently only supports running a single template per assignment. It needs to be refactored to support multiple templates (accepting a list of template names), so teachers can use both
input_outputandstatic_analysistests in the same assignment without duplicate grading runs.Acceptance Criteria
static_analysistemplate containing AST and static analysis tests.input_output.PipelineExecutionandTemplateLoaderStepto support a list of templates.CriteriaTreeServiceto lookup tests across all loaded templates.SandboxStepandGradeStepto check if any of the loaded templates require a sandbox.