Skip to content
BranchesTags

Repository files navigation

Scoring Pipeline

A scoring pipeline for the AIxCC competition.

Dev Setup

This repo uses Git submodules. Please make sure to run git submodule update --init --recursive if this is the first time you've cloned the repo.

We use dev containers for our development process. In VSCode, usage is as simple as using the dev containers extension; you may have a different experience with your chosen editor.

The scoring pipeline uses PostgreSQL for database functionality. Paired with SQLModel ( a rider on top of SQLAlchemy ), we've got a powerful toolkit. To interact with a development version of this database, ensure that you copy .devcontainer/.env.example to .devcontainer/.env, and add your desired credentials into it.

In VSCode, click the little green arrow icon in the bottom-left corner, and say "open in container". The container should build, and you should be able to drop into a shell.

If you don't wish to use dev containers, your mileage may vary. As it stands, we're using ubuntu linux 22.04 with python 3.11.6 and poetry 2.1.1.

When using Ubuntu, dependencies can be installed using the setup script in the .devcontainer folder:

./.devcontainer/setup_devcontainer.sh

This script does not need to be run if using a dev container as it is called during setup.

This is a poetry project. To generate the necessary dependencies and install the package, you can run ( on linux ):

poetry install  # Installs the package and dependencies
eval $(poetry env activate)  # Activates the virtual environment

You don't need to run poetry install if you're using a dev container. There will be an environment ready for you, and inside of the container you can simply run the activation command.

We use pre-commit to vet our code. You can add it with poetry run pre-commit install, and then utilize it against the python files in the project manually by running the top-level run_pre_commit_all.sh script. You can skip the pre-commit step by running git commit -m "... your message ..." --no-verify, but it isn't really recommended.

How do I run the pipeline?

You can run the scoring pipeline in two ways:

Method 1: Command Line Interface (CLI)

  1. Follow "Dev Setup" above
  2. Get the correct environment variables together
  3. Download an audit log and place it at the top of the repository with the correct name, e.g. audit-<round id>.jsonl
  4. Run poetry run python3 scoring/plugins/runner.py -s -r <round id> ( e.g. final -- this is based on your audit log file name. Collaborate with other scoring team members, because caching depends on this ) from the top of the repository

And the pipeline will execute.

Method 2: REST API

The scoring pipeline also provides a REST API for remote execution. The API server can be started with:

poetry run uvicorn --reload --host 0.0.0.0 --port 8000 scoring.api.main:app

API Endpoints

POST /v1/scoring/start

Start a new scoring run with an uploaded audit log file.

Parameters:

  • audit_log: (file, required): The audit log file to process
  • round_id: (string, required): Round ID for the scoring run
  • disable_pov_validation: (boolean, optional): Disable PoV validation during scoring (default: false)
  • force_reprocessing: (boolean, optional): Force reprocessing of cached results (default: false)
  • skip_scantron_tests: (boolean, optional): Skip scantron tests during scoring (default: true)
  • rescore_passed: (boolean, optional): Rescore passed results from the audit log (default: false)
  • rescore_failed: (boolean, optional): Rescore failed results from the audit log (default: false)
  • rescore_errored: (boolean, optional): Rescore errored results from the audit log (default: false)

Example using curl:

# Basic usage with defaults
curl -X POST "localhost:8000/v1/scoring/start" \
  -F "audit_log=@audit-final.jsonl" \
  -F "round_id=final"

# With custom parameters
curl -X POST "localhost:8000/v1/scoring/start" \
  -F "audit_log=@audit-final.jsonl" \
  -F "round_id=final" \
  -F "disable_pov_validation=true" \
  -F "force_reprocessing=true" \
  -F "skip_scantron_tests=false" \
  -F "rescore_failed=true"

Response:

{
  "scoring_run_uuid": "550e8400-e29b-41d4-a716-446655440000",
  "message": "Scoring run 550e8400-e29b-41d4-a716-446655440000 started successfully"
}

GET /v1/scoring/{scoring_run_uuid}/status

Get the current status of a scoring run.

Response:

{
  "scoring_run_uuid": "550e8400-e29b-41d4-a716-446655440000",
  "status": "running",
  "round_id": "final",
  "created_at": "2025-01-15T10:30:00Z",
  "database_name": "scoring_run_550e8400_e29b_41d4_a716_446655440000"
}

GET /v1/scoring/{scoring_run_uuid}/logs

Get logs for a specific scoring run (placeholder implementation).

GET /v1/scoring

List all scoring runs (placeholder implementation).

GET /health

Health check endpoint.

Response:

{
  "status": "healthy",
  "service": "scoring-pipeline-api",
  "version": "1.0.0"
}

API Parameter Details

The API exposes the same configuration options available in the CLI:

  • disable_pov_validation: When true, skips validation of Proof-of-Vulnerability (PoV) execution results. Useful for faster processing when PoV validation isn't required.

  • force_reprocessing: When true, forces reprocessing of all cached results, ignoring previous computation results. This will significantly increase processing time but ensures fresh results.

  • skip_scantron_tests: When false, includes scantron API tests in the scoring process. By default (true), these tests are skipped for faster execution.

  • rescore_failed: When true, attempts to rescore results that were previously marked as failed. Useful for reprocessing after fixes to scoring logic.

  • rescore_passed: When true, attempts to rescore results that were previously marked as passed. Useful for reprocessing after fixes to scoring logic.

  • rescore_errored: When true, attempts to rescore results that were previously marked as errored. Useful for reprocessing after fixes to scoring logic.

Production API Deployment

In production, the API server starts automatically as part of the Docker container's ENTRYPOINT. The server is accessible on port 8000 and joins the Tailscale network for secure access.

Environment Variables

Various environment variables can be configured to connect to external services and modify the operation of the scoring pipeline.

Logging

Logging level configuration:

  • LOG_LEVEL: Defaults to INFO. Other options are DEBUG, INFO, WARNING, ERROR, and CRITICAL

Postgres

Postgres database configuration variables:

  • POSTGRES_HOST: Address of the Postgres host
  • POSTGRES_PORT: Port for the Postgres host
  • POSTGRES_DB: Name of the database on the Postgres host
  • POSTGRES_USER: Username for the Postgres database
  • POSTGRES_PASSWORD: Password for the Postgres user

Amazon Web Services Storage

AWS storage is used to store artifacts that are related to the competition. If these need to be downloaded, we source them from S3.

  • AWS_STORAGE_URI: AWS server URI
  • AWS_ACCESS_KEY: AWS access key
  • AWS_SECRET_KEY: AWS secret key

NOTE: By default, we assume that the artefacts will be under the prefix workingset/auditlog/final/objects. If your artefacts are somewhere else, you can set a different prefix with the environment variable STORAGE_PREFIX_OVERRIDE. DO NOT include a leading /. For instance, nightly runs' artefacts are stored elsewhere.

Threading

Maximum thread count management for parent and child tasks.

  • MAX_PARENT_TASK_THREADS: Maximum number of parent threads used to spawn child threads. Defaults to 1/2 the number of processors for the host operating system.
  • MAX_CHILD_TASK_THREADS: Maximum number of child threads used to perform work. Defaults to the number of processors for the host operating system.

Deduplication Plugins

Deduplication Plugin Configuration. These variables control which plugins are used for PoV and Patch deduplication:

  • POV_DEDUP_PLUGINS: Controls which PoV deduplication plugins to use
  • PATCH_DEDUP_PLUGINS: Controls which patch deduplication plugins to use

Both variables can be set to:

  • "DEFAULT": Use the default plugin set for that type

  • Comma-separated Plugin Names: Use only the fowling available plugins in a comma separated list:

    • PoV plugins: PoVIsCopy, PoVIsMatch, PoVPatchValueAddedDeduplicator

    • Patch plugins: PatchIsCopy, PatchIsMatch, PatchVulnDeduplicator

Example usage:

# Use only specific PoV plugins
POV_DEDUP_PLUGINS="PoVIsCopy,PoVIsMatch"

# Use default patch plugins
PATCH_DEDUP_PLUGINS="DEFAULT"

Competition API

Note: 'Scantron' is our colloquial term for the Competition API built by the AIxCC team. Those terms can ( and will ) be used interchangeably.

'Scantron' Scoring API Credentials. These help us do our patch computations. Scantron will store previously calculated results for requests that are retrieved using the cache key when available.

  • SCANTRON_FQDN: Fully qualified domain address of the scantron server
  • SCANTRON_PORT: Port of the scantron API
  • SCANTRON_USER: Username for the scantron API
  • SCANTRON_TOKEN: User token for the scantron API
  • SCANTRON_SCHEME: Request type for scantron. Defaults to HTTPS
  • SCANTRON_TIMEOUT: Scantron job timeout. Defaults to 60 seconds
  • SCANTRON_PRERUN_TIMEOUT_HOURS: Max runtime for calculating all of the patch and PoV combination with scantron. Defaults to 48 hours
  • SCANTRON_CACHE_VERSION: Version key to seed the cache key for cached scantron requests. Defaults to 1

Resource Directory

Resources directory stores the input and output files for the scoring run.

  • RESOURCES_PATH: Path to the resources directory. Defaults to /resources

Scantron Cache Remap and Override

Scantron cache depends on the UUIDs of challenge tasks, patches, and PoVs. These UUIDs can be used to override or generate new scantron caches. The file stored at CACHE_REMAP_FILEPATH will remap old UUIDs to new UUIDs. CACHE_OVERRIDE_FILEPATH Points to a file that invalidates the cache entries containing any UUIDs stored in the file.

  • CACHE_REMAP_FILEPATH: Path to a cache JSON file used to remap UUIDs. This file should contain a mapping from old Task, Patch, and PoV UUIDs to new UUIDs. When a cache entry contains an old UUID as a key in this mapping, it will be remapped to the corresponding new UUID, and a new calculation will be performed. The file must be in the following JSON format:
{
  "old_uuid_a": "new_uuid_a",
  "old_uuid_b": "new_uuid_b",
  ...
}
  • CACHE_OVERRIDE_FILEPATH: Path to a JSON file used to override cached scantron results. This file contains a list of UUIDs that will invalidate a current cache entry in scantron and request a new calculation to be performed. The file must be in the JSON format:
[
  "uuid_a",
  "uuid_b",
  ...
]

Data Model Discussion + ORM Usage Outline

Please see the README at scoring/model/README.md.

Running Tests

Tests are found in the tests directory. Using a containerized setup like above, you can run the current set of tests using poetry run pytest tests/db/, for example. We use a pytest fixture for our testing setup: you can see an example of getting a test database adapter and using it in a test under tests/db/test_plugin_load_audit_logs.py.

Database Interaction

Use your favorite client. We recommend pgAdmin, because it is free and open-source.

You'll need to add the server. Press the big 'Add Server' button, and navigate to the 'Connection' tab. It should look something like this, based on the credentials in your .devcontainer/.env:

PgAdmin Connection Setup.

The exception is Host name/address, which needs to match the service entry for the database in .devcontainer/docker-compose.yml.

Creating Database Dumps and Loading Them

You may want to just give a copy of the database to somebody. Good for you, helping out. You can use the pg_dump command like this to do so:

pg_dump -h localhost \
        -p 5432 \
        -U postgres \
        -d scoring_pipeline_database \
        --create \
        --format=c \
        -f 2025-05-15-commit-4e18b50ae54-final.dump

Where -f supplies the name of the dump once it's done. As a note, if your customer is going to use the scoring pipeline ORM for database accesses, it's important that you provide them the commit at which the data was generated so that they can check out the correct commit of the scoring pipeline. I put mine right in the filename. No, .dump is not the required file extension.

Restoring the dump is not difficult. You should be running a postgres instance locally. May I suggest:

docker run -e POSTGRES_PASSWORD=postgres -p 5433:5432 postgres:15

The version of postgres that you run in order to restore the dump needs to coincide with the version of pg_dump that was used to produce the dump in the first place. Postgres is very finicky with versions, and so if I have e.g. pg_dump version 16 but you run a Postgres 14 container, you may not be able to pg_restore my dump to that container.

From a terminal you can run

pg_restore -h localhost \
           -p 5433 \
           -U postgres \
           -d postgres \
           --create /path/to/the/file/to/restore.dump

Database dumps can be large. The de-facto standard way I've come up with for sharing them with our team is to place them in Google Cloud Storage, in the afc-dev-share bucket, and then pointing people at them. This allows us to keep a record of the database dumps we've saved and shared.

Design Principles

The scoring pipeline is an extremely basic data pipeline with an underlying postgres database and a finite number of worker plugins.

The Main Service

The Scoring Pipeline runs very simply: it hosts a number of plugins that are run in a particular order. When one plugin ends, the runner will run the next plugin. Statuses of the runs are saved to the database. Barring a serious error, the pipeline will run to the end.

flowchart LR;
   scorer_runner(Scorer Runner);
   plugin_1(Plugin 1);
   plugin_n(Plugin N);
   postgres[(Postgres)];
   scorer_runner -- Do Work --> plugin_1;
   plugin_1 -- Work Results, Processing Status --> postgres;
   plugin_1 -- OK, Done --> scorer_runner
   scorer_runner -- Do Work --> plugin_n;
   plugin_n -- OK, Done --> scorer_runner;
  
   plugin_n -- Work Results, Processing Status --> postgres;
Loading

The scorer runner does its best to run idempotently, and also attempts to skip re-execution of tasks whenever it identifies that it has already done a block of work. The most common skipping criteria is the identification of a completed work run for a combination of Task, Team, and Plugin Type. That is to say -- if the plugin has already run for a given task for a given team, it won't run again unless force_reprocess is found in the input event. This determination is made based on the process status table.

Workloads: Using the Competition API

Some plugins need to evaluate vulnerabilities and patches, and therefore need our infrastructure for build_cr, etc. These tasks will outsource these efforts to the Competition API, which has a number of capabilities:

  • Running a PoV in a certain challenge repository
  • Patching a particular PoV and verifying that the patch functions correctly ( or not! )
  • Verifying that a certain challenge repository works correctly

Patch Computation and Parallelism

Patch computation is very easily the most difficult and time consuming part of the pipeline. In order to determine whether a patch mitigates a given vulnerability, we must determine that the patch mitigates every PoV bucketed to that vulnerability, and thus we must at least test patches against PoVs until one fails. We do have some deduplication measures in-place, but no matter what, this is going to be a large amount of compute. Without concurrency and a very large amount of compute power, it would take far too long to compute everything that we need to.

Our system does not make these computations. Instead, we use the Competition API to make the computations happen. You can see the calls for this in the Patch Evaluation plugin in patch_evaluation.py.

Each individual patch evaluation call isn't necessarily fast. It can take on the order of tens of minutes to confirm that a given patch mitigates a given PoV. It's necessary, therefore, that we scale this as much as we possibly can. So, the scoring pipeline threads, carefully ensuring that we do not overload our host machine. We will send all of the requests as quickly as possible, and then iterate over them to assess their statuses according to scantron.

In order to set the maximum number of threads that your machine will use to execute, you can use the MAX_THREADS environment variable. See scoring/environment.py.

Running Individual Plugins

You can run just the work method of a given plugin if you want to test it alone with data from the database using the included script at scoring/scripts/run_plugin.py. Its typical usage it like

.../scoring/scripts $ poetry run python3 run_plugin.py \
                             -c <challenge task UUID>
                             -t <team UUID>
                             -p <plugin name>

Where Stuff Is

  • The database model can be found in scoring/model/__init__.py
  • The task modules can be found under scoring/plugins/tasks
  • Once you've got a shell in the dev container...
    • Activate the virtual environment with eval $(poetry env activate)
    • You can run the Scorer Runner with python3 scoring/plugins/runner.py
    • Access the database at localhost:5432 with your favorite database tools
    • If you want to destroy all of the data, execute the commands from scoring/scripts/trash_all_data.sql

About

No description, website, or topics provided.

Resources

Readme
MIT license
Activity
Custom properties

Stars

3 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages