Our team is participating in the MSR 2027 Mining Challenge using the SpecMine dataset Specification can be found here: https://2027.msrconf.org/track/msr-2027-mining-challenge
Software developers are increasingly using AI agents to write technical specification files (spec.md). This project aims to build an AI classifier that analyzes specification files and predicts whether AI was used at any point in creating a specific file.
From the FinalProject-CS440 directory (the directory containing this README),
install the package and its dependencies into the Python environment you intend
to use:
cd path/to/FinalProject-CS440
python -m pip install -e ".[dev]"Choose the same Python environment for installation and the notebook kernel.
In VS Code, use Select Kernel in each notebook. To confirm which interpreter
the kernel is using, run import sys; print(sys.executable) in a code cell.
Installing the package in that environment as shown above is recommended;
some notebooks also add the repository's src/ folder to their import path.
If a notebook reports a missing dependency, install the requirements into the
selected environment with
<python-interpreter> -m pip install -r requirements.txt.
Import load_data from the installed specmine package:
python -c 'from specmine.data import load_data; df = load_data(sample_size=5, seed=0); print(df.head() if df is not None else "Dataset load failed")'The default call reads spec_files.parquet from the local cache when present;
otherwise it downloads that parquet from Hugging Face into
data/data/. sample_size limits the returned DataFrame but does not
reduce the first download or the memory needed to read the full metadata
parquet. data_dir=... changes the cache root.
Use curated_only=True to load the curated repositories. The filtered metadata
is saved as curated_spec_files.parquet and reused on later calls, so the full
metadata parquet is not scanned again. Raw Markdown is downloaded to the cache
root's spec_files/ subdirectory and reused; sample_size limits the selected rows/files. With no
sample_size, all curated rows are selected. To load curated metadata without
downloading Markdown, set download_raw_specs=False and pass the metadata to
download_random_spec_sample or download_spec_files:
from specmine.data import download_random_spec_sample, load_data
metadata = load_data(
curated_only=True,
data_dir="data",
download_raw_specs=False,
)
sample = download_random_spec_sample(
metadata,
sample_size=3,
seed=0,
data_dir="data",
)notebooks/data_init.ipynb uses CURATED_ONLY = True by default to select the
curated repositories and samples up to SPEC_SAMPLE_SIZE (3 by default), with
at most one spec per repository. Set it to False to sample from the full
metadata table. Parquets and Markdown files are cache-first: existing files are
reused and missing files are downloaded into data/. The notebook
chooses an unreviewed repository from the sample and displays its Markdown,
exact feature row, and metadata. Repeated spec IDs and repositories are
excluded; failed downloads are replaced with candidates from other
repositories. The first run and each newly selected, uncached spec may require
network access; rerunning the same seeded sample reuses its local files. Change
SPEC_SAMPLE_SIZE or SEED near the top of the notebook. The sample manifest
is saved as data/spec_sample.csv.
Normal load_data and load_feature_data calls also use the local cache first
and download only a missing required parquet from Hugging Face. For explicitly
strict offline reads, use local_data_only=True:
from specmine.data import load_data, load_feature_data
metadata = load_data(local_data_only=True)
features = load_feature_data(local_data_only=True)Local-only loading looks under this repository's data/data/; pass
data_dir=... to use a different cache root. If a parquet file is missing,
local-only mode raises FileNotFoundError and never falls back to the network.
curated_only=True is cache-first rather than a strict no-network option: it
may fetch missing curated metadata on its first run, then reuse the local
subset. local_data_only=True cannot be combined with curated_only=True.
Use notebooks/data_init.ipynb for cache initialization, sampling, and review.
Use notebooks/show_specs.ipynb to look up IDs and inspect all repository/spec
properties, matched features, and Markdown. Only missing parquet files and
requested Markdown are downloaded into the data directory. Markdown is shown
as escaped text so remote images and embedded HTML are not rendered. The raw
spec downloader accepts only HTTPS URLs on github.com and
raw.githubusercontent.com and validates every redirect before following it.
Record one reviewed row in data/labeled_data/real_labels.csv using
repo_id,spec_id,label,confidence. repo_id identifies the repository for
leakage prevention; spec_id identifies the exact spec and remains the model
feature join key. Use 1 if AI was used at any point, 0 if it was not, and
leave label and confidence blank if evidence is ambiguous. A confidence score
is a numeric value from 0 to 100 describing certainty in the assigned label.
The notebook rejects repositories shared by real_labels.csv and
fake_labels.csv, and excludes repositories already in either file from
selection. Fake-label spec IDs are resolved to repository IDs through the
metadata parquet. The displayed feature table is content-based; commit history
is not used for labeling or by the current training/evaluation workflow.
data/labeled_data/fake_labels.csv is a test fixture that can be used by the training
and results notebooks. It includes both spec_id and the corresponding repo_id for
repository-level leakage checks. Its deterministic confidence values are test
data, not verified assessments. Do not copy fake rows into the real-label file
or interpret fake confidence values as reviewed assessments. Use
notebooks/predict.ipynb to train and evaluate a selected baseline using
matched, nonblank labels, refit on all available labeled specs, and predict a
random sample of 100 specs not listed in the labels CSV. Specs without the
selected model's required features or Markdown are replaced from the unlabeled
pool; if the full requested sample is not available, the notebook stops rather
than saving an undersized result. The sample size and model are configurable.
Each run publishes predictions to a unique CSV filename after a complete
temporary-file write, so concurrent runs do not overwrite one another.
Open notebooks from the repository and run their code cells from top to
bottom. All notebooks use the same cache-first loaders. An internet connection
is needed only when a required parquet or selected Markdown file is missing;
cached files are reused from data.
| Notebook | Purpose and how to use it |
|---|---|
notebooks/data_init.ipynb |
Reuses or fills the cache, samples up to three specs from distinct repositories, writes data/spec_sample.csv, and displays one unreviewed repository with its Markdown, properties, and features. Set CURATED_ONLY, SPEC_SAMPLE_SIZE, and SEED near the top. |
notebooks/show_specs.ipynb |
Looks up all distinct specs for one or more REPO_IDS, or specific SPEC_IDS (list or comma-separated string). It reuses cached files and downloads only missing metadata, features, or requested Markdown. |
notebooks/predict.ipynb |
Trains and evaluates baselines, refits a selected model, then predicts an unlabeled sample. Set LABELS_CSV, MODEL_NAME, TEXT_FEATURE, PREDICTION_SAMPLE_SIZE, and LOCAL_DATA_ONLY near the top. It saves models under data/models/ and a uniquely named prediction CSV under notebooks/. |
notebooks/results.ipynb |
Retrains and compares all five models on a shared, seeded repository-disjoint holdout. Displays figures and publishes PNG and PDF files in a unique run directory under notebooks/figures/ in its final cell. Configure LABELS_CSV, SEED, TEXT_FEATURE, or LOCAL_DATA_ONLY near the top. The default AI-assigned labels are not verified authorship. |
For labeling, ambiguous evidence should be recorded with blank
label and confidence fields. Nonblank labels must be 0 or 1, and their
confidence must be from 0 to 100. Do not copy rows from fake_labels.csv into
real_labels.csv; the fake labels and their confidence values are test data.
The data_init.ipynb workflow selects its review repository from the downloaded
sample and reports an error if no eligible downloaded spec is available; increase
SPEC_SAMPLE_SIZE if the sample contains no repository that is still available
to review.
The optional labeler analyzes the Markdown files already cached in
data/spec_files/; it also recognizes files in the old cache root. It does
not download additional specs or metadata. Install the
labeling extra:
python -m pip install -e ".[labeling]"Set GEMINI_API_KEY in your environment using your usual secure secret
mechanism; do not put the key in source files, shell history, or commits. In
bash or zsh, you can enter it without echoing it to the terminal:
printf "Gemini API key: "
IFS= read -rs GEMINI_API_KEY
printf '\n'
export GEMINI_API_KEYThe labeler sends each selected file's complete contents to Gemini, so only run it on files you are permitted to share and that contain no sensitive data or source code. Each request can consume API quota or incur charges.
Run the following command to analyze all cached Markdown files:
python src/specmine/ai-labeler.pyTo try only particular files first, repeat --spec-id with their 16-character
SpecMine IDs:
python src/specmine/ai-labeler.py --spec-id 0123456789abcdef \
--spec-id fedcba9876543210Results are appended to data/ai_analysis_results.csv with columns
repo_id,spec_id,label,confidence,reasoning. Existing valid spec results are
reused on subsequent runs, including when multiple specs belong to the same
repository. Errors are reported and produce a nonzero exit status; unsuccessful
specs can be retried by running the command again. Formula-like reasoning is
escaped for spreadsheet safety. The default model is gemini-3.5-flash-lite;
use --model to select another available model and --delay-seconds to adjust
the four-second delay between requests. The --data-dir and --output options
override the default cache and CSV paths.
These are model-generated estimates, not verified evidence of authorship. Confidence is the model's stated confidence in its binary label, not a calibrated probability. Review the results before using them as training data.
FinalProject-CS440/
├── .gitignore # Excludes local data, models, and generated files
├── README.md # Installation, workflows, and project guide
├── data/ # Shared local cache and outputs
│ ├── data/ # Cached dataset parquets
│ ├── spec_files/ # Downloaded Markdown specs
│ ├── labeled_data/
│ │ ├── fake_labels.csv # Test labels; not verified authorship data
│ │ └── real_labels.csv # Current AI-assigned labels
│ └── models/ # Saved scikit-learn model pipelines
├── notebooks/
│ ├── data_init.ipynb # Combined sample and review workflow
│ ├── show_specs.ipynb # Inspect specs by repo or spec ID
│ ├── predict.ipynb # Train, evaluate, and predict
│ ├── results.ipynb # Evaluation summaries and plots
│ └── figures/ # Generated result plots
├── src/
│ └── specmine/
| ├── __init__.py
│ ├── ai-labeler.py # Optional Gemini-assisted labeler
│ ├── data.py # Dataset loading, caching, and downloads
│ ├── diagnostics.py
│ ├── evaluate.py # Model evaluation and prediction outputs
│ ├── features.py # Feature preparation
│ ├── labeling.py # Leakage-safe review selection and matching
│ ├── labels.py # Label CSV parsing and validation
│ ├── model_cache.py
│ ├── outputs.py
│ ├── train.py # Training and repository-disjoint splitting
│ └── evaluate.py
├── tests/ # Offline unit tests and opt-in network tests
├── DATA_DICTIONARY.md # SpecMine dataset and feature reference
├── schema.sql # Database schema reference
├── pyproject.toml # Package metadata and development dependencies
├── requirements.txt # Runtime dependency list
├── LICENSE
└── LICENSE-DATA
The parquet cache, curated metadata subset, downloaded Markdown, sample
manifest, and generated plots are local workflow data. Cache-first loaders
reuse existing files and download only missing required files. Downloaded
Markdown filenames use repo_name-repo_id-spec_id.md (with / in the
repository name replaced by _) so the repository identity is retained in the
filename. Remove a cached file when you intentionally want to fetch it again;
other notebooks may create prediction CSVs and model files when run.
The cache and saved models were previously stored in notebooks/data/ and
models/. After updating, copy their contents into the new locations without
overwriting existing files:
python -m specmine.migrate_cache notebooks/data data # if notebooks/data exists
python -m specmine.migrate_cache models data/models # if models existsSkip either command if its source does not exist. Each file is copied to a
temporary file and published only after the copy completes. Identical destination
files are retained; a differing file, including a partial copy from an earlier
migration, causes an error. Compare and move the conflicting destination aside
before retrying. Check the copied files before removing the old directories.
Existing sample manifests may contain old
absolute paths; rerun data_init.ipynb to generate a fresh manifest.
The label CSVs now live in data/labeled_data/; preserve and reconcile any
local edits to the old examples/ files before switching.
New Markdown downloads go into data/spec_files/. For existing caches, the
loaders and labeler also read Markdown from the cache root, so those files can
be moved to data/spec_files/ without downloading them again. Avoid creating
duplicate copies: the labeler reports duplicate spec IDs instead of choosing
between them.
The training module provides five baselines:
| Model name | Inputs and estimator |
|---|---|
logistic_regression |
All numeric spec-content features, with Logistic Regression. |
random_forest |
All numeric spec-content features, with Random Forest. |
naive_bayes |
Full Markdown represented by character TF-IDF, with Multinomial Naive Bayes; no numeric features. |
uniform_guess |
Independent, seeded 50/50 class guesses; it ignores feature values. |
markdown_plus_feature |
Full Markdown represented by character TF-IDF plus exactly one selected numeric feature, with Logistic Regression. |
Both text models use case-sensitive character 3- to 5-gram TF-IDF with at most
50,000 vocabulary terms. The vectorizer and numeric preprocessing are fitted
only on the training partition. markdown_plus_feature defaults to n_words;
choose another numeric column with --text-feature, the text_feature
argument to train_models, or TEXT_FEATURE in the notebooks. The existing
numeric baselines remain the CLI and Python API defaults. Commit history is
not used by the current training/evaluation workflow.
Training requires --labels-csv. Use 1 when AI was used for any part of
creating a spec, 0 only when no AI was used, and a blank label for ambiguous
cases to exclude. Repositories are kept separate between training and holdout
data.
python -m specmine.train --labels-csv labels.csv --seed 42 \
--local-data-only \
--model-dir data/models/baselinesFor a comparison including the new models:
python -m specmine.train --labels-csv labels.csv --seed 42 \
--models logistic_regression random_forest naive_bayes uniform_guess markdown_plus_feature \
--text-feature n_words --model-dir data/models/baselinesSelecting a text model loads Markdown for the matched labeled specs, reusing
the cache and downloading missing files from approved GitHub URLs. Unavailable
or unreadable Markdown is reported by spec ID and excluded from all
selected models before their shared repository-disjoint split. Set
--local-data-only to prevent network access; the same exclusions apply to
uncached files, and trainable models require a split with both classes.
Numeric-only and uniform-only selections do not download Markdown. When
calling the Python API directly, request
prepare_training_data(..., include_markdown=True) before training a text
model.
The training CSV requires spec_id,label,confidence: spec_id matches
file_url_sha16, label is 0, 1, or blank, and confidence is a numeric
score from 0 through 100. The manual-label CSV may additionally include
repo_id for repository-level exclusion checks; training continues to join
labels to features by spec_id. Every nonblank label must have a confidence
score.
Blank labels are excluded from training. The training code passes
confidence / 100 as a sample weight to the trainable estimators, so a score
of 0 contributes no weight and 100 contributes full weight. The uniform
guesser ignores those weights. Holdout metrics remain unweighted. Omit
--sample-size to use all matched labeled examples; if supplied, it caps the
matched rows after ID matching. Training reads the full metadata parquet to
ensure labeled IDs are not lost to random sampling, then loads features only
for labeled specs. Omit --model-dir to skip saving models.
The results notebook separately visualizes the labeler's reported confidence
(0–100) and each model's held-out probability of label 1 (0–1). Its
prediction_table includes the probability assigned to the predicted class,
alongside the original label confidence. Neither score is necessarily
calibrated. The uniform guesser reports a probability of 0.5 for each class
and makes seeded random class guesses; its predictions do not use the
probability threshold.
The notebook also shows class and aggregate repository composition, test
metrics with repository-cluster bootstrap intervals, confusion matrices,
ROC/precision–recall curves, and model-specific training signals. Numeric
coefficients and forest importances and the text models' character n-grams
describe associations with the supplied labels, not causal evidence of AI use.
Calibration and some curves are omitted when the held-out sample lacks
adequate class support. Missing Markdown is excluded from all five models;
the included sample may therefore differ from the original label file.
Figures are staged during execution and published together as 300-dpi PNG and
vector PDF files in a unique notebooks/figures/run-<id>/ directory when the
final cell succeeds. Concurrent runs do not overwrite one another. Interrupted
.staging-* directories are unpublished and may be removed after the run has
stopped. Older figures in the parent directory are not part of the new run.
A separate incremental SGD logistic diagnostic plots training log loss by
epoch; it is not the fitted baseline. An exploratory three-fold,
repository-grouped cross-validation plot compares numeric logistic regression
C values using training data only. It does not tune the evaluated holdout
models, and is omitted if the folds lack sufficient class support.
The training notebook defaults to data/labeled_data/real_labels.csv; its current
AI-assigned labels and confidence values are not verified ground truth.
Choose data/labeled_data/fake_labels.csv in LABELS_CSV for testing only. All IDs appearing
in the selected CSV (including blank-label rows) are excluded from the
extrapolation sample. Repeated rows for the same ID and label collapse to one
row; conflicting labels for an ID are treated as blank/ambiguous and reported
with a warning. Prediction CSVs use a unique filename based on the
notebooks/unlabeled_predictions.csv output name by default.
The released SpecMine schema does not include AI-authorship labels. Supply a
CSV of manually labeled examples with spec_id values matching
file_url_sha16, a binary label column (1 if AI was used at any point in
creating the spec, 0 if no AI was used), and a confidence score from 0 to
100. Examples without a label match are excluded and counted. Leave ambiguous
examples' label and confidence blank; training excludes them.
spec_id,label,confidence
0123456789abcdef,1,90
fedcba9876543210,0,75Save a fitted sklearn pipeline with joblib, then evaluate a split:
python -m specmine.evaluate --model-path data/models/specmine.joblib \
--labels-csv labels.csv --split test --local-data-only \
--output-json metrics.json --output-csv predictions.csvWithout --labels-csv, the command emits predictions only and does not compute
supervised metrics. For a text model without labels, provide --sample-size
to bound the number of requested Markdown files. Missing Markdown is reported
and excluded, and the JSON report includes n_unavailable_markdown.
--local-data-only prevents Markdown downloads as well as parquet downloads.
--split accepts train, test, or all; use
--make-val --split val to deterministically carve a validation split out of
train. CSV metadata is escaped when it begins with a spreadsheet formula
prefix. Curated downloads use --data-dir (default data).
Run the local tests with:
pytest -vNetwork smoke tests are skipped by default so the local test command works offline. To run them explicitly with an internet connection:
SPECMINE_RUN_NETWORK_TESTS=1 pytest -v -m networkThis project participates in the MSR 2027 Mining Challenge and utilizes the spec_files schema provided by the SpecMine research corpus (2026).
If you use this work or pipeline, please cite both the foundational paper and the Hugging Face data repository (see the full BibTeX entries in the referenced documentation).
- Reference: SpecMine: A Large-Scale Corpus of Spec-Driven Development Artifacts (MSR '27, IEEE) arXiv:2608.25202 [cs.SE].
- Reference: SpecMine Dataset Parquet Mirror (Hugging Face datasets/ShyAgarwal/specmine) Dataset.