SLM4IE develops small language models (SLMs) for zero-shot information extraction across European languages, with emphasis on Slovenian. The project targets three limitations of current LLMs:

• Compute cost: LLMs require infrastructure beyond reach of smaller organizations for local deployment
• Low-resource gaps: Limited training data for sensitive domains and underrepresented languages
• Output inconsistency: Unreliable structured extraction from generative models

We build computationally efficient models optimized for commodity hardware, create multilingual benchmark datasets for sensitive domains, and evaluate against existing SLMs and LLMs. All artifacts (models, datasets, code) will be released publicly where possible.

Requirements

• Python ≥ 3.13 (declared in pyproject.toml and .python-version)
• uv — recommended package and environment manager
• Git — for cloning the repository and CLARIN.SI dataset access
• HuggingFace account — required for gated datasets (e.g., FineWeb-2)
• Disk space — pretraining corpora total tens of GB; plan accordingly
• GPU (optional) — required for tokenizer/model training; CPU sufficient for data preparation

Setup

Install dependencies

Clone the repository and create the virtual environment via uv:

git clone https://github.com/eriknovak/SLM4IE.git
cd SLM4IE
uv sync

This creates .venv/ and installs both runtime and dev dependencies pinned in uv.lock. Activate the environment for ad-hoc commands:

source .venv/bin/activate

Or prefix individual commands with uv run to skip activation.

Enable the git hooks

The repository ships a pre-commit hook under .githooks/ that blocks commits containing presigned-URL credentials (X-Amz-Signature / X-Amz-Credential). Activate it once per clone:

git config core.hooksPath .githooks

Secrets and ephemeral values (such as presigned download URLs) belong in a gitignored configs/**/*.local.yaml overlay, which load_config deep-merges over the matching base config — never in the committed YAML.

HuggingFace authentication

Some datasets (e.g., FineWeb-2, gated corpora) require a HuggingFace access token. Authenticate once via the new unified hf CLI (shipped with huggingface_hub ≥ 0.34, which replaces the deprecated huggingface-cli):

hf auth login

Paste a token from huggingface.co/settings/tokens when prompted. The token is stored under ~/.cache/huggingface/ and picked up automatically by huggingface_hub and datasets — no HF_TOKEN environment variable or .env file needed.

For non-interactive use (e.g., CI, SLURM), pass the token directly:

hf auth login --token "$HF_TOKEN"

To verify:

hf auth whoami

Project Structure

SLM4IE/
├── configs/                # YAML configuration files
│   ├── data/                 # download, extract, pretrain, tasks, tokenization, synthetic
│   ├── models/               # model architecture configs
│   ├── tokenizers/           # tokenizer training configs
│   ├── training/             # pretrain.yaml, finetune_ner.yaml
│   └── experiments/          # end-to-end experiment configs
├── slm4ie/                 # Library source
│   ├── data/                 # download, extract, processing, schema, synthetic
│   ├── models/               # model components and registry
│   ├── tokenizers/           # tokenizer training and analysis
│   ├── training/             # trainer, callbacks, evaluation
│   └── utils/                # config, I/O, MLflow helpers
├── scripts/                # CLI entry points (thin wrappers around slm4ie/)
│   ├── data/                 # download.py, extract.py, to_pretrain.py, to_tokenization.py,
│   │                         #   to_spans.py, to_sentiment.py, to_superglue.py, generate_synthetic.py
│   ├── tokenizers/           # train.py, analyze.py, export.py
│   ├── train.py              # model pretraining/fine-tuning
│   └── evaluate.py           # benchmark evaluation
├── slurm/                  # SLURM batch scripts for HPC training
├── notebooks/              # exploratory Jupyter notebooks
├── tests/                  # pytest test suite
├── docs/                   # documentation and assets
├── pyproject.toml          # project metadata and dependencies
└── uv.lock                 # locked dependency versions

Running Scripts

All scripts are CLI wrappers around slm4ie/ modules and read from YAML configs in configs/. Run them via uv run (recommended) or after activating .venv/.

Data pipeline

The data pipeline materializes a five-tier on-disk tree under /vault/data/SLM4IE/:

raw/<key>/...                                  # original downloads (download.py)
extracted/                                     # canonical unified form (extract.py)
  <key>.jsonl
  <key>.annotations.jsonl.gz
pretrain/                                      # corpus-wide curate output (to_pretrain.py)
  00_convert/<key>/*.jsonl.gz                    # datatrove `Document` shape
  01_language/<key>/*.jsonl.gz
  02_spam/<key>/*.jsonl.gz                       # adult/SEO-spam removal
  03_quality/<key>/*.jsonl.gz
  04_repetition/<key>/*.jsonl.gz
  05_1_dedup/<key>/*.jsonl.gz                    # exact dedup
  05_2_dedup/<key>/*.jsonl.gz                    # sentence dedup — final corpus
  06_statistics/                                 # corpus-wide stats
tasks/<task>/<dataset>/{train,val,test}.jsonl.gz  # SFT + eval (to_spans/sentiment/superglue)
tokenization/<dataset>.jsonl.gz                # tokenizer-quality data (to_tokenization.py)

Five YAML configs drive the seven data scripts:

Config	Script(s)	Purpose
configs/data/download.yaml	download.py	Raw corpus + benchmark download catalog.
configs/data/extract.yaml	extract.py	Sources to normalize into extracted/.
configs/data/pretrain.yaml	to_pretrain.py	Eight-stage curate pipeline (stage 0 = datatrove convert; stages 1–7 = filter/spam/dedup/stats).
configs/data/tokenization.yaml	to_tokenization.py	Tokenizer-quality datasets (lexicon-derived).
configs/data/tasks.yaml	to_spans.py, to_sentiment.py, to_superglue.py	Registry of <task>/<dataset> entries with roles, sources, splits, labels.

End-to-end command flow:

uv run python scripts/data/download.py       --config configs/data/download.yaml      --all
uv run python scripts/data/extract.py        --config configs/data/extract.yaml       --all
uv run python scripts/data/to_pretrain.py    --pretrain-config configs/data/pretrain.yaml --all
uv run python scripts/data/to_tokenization.py --all
uv run python scripts/data/to_spans.py       --all       # reads tasks.yaml automatically
uv run python scripts/data/to_sentiment.py   --all
uv run python scripts/data/to_superglue.py   --all

Parallelism and per-dataset logs

download.py, extract.py, to_tokenization.py, to_spans.py, to_sentiment.py, and to_superglue.py all accept a --max-workers flag and process multiple datasets concurrently:

• --max-workers 0 (default) — auto: min(cpu_count // 2, n_datasets) for CPU-bound steps, capped at 4 for download.py to stay polite to remote servers.
• --max-workers 1 — serial path; tracebacks are unwrapped, console keeps today's verbose output, and the inner per-dataset progress bar is shown.
• --max-workers N — that many workers, capped at the number of selected datasets.

Per-dataset logs are always written to logs/<script>/<UTC-timestamp>/<key>.log, regardless of worker count. The log directory is printed to stderr at startup. In parallel mode (> 1) the console only prints a periodic summary line (running=R done=D skipped=S failed=F waiting=W) every 30 seconds — the per-dataset INFO lines and inner tqdm bars are routed to the log files instead, so concurrent workers don't garble each other on stderr.

to_pretrain.py accepts the same --max-workers flag but is whole-pipeline, not per-dataset — every parallel datatrove executor inside one stage uses the same worker count, so the per-dataset log routing above does not apply. Its default is --max-workers 1 (serial) so a casual --all invocation does not silently saturate the box; --max-workers 0 falls back to cpu_count // 2 and --tasks is accepted as a back-compat alias.

Download

Download raw corpora declared in configs/data/download.yaml:

Selection is explicit: pass one or more dataset keys as positional arguments, or pass --all. Bare invocation errors out.

# Download every enabled dataset in the config
uv run python scripts/data/download.py --all

# Download specific datasets (positional, mutually exclusive with --all)
uv run python scripts/data/download.py fineweb2 cc100

# Force re-download with custom output directory
uv run python scripts/data/download.py --all --output-dir /path/to/data --force

# Download only non-pretraining datasets (`role: benchmark` or `role: lexicon`)
uv run python scripts/data/download.py --all --only-benchmarks

# Download only pretraining corpora (`role: pretrain`, the default role)
uv run python scripts/data/download.py --all --exclude-benchmarks

# Use a different YAML in configs/data/ — `--config-name benchmarks` reads
# configs/data/benchmarks.yaml instead of the default `download.yaml`.
uv run python scripts/data/download.py --all --config-name benchmarks

# Download four datasets in parallel (thread pool; default cap is 4)
uv run python scripts/data/download.py fineweb2 cc100 mc4 hplt --max-workers 4

Extract

Extract and convert raw downloads to unified JSONL using configs/data/extract.yaml. Selection is explicit: pass dataset keys as positional arguments, or pass --all.

# Extract every dataset declared in extract.yaml
uv run python scripts/data/extract.py --all

# Extract specific datasets (positional, mutually exclusive with --all)
uv run python scripts/data/extract.py macocu_sl

# Re-extract a dataset whose output already exists
uv run python scripts/data/extract.py macocu_sl --force

# Extract several datasets in parallel (process pool)
uv run python scripts/data/extract.py macocu_sl classla_web_sl kzb --max-workers 3

# Use a different YAML in configs/data/ (without the .yaml suffix)
uv run python scripts/data/extract.py --all --config-name extract_dev

# Override the configured input/output directories from the CLI
uv run python scripts/data/extract.py --all \
    --input-dir /vault/data/SLM4IE/raw \
    --output-dir /vault/data/SLM4IE/extracted

For annotated corpora (CoNLL-U, TEI with <w>, CLASSLA-web JSONL, COLESLAW), extraction writes two files per dataset under extracted/:

• <key>.jsonl — text + source / domain / doc_id / metadata, consumed both by to_pretrain.py's stage 0 (which lifts it into datatrove's Document shape) and by the task converters.
• <key>.annotations.jsonl.gz — gzipped per-document annotations as parallel arrays (forms, lemmas, upos, feats, sentences, plus spans when present), kept separate to avoid loading them during text-only training.

The downstream task converters (to_spans, to_sentiment, to_superglue) join these two files on the fly via slm4ie.data.io_utils.iter_joined_records, so no intermediate merged file is materialized.

Pretraining corpus (to_pretrain.py)

to_pretrain.py builds the final pretraining corpus as a sequence of eight independent, sentinel-skippable stages on top of datatrove. Stage 0 lifts extracted/*.jsonl into datatrove's Document shape; stages 1–7 cover language filtering, adult/SEO-spam removal, Gopher within-document quality and repetition heuristics, cross-corpus exact and sentence deduplication, and corpus statistics. There is no separate datatrove-conversion step — the old to_datatrove.py lives inside stage 0 now. Each stage writes a durable on-disk artifact and a .complete sentinel under output_dir; on rerun, a stage whose config slice hash is unchanged is skipped, and editing one section of configs/data/pretrain.yaml cascade-invalidates that stage plus every downstream stage. input_dir is the folder of <key>.jsonl files from extract.py; output_dir is the pretrain-owned tree. The dataset key list still comes from configs/data/extract.yaml.

Install the extra

uv sync --extra curate

The curate extra pulls in datatrove, lingua-language-detector, spacy (Slovenian word/sentence tokenization), classla (Slovenian lemmatizer for the keyword TF-IDF pass), orjson, tokenizers, xxhash, nltk, and a few smaller helpers. We deliberately skip datatrove's own processing and multilingual extras because those transitively pull in fasttext-numpy2-wheel, which has no Python 3.13 wheel and would require a C++17 toolchain to build.

Canonical command

uv run python scripts/data/to_pretrain.py --all

This iterates all eight stages in order, skipping any whose sentinel hash matches the current config. The final corpus lands at <output_dir>/05_2_dedup/<dataset>/<rank>.jsonl.gz; statistics at <output_dir>/06_statistics/.

Eight user-facing stages

Each stage reads its predecessor's output and writes a numbered folder. The two dedup sub-stages are independent: 05_1_dedup cleans whole-document duplicates across the corpus; 05_2_dedup runs sentence-level dedup over that result.

CLI name	Folder	Operates on	What it does
convert	00_convert/	per-doc	lift extracted/<key>.jsonl into datatrove Document shards (text / id / metadata); carries dataset and domain for source-weighted sampling
language	01_language/	per-doc	lingua-py language detection (tag or filter)
spam	02_spam/	per-doc	adult/SEO-spam removal via per-language lexicons + URL/domain blocklist + optional model hook
quality	03_quality/	per-doc	Gopher within-document quality heuristics (length, word lengths, symbol/bullet/ellipsis ratios, stopword floor)
repetition	04_repetition/	per-doc	Gopher within-document repetition heuristics (duplicate paragraphs/lines, top-n-gram saturation, dup-n-gram fractions)
exact_dedup	05_1_dedup/	corpus-wide	whole-document exact dedup (xxhash64 of doc.text)
sentence_dedup	05_2_dedup/	corpus-wide	N-sentence sliding-window dedup (final corpus)
stats	06_statistics/	corpus-wide	word/n-gram tables and (optional) classla TF-IDF keywords (single-process)

Each stage's sentinel hash covers its own top-level pretrain.yaml section. The quality and stats hashes additionally fold in the contents of the stopword file, the spam hash folds in the spam lexicon and domain-list contents, and every stage's hash folds in the sorted list of dataset keys this run will process — so editing stopwords/sl.txt, editing a spam/<code>/*.txt list, switching between --all and a positional subset, or adding a dataset to extract.yaml all correctly trigger rebuilds.

Note — refreshed inputs auto-rebuild. convert is the only stage that reads the extracted/ tier, so it also tracks a size + modification-time fingerprint of each source <key>.jsonl (and its .annotations.jsonl.gz sidecar when include_annotations is on). Re-extracting a dataset — e.g. folding new weekly windows into a living corpus with extract.py <key> --force — changes that fingerprint, which marks convert stale for that key and cascades through every downstream scoped and corpus stage. So a plain to_pretrain.py --all after re-extraction re-folds the updated data with no --force needed. The fingerprint is size and time only, never the file contents (hashing the whole multi-hundred-GB tier on every run would be prohibitive): an in-place edit that somehow preserved both byte size and mtime would go undetected, and a content-preserving copy/touch that bumps mtime triggers a harmless rebuild. Sentinels written before fingerprints existed are grandfathered — a key is rebuilt only if its source file is newer than the recorded completion time — so this upgrade does not force a one-off reconvert of the whole corpus.

Note — per-dataset overrides. An optional top-level overrides: block in pretrain.yaml lets a single dataset patch any scoped stage's config without forking the file. It is keyed by dataset, then by stage, and deep-merges over the global section (unspecified knobs inherit the default):

overrides:
  slovenian_news:
    quality:
      max_ellipsis_lines_ratio: 0.9   # news prose uses "…" mid-article

Only scoped stages (convert, language, spam, quality, repetition) are overridable — naming a corpus stage (exact_dedup/sentence_dedup/stats) or a global key (input_dir/output_dir/stopwords), an unknown dataset, or an unknown knob is a hard error at load. Datasets are bucketed by their effective config: those sharing one run together in a single executor, so only overridden datasets pay isolation cost. Each dataset's sentinel hashes its effective (merged) config, so adding/editing an override re-runs only that dataset's stage plus its downstream and the corpus dedup/stats; a dataset with no override is byte-identical to before and never re-runs spuriously. (repetition exposes no knobs today, so it is effectively non-overridable until some are surfaced.)

Internally each dedup stage chains three datatrove executors via depends=: signature → find (single-worker reducer over signatures) → filter + write. The sig/find scratch lives at <output_dir>/_dedup_state/ and is purged when the stage's sentinel lands. The stats stage is single-process because CorpusStats keeps global counters on its instance. The sentence-dedup blocks use Languages.slovenian so datatrove dispatches its bundled Slovenian SpaCyTokenizer for sentence boundaries.

Output layout

<input_dir>/                                upstream input (extract.py)
├── <key>.jsonl
└── <key>.annotations.jsonl.gz

<output_dir>/                               to_pretrain.py owns this entire tree
├── 00_convert/
│   ├── <key>/<rank>.jsonl.gz               ← datatrove `Document` shards
│   └── .complete                           sentinel: stage hash + counts
├── 01_language/
│   ├── <key>/<rank>.jsonl.gz               ← post-language-filter shards
│   └── .complete
├── 02_spam/
│   ├── <key>/<rank>.jsonl.gz               ← post-spam-filter shards
│   └── .complete
├── 03_quality/
│   ├── <key>/<rank>.jsonl.gz
│   └── .complete
├── 04_repetition/
│   ├── <key>/<rank>.jsonl.gz
│   └── .complete
├── 05_1_dedup/
│   ├── <key>/<rank>.jsonl.gz               ← post-exact-dedup shards
│   └── .complete
├── 05_2_dedup/
│   ├── <key>/<rank>.jsonl.gz               ← final pretraining corpus
│   └── .complete
├── 06_statistics/
│   ├── aggregate.json                      corpus-wide totals + tables
│   ├── per_dataset/<key>.json              per-dataset doc/word breakdowns
│   └── .complete
├── _dedup_state/                           sig/find scratch (auto-purged
│                                           when each dedup sentinel lands)
└── _logs/<stage>/                          datatrove per-executor logs

Useful invocations

# Run all eight stages, skipping any whose config slice hash is unchanged.
uv run python scripts/data/to_pretrain.py --all

# Run only one stage. If its hash diverges from the recorded sentinel,
# downstream sentinels are dropped so the next --all picks them up.
uv run python scripts/data/to_pretrain.py --all --stage quality

# Force-rebuild a stage and every downstream stage. Removes their data
# folders AND sentinels; --force without --stage clears <output_dir>.
uv run python scripts/data/to_pretrain.py --all --force --stage quality

# Single dataset, or a subset. The dataset key list folds into every
# stage's hash, so a subset rerun will not silently reuse a previous
# full-corpus output. (Switching between subsets / --all triggers rebuilds.)
uv run python scripts/data/to_pretrain.py kzb solar

# Parallelism. Default is 1 (serial). 0 = cpu_count // 2. --tasks is an alias.
uv run python scripts/data/to_pretrain.py --all --max-workers 8
uv run python scripts/data/to_pretrain.py --all --max-workers 0

# Override pretrain.yaml paths from the CLI.
uv run python scripts/data/to_pretrain.py --all \
    --input-dir /tmp/in --output-dir /tmp/out

Configuration

configs/data/pretrain.yaml has one top-level section per stage (convert:, language:, quality:, repetition:, exact_dedup:, sentence_dedup:, stats:) plus shared input_dir, output_dir, and a stopwords: path used by both quality and stats. Each section is the exclusive input to that stage's sentinel hash slice, so edits propagate as far downstream as needed and no further. Defaults match the Gopher paper for the heuristic filters, 64-bit xxhash for exact dedup, 3-sentence windows for sentence dedup, and top-5000 word / bigram / trigram + top-200 TF-IDF keyword tables for stats. To skip the (slow) classla-lemmatized keyword pass, set stats.compute_keywords: false in the YAML — there is no longer a --no-keywords CLI flag.

The first run of the keyword stage downloads the Slovenian classla model (~200 MB) under ~/.classla_resources/.

Tokenizer-quality data (to_tokenization.py)

to_tokenization.py materializes lexicon-derived datasets used only for tokenizer / morphology evaluation — they never enter the pretraining corpus. This covers Sloleks 3.1 (Slovenian inflectional lexicon → inflectional gold) and the Sloleks 2.0 word relations (CLARIN 11356/1986 → derivational silver gold; ~66k lemmas decomposed with underscores, ~5k linguist-verified). Both are CC BY-SA 4.0. Configuration lives in configs/data/tokenization.yaml; the script also reads configs/data/download.yaml to resolve per-dataset raw subdirectories.

# Convert every dataset declared in tokenization.yaml
uv run python scripts/data/to_tokenization.py --all

# Convert one dataset, overwriting if the output already exists
uv run python scripts/data/to_tokenization.py sloleks --force

# Run in parallel
uv run python scripts/data/to_tokenization.py --all --max-workers 4

Output goes to tokenization/<dataset>.jsonl.gz. Existing outputs are skipped unless --force is passed.

Task datasets (to_spans, to_sentiment, to_superglue)

The three task converters all read configs/data/tasks.yaml, a flat registry keyed <task>/<dataset>. They write to tasks/<task>/<dataset>/<split>.jsonl.gz using a task-family schema (TypedDicts in slm4ie/data/schema.py). Each entry declares:

• role — finetune_and_eval or held_out; the registry, not directory placement, enforces train/test isolation across families.
• source — {kind: extracted, keys: […]} for document-shaped sources joined via extracted/, or {kind: raw, keys: […]} for task-native bundles (SuperGLUE-SL) read straight from raw/.
• splits, labels, suite, language, license.

Adding a new task dataset is a one-entry edit to tasks.yaml; the appropriate converter (defaulted by the converters: map at the top of the file) will pick it up.

# NER (GLiNER-style output)
uv run python scripts/data/to_spans.py --all                        # every ner/* entry
uv run python scripts/data/to_spans.py ner/ssj500k ner/suk          # subset

# Sentiment
uv run python scripts/data/to_sentiment.py --all                    # every sentiment/* entry
uv run python scripts/data/to_sentiment.py sentiment/sentinews

# SuperGLUE-SL families (nli, qa, coref, wsd, commonsense)
uv run python scripts/data/to_superglue.py --all                    # HumanT variant by default
uv run python scripts/data/to_superglue.py --variant googlemt --all
uv run python scripts/data/to_superglue.py nli/cb nli/rte

All three skip existing outputs unless --force is passed and accept --max-workers for per-entry parallelism. The legacy --schema {gliner|conll|generic} flag on to_spans.py is gone — only the GLiNER-compatible schema is produced now.

Synthetic data

uv run python scripts/data/generate_synthetic.py # synthetic IE data via LLM APIs

Tokenizer

Trains six tokenizers, each faithful to its original work: byte-level BPE (GPT-2/RoBERTa style), character-level charBPE (the clean MorphBPE ablation), BERT-style WordPiece, SentencePiece Unigram, and the from-scratch MorphBPE (morpheme-constrained training, standard inference — arXiv 2502.00894) and MorphPiece (byte-level BPE + a Sloleks-derived MorphTable — arXiv 2307.07262). Each runs at three vocab sizes (16k/32k/64k) and is scored with six metrics to determine which segments Slovenian most morphologically. Driven by configs/tokenizers/tokenizers.yaml. Requires the tokenize extra: uv sync --extra tokenize.

The stage consumes the deduplicated corpus (pretrain/05_2_dedup/) for training and two Sloleks-derived golds produced by to_tokenization.py: the inflectional gold (tokenization/sloleks.jsonl.gz) and the derivational gold (tokenization/sloleks_relations.jsonl.gz). The morph metrics report against the inflectional gold (with bootstrap CIs); the derivational gold adds point-estimate *_deriv columns and also enriches the morphological backends' morpheme table. Run both converters first.

uv run python scripts/data/to_tokenization.py sloleks sloleks_relations  # prerequisite: morph golds
uv run python scripts/tokenizers/prepare_sample.py       # materialize the persistent sample + lexicon
uv run python scripts/tokenizers/train.py     --all      # train the 6x3 sweep
uv run python scripts/tokenizers/analyze.py   --all      # 6 metrics + report.md/json
uv run python scripts/tokenizers/export.py    --all      # HuggingFace tokenizer dirs

# Or one tokenizer (optionally one vocab size) instead of --all:
uv run python scripts/tokenizers/train.py --tokenizer bpe                   # all its vocab sizes
uv run python scripts/tokenizers/train.py --tokenizer bpe --vocab-size 16000  # one run

prepare_sample.py materializes the shared, seeded training sample (and the morpheme lexicon) once into tokenizers/corpus_sample.txt.gz, so every train run reuses the identical sample instead of re-drawing it. It is optional — train.py builds the sample on first use — but running it up front keeps the sample persistent and reproducible across reruns (and across a --max-workers change after a crash). Pass --force to rebuild it.

train/analyze/export share a one-or-all selection: --all, or one --tokenizer <name> optionally narrowed by --vocab-size <n> (the two modes are mutually exclusive). train/analyze also take --force and --max-workers. Artifacts land under /vault/data/SLM4IE/tokenizers/<name>-<vocab>/; the comparison report is written to tokenizers/_reports/report.md. The sweep logs to MLflow by default under experiment slm4ie/tokenization/slovenian; the tracking URI is read from MLFLOW_TRACKING_URI (falling back to a local SQLite store), overridable via mlflow.tracking_uri in a tokenizers.local.yaml overlay. Disable with mlflow.enabled: false.

Using a tokenizer downstream. export.py writes a HuggingFace tokenizer directory into each artifact. The five fast tokenizers load with AutoTokenizer.from_pretrained(<dir>) and expose decode and return_offsets_mapping natively; MorphPiece is a custom slow tokenizer loaded with slm4ie.tokenizers.hf_export.load_pretrained(<dir>), exposing encode_with_offsets. Offset mapping (token → source-character span) is the mechanism for aligning encoder predictions back to the original text.

The six metrics: Fertility (tokens/word, ↓), CTC compression (tokens-per-byte ↓ / chars-per-token ↑), Rényi efficiency (↑), MorphScore boundary F1 (↑), Morph-Edit-Distance (raw, ↓), and Morph-Consistency F1 (↑) — the last two following the MorphBPE paper (arXiv 2502.00894). The morph metrics are offset-based and use a Sloleks-derived inflectional silver gold, so read them as relative comparators, not absolute morphological accuracy.

Model training and evaluation

uv run python scripts/train.py     # pretrain or fine-tune from configs/training/*.yaml
uv run python scripts/evaluate.py  # evaluate on benchmarks/*.yaml

SLURM (HPC)

Batch scripts for cluster execution live under slurm/:

sbatch slurm/tokenizer_train.sbatch
sbatch slurm/tokenizer_analyze.sbatch
sbatch slurm/tokenizer_export.sbatch
sbatch slurm/train.sbatch
sbatch slurm/evaluate.sbatch
sbatch slurm/generate.sbatch

Tests

uv run pytest                # full test suite
uv run pytest tests/data     # subset

Pretraining Corpora

Slovenian text corpora used for language model pretraining (configured in configs/data/download.yaml).

CLARIN.SI sources

Dataset	Domain	Description
CLASSLA-web.sl 2.0	web	Annotated Slovenian web corpus from the CLASSLA project.
CLASSLAWiki-sl	wiki	Slovenian Wikipedia with linguistic annotations (CoNLL-U).
MaCoCu-sl 2.0	web	Slovenian web corpus from the MaCoCu project (XML/TEI).
ParlaMint-SI 5.0	parliamentary	Slovenian parliamentary minutes, annotated TEI.
COLESLAW 1.0	legal	Corpus of Slovenian legal texts.
PoVeJMo-VeMo-Med 1.0	medical	Slovenian medical texts from the PoVeJMo project.
OSS 1.0	scientific	2.59B words / 3.26B tokens from 151K scientific texts (monographs, articles, theses) from Slovenian universities (2000–2022).
siParl 4.0	parliamentary	239M words from parliamentary minutes (1990–2022), TEI XML. May overlap with ParlaMint-SI.
Janes-News 1.0	news	14.8M tokens from news article comments (2007–2015). Informal register.
KZB 1.0	scientific	25M words / 33.6M tokens of curated scientific monographs and papers (2000–2023).

HuggingFace sources

Dataset	Domain	Description
FinePDF	web	Slovenian (slv_Latn) PDF-derived text.
FineWeb-2	web	Slovenian (slv_Latn) high-quality web corpus.
mC4	web	Cleaned multilingual Common Crawl, ~5 GB+ for Slovenian.
HPLT 2.0 Cleaned	web	HPLT project web crawl (CommonCrawl + Internet Archive), cleaned tier; Slovenian config slv_Latn (~10.3M rows).

Direct HTTP sources

Dataset	Domain	Description
CC100	web	Monolingual CommonCrawl filtered with fastText (Facebook AI, XLM-R), ~1.4 GB compressed for Slovenian. Fetched directly from statmt.org; the HuggingFace mirror is script-based and no longer supported by datasets.
Legal-mC4	legal	Legal-domain text filtered from mC4, ~32.5K documents / ~107M words for Slovenian. Fetched directly from the HuggingFace LFS endpoint; the repo's loading script is no longer supported by datasets.

Disabled by default

Optional sources requiring extra access (gated datasets, manual login, copyright restrictions): KAS 2.0 (CLARIN academic login), Janes-Forum/Blog, Solar 3.0, CulturaX (HF gated). Not bulk-downloadable: Gigafida 2.x, Metafida 1.0, Trendi.

Benchmarks

Slovenian evaluation datasets used for downstream IE tasks. Benchmarks are declared in configs/data/download.yaml with role: benchmark (tokenizer lexicons such as Sloleks use role: lexicon) and a tasks: list, so they share the download pipeline with pretraining corpora. Use --only-benchmarks to fetch just the non-pretraining datasets.

Dataset	Source	Tasks	Description
SUK 1.1	CLARIN.SI	POS, LEMMA, DEP, NER, SRL, COREF, WSD, SA	~1M tokens / 881K words / 2,913 texts manually annotated with MULTEXT-East V6, JOS, and Universal Dependencies. Integrates ssj500k 2.3, Ambiga, ElexisWSD, and SentiCoref subcorpora. License: CC BY-SA 4.0.
ssj500k 2.3	CLARIN.SI	POS, LEMMA, DEP, NER, SRL	~500K tokens manually annotated with MSD tags, lemmas, UD syntax (UD 2.8), named entities, and semantic role labels. Foundation corpus for SUK 1.1. License: CC BY-NC-SA 4.0.
Slovene SuperGLUE	CLARIN.SI	QA, NLI, WSD, COREF, MRC	Slovene translation of SuperGLUE (BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, WSC). Mix of human and Google MT translation. License: CC BY 4.0. Convert to per-task evaluation files with scripts/data/to_superglue.py.
SentiNews 1.0	CLARIN.SI	SA	Slovene news sentiment with three-level annotations (sentence, paragraph, document) and 3-class labels. Directly downloadable. License: CC BY-SA 4.0. Convert to evaluation JSONL with scripts/data/to_sentiment.py.
Sloleks 3.1	CLARIN.SI	TOKENIZER	Slovenian inflectional lexicon (lemmas + word forms with MULTEXT-East V6 / JOS MSDs). Tokenizer / morphology evaluation only — intentionally absent from extract.yaml, never enters the pretraining corpus. Distributed as TEI XML. License: CC BY-SA 4.0. Convert with scripts/data/to_tokenization.py.

Task abbreviations

• POS — part-of-speech tagging
• LEMMA — lemmatization
• DEP — dependency parsing
• NER — named entity recognition
• SRL — semantic role labeling
• COREF — coreference resolution
• WSD — word sense disambiguation
• SA — sentiment analysis
• NLI — natural language inference
• QA — question answering
• MRC — machine reading comprehension
• TOKENIZER — tokenizer / morphology evaluation (lexicon-based, not a downstream IE task)

Acknowledgments

The project is funded by ARIS (Slovenian Research and Innovation Agency) under the project number Z2-70067.