From db22afea7ddebb1eba8abe3dfeeeec56bf690611 Mon Sep 17 00:00:00 2001 From: Attila Erbilir Date: Sat, 4 Jul 2026 13:54:52 +0300 Subject: [PATCH] docs: add Turkish lexicon expansion policy Co-authored-by: Cursor --- CHANGELOG.md | 5 +- CONTRIBUTING.md | 23 ++++ docs/dictionary-expansion-policy.md | 165 +++++++++++++++++++++++++++- docs/specification.md | 19 +++- 4 files changed, 209 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 714b07d..91318a2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,10 +13,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `CorpusLoader` and `CorpusQualityReport` support classes for fixture validation and quality metrics - `tests/TurkishCorpusQualityTest.php` with aggregate FP/FN/coverage gate - `docs/dictionary-expansion-policy.md` — criteria for future Turkish dictionary entry PRs +- Inline Turkish lexicon taxonomy in `docs/dictionary-expansion-policy.md` — four supported categories (`profanity`, `insult`, `sexual`, `abbreviation`), severity guidance, Batch 1 acceptance policy, corpus requirements, and maintainer curation workflow ### Changed -- `CONTRIBUTING.md` — corpus authoring rules and dictionary expansion policy link +- `docs/dictionary-expansion-policy.md` — expanded with v0.4 three-PR workflow, inline taxonomy, severity guidance, Batch 1 rules, maintainer review stages, and explicit slur out-of-scope note +- `docs/specification.md` — supported dictionary categories documented with link to expansion policy +- `CONTRIBUTING.md` — v0.4 three-PR lexicon expansion workflow, offensive language contribution expectations, maintainer review requirements, and corpus-first workflow ## [0.2.0] - 2026-07-03 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 843fe5b..3f1ac5f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -91,6 +91,29 @@ New TR entry PRs **must** include: Do not add dictionary entries without corresponding corpus coverage. +### v0.4 Turkish lexicon expansion (three-PR workflow) + +Controlled Turkish dictionary expansion in v0.4 is split across three pull requests. Do not combine policy, research, and implementation in a single PR. + +| PR | Purpose | Changes allowed | +|----|---------|-----------------| +| **PR 1 — Documentation and policy** | Inline taxonomy, severity guidance, Batch 1 rules | Policy docs only — no dictionary or code changes | +| **PR 2 — Research and curation** | Candidate pool, maintainer evaluation, Approved Batch 1 list | Research document only — no `data/tr.php` or fixture changes | +| **PR 3 — Implementation** | Approved entries and corpus | `data/tr.php` + fixture corpus for exactly 15 approved entries | + +**Maintainer review is required.** PR 3 may only implement entries explicitly approved in PR 2 (Stage 3 — Approved Batch 1). Do not auto-generate dictionary rows from research output. + +**Corpus-first workflow.** Every approved entry must have a corpus plan before implementation. PR 3 must include clean neighbor, profane, and obfuscated cases per [docs/dictionary-expansion-policy.md](docs/dictionary-expansion-policy.md) before merge. + +**Offensive language in contributions.** Research and fixture PRs may reference offensive terms when necessary for moderation quality. Follow these expectations: + +- Summarize usage patterns — do not paste large blocks of raw user-generated content. +- Fixture cases must be minimal, purposeful, and tied to detection or false-positive documentation. +- Offensive fixture additions require maintainer review. +- Do not scrape or store raw comment archives in the repository. + +See [docs/dictionary-expansion-policy.md](docs/dictionary-expansion-policy.md) for category taxonomy, Batch 1 acceptance criteria, and the three-stage curation workflow. + ## Reporting Bugs Use the [bug report issue template](.github/ISSUE_TEMPLATE/bug_report.yml). Include: diff --git a/docs/dictionary-expansion-policy.md b/docs/dictionary-expansion-policy.md index 9192b15..8b7264c 100644 --- a/docs/dictionary-expansion-policy.md +++ b/docs/dictionary-expansion-policy.md @@ -1,9 +1,11 @@ # Turkish Dictionary Expansion Policy -This document defines the quality gate for adding or changing Turkish dictionary entries in VerbaGuard. +This document defines the quality gate, inline taxonomy, and maintainer workflow for adding or changing Turkish dictionary entries in VerbaGuard. For corpus format and test infrastructure, see [`tests/Fixtures/tr/README.md`](../tests/Fixtures/tr/README.md). +For dictionary author format, see [`docs/specification.md`](specification.md). + --- ## Principles @@ -11,6 +13,167 @@ For corpus format and test infrastructure, see [`tests/Fixtures/tr/README.md`](. 1. **False positives are worse than false negatives** — see [`FOUNDATION.md`](../FOUNDATION.md). 2. **Corpus before expansion** — measurable quality ground must exist before growing the dictionary. 3. **Short terms need extra care** — tokens of three characters or fewer have higher false-positive risk in real text. +4. **Quality over volume** — expansion success is measured by corpus-backed confidence and zero false positives, not by entry count. + +--- + +## Inline taxonomy (v0.4) + +VerbaGuard v0.4 uses four supported dictionary categories. Category definitions live in this document. A separate taxonomy reference file (`docs/turkish-lexicon-taxonomy.md`) is intentionally **deferred** until the Turkish dictionary reaches approximately **100 curated entries**. + +### `profanity` + +- **Definition:** General profanity and vulgar slang where profanity is the dominant intent (not a personal insult). +- **Example types:** Verb roots, common standalone profanity tokens. +- **Typical severity:** `medium`–`high` +- **False-positive risk:** Medium + +### `insult` + +- **Definition:** Personal degradation — attacks on worth, competence, or identity. +- **Example types:** Standalone insult adjectives (prefer ≥4 characters). +- **Typical severity:** `low`–`medium` +- **False-positive risk:** High — substring collisions with innocent words are common. + +### `sexual` + +- **Definition:** Explicit sexual slang or sexual profanity. +- **Example types:** Sexual organ or act slang (prefer ≥4 characters, clearly standalone). +- **Typical severity:** `medium`–`high` +- **False-positive risk:** Medium — medical or biological substring risk. + +### `abbreviation` + +- **Definition:** High-confidence abbreviation of profanity or insult; normalized form must be unique in the dictionary. +- **Example types:** Chat abbreviations, short forms related to an existing entry. +- **Typical severity:** `low`–`medium` +- **False-positive risk:** Very high — use sparingly; only when collision risk is low. + +### Explicitly excluded: `slur` + +> Slur moderation is a distinct moderation domain with additional ethical, cultural, and review requirements. It is intentionally **out of scope for v0.4**. Reconsider in a later major/minor roadmap after a separate ethical, cultural, and review policy is defined. + +Do not add slur-category terms in v0.4 expansion work. + +--- + +## Severity guidance + +Severity reflects moderation urgency, not term frequency. Category does **not** automatically determine severity. + +| Severity | Score | Usage criteria | +|----------|-------|----------------| +| `low` | 10 | Lower aggression; precedent: existing `aq`, `mal` | +| `medium` | 25 | Clear profanity; standalone token (`amk`) | +| `high` | 50 | Severe profanity or explicit sexual slang (`siktir`, `orospu`) | + +Rules: + +1. Assign severity based on moderation impact, not how often a term appears online. +2. Document severity rationale in research and PR risk assessments. +3. The severity enum is fixed — see [`src/Severity.php`](../src/Severity.php). + +--- + +## v0.4 expansion workflow (three PRs) + +Turkish lexicon expansion in v0.4 follows a controlled three-pull-request sequence: + +| PR | Scope | Dictionary / code changes | +|----|-------|---------------------------| +| **PR 1** | Policy and taxonomy documentation | None | +| **PR 2** | Research and curation | None | +| **PR 3** | Approved entries + corpus | `data/tr.php` + fixtures | + +PR 3 cannot start until maintainer approval of the Batch 1 entry list from PR 2. + +--- + +## Batch 1 acceptance policy + +Batch 1 adds exactly **15** high-confidence Turkish entries in a single implementation PR (PR 3). + +### Target distribution (flexible — planning guidance only) + +These numbers are planning guidance, not hard quotas. Research may justify a different category mix when supported by evidence. + +| Category | Target (flexible) | +|----------|-------------------| +| `profanity` | ≈ 6 | +| `insult` | ≈ 3 | +| `sexual` | ≈ 4 | +| `abbreviation` | ≈ 2 | +| `slur` | **0** | + +Maintainer approval may adjust category counts based on real-world frequency, false-positive risk, normalized collisions, corpus quality, and research evidence. Total implementation size remains exactly **15 entries**. + +### Selection rules + +- Prefer **≥4 character** terms. +- **No new ≤2 character** entries in Batch 1. +- Avoid ambiguous common words unless strongly justified with clean neighbor coverage. +- Abbreviations: high-confidence only, low normalized-collision risk. +- Prefer **broad online usage** over niche slang. +- Every candidate must have a **corpus plan** before PR 3 approval. +- Do not copy large amounts of raw offensive user-generated content into the repository. + +### Entry acceptance criteria + +An entry may enter PR 3 only when all of the following hold: + +1. Listed in PR 2 research as **Approved Batch 1** (selected from the Recommended group). +2. Canonical `term` with a unique normalized key (build-time check via `DictionaryBuilder`). +3. Category ∈ {`profanity`, `insult`, `sexual`, `abbreviation`}. +4. **Not** ≤2 characters (Batch 1 rule). +5. Not an ambiguous common word without strong justification and clean neighbors. +6. Corpus complete per minimum table below. +7. Aggregate quality gate passes (FP = 0, FN = 0, coverage = 100%). +8. PR risk assessment block filled (template below). + +**Automatic reject:** slur-category terms, new ≤2 character entries, normalized key collision, false-positive gate failure. + +--- + +## Maintainer review workflow (PR 2 curation) + +Research should maximize **discovery**. Implementation should maximize **quality**. The candidate pool is larger than the implementation batch. + +### Stage 1 — Candidate pool + +- Collect approximately **30–40 candidate terms** through research. +- Summarize usage patterns at a high level — do not archive raw user comments in the repository. + +### Stage 2 — Maintainer evaluation + +Classify every candidate as **Recommended**, **Deferred**, or **Rejected**. Each classification must include a short rationale. + +| Status | Meaning | +|--------|---------| +| **Recommended** | Strong fit for Batch 1; passes selection rules | +| **Deferred** | Potentially valid; postponed (risk, collision, evidence gap) | +| **Rejected** | Does not meet acceptance criteria | + +### Stage 3 — Approved Batch 1 + +- Maintainer selects exactly **15 entries** from the **Recommended** group only. +- Final list includes category, severity, false-positive/collision assessment, and corpus plan per entry. +- **PR 3 may only use these 15 approved entries.** + +### Research output format (per candidate) + +| Field | Description | +|-------|-------------| +| Canonical term (proposed) | Maintainer-facing; high-level in research doc | +| Category | profanity / insult / sexual / abbreviation | +| Likely severity | low / medium / high + rationale | +| FP risk | low / medium / high | +| Normalized collision risk | vs existing entries + other candidates | +| Corpus plan | min profane / obfuscated / clean counts | +| Classification rationale | Reason for Recommended / Deferred / Rejected | + +### Approval gate + +PR 2 is complete when the research document includes all three stages and exactly **15 Approved Batch 1** entries are signed off. PR 3 cannot begin until this gate passes. --- diff --git a/docs/specification.md b/docs/specification.md index 1b26080..6e58cdf 100644 --- a/docs/specification.md +++ b/docs/specification.md @@ -93,7 +93,7 @@ Each author row contains only user-written canonical fields: | Field | Description | |------------|-----------------------------------------------| | `term` | Canonical dictionary term | -| `category` | Semantic category, e.g. `profanity`, `insult` | +| `category` | Semantic category (see [Supported categories](#supported-dictionary-categories)) | | `severity` | One of `clean`, `low`, `medium`, `high` | Do **not** include `normalized` in author rows. It is derived at dictionary build time. @@ -135,6 +135,23 @@ At build time, each `term` is passed through `normalizeKey` to produce the deriv - `Dictionary::fromArray()` removed — use `Dictionary::fromRows()` instead. - Author dictionary rows no longer accept a `normalized` field. +### Supported dictionary categories + +The `category` field is author-defined metadata carried on each dictionary entry. VerbaGuard v0.4 recognizes four supported categories for Turkish expansion: + +| Category | Description | +|----------|-------------| +| `profanity` | General profanity and vulgar slang | +| `insult` | Personal degradation or attack on worth/competence | +| `sexual` | Explicit sexual slang or sexual profanity | +| `abbreviation` | High-confidence abbreviation of profanity or insult | + +Category definitions, severity guidance, false-positive risk notes, and expansion rules are documented in [`docs/dictionary-expansion-policy.md`](dictionary-expansion-policy.md). + +**Out of scope for v0.4:** the `slur` category. Slur moderation requires a separate ethical, cultural, and review policy and is not a supported category in current expansion work. + +A standalone taxonomy reference document is intentionally deferred until the Turkish dictionary reaches approximately 100 curated entries. + --- ## Normalization stages