Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `CorpusLoader` and `CorpusQualityReport` support classes for fixture validation and quality metrics
- `tests/TurkishCorpusQualityTest.php` with aggregate FP/FN/coverage gate
- `docs/dictionary-expansion-policy.md` — criteria for future Turkish dictionary entry PRs
- Inline Turkish lexicon taxonomy in `docs/dictionary-expansion-policy.md` — four supported categories (`profanity`, `insult`, `sexual`, `abbreviation`), severity guidance, Batch 1 acceptance policy, corpus requirements, and maintainer curation workflow

### Changed

- `CONTRIBUTING.md` — corpus authoring rules and dictionary expansion policy link
- `docs/dictionary-expansion-policy.md` — expanded with v0.4 three-PR workflow, inline taxonomy, severity guidance, Batch 1 rules, maintainer review stages, and explicit slur out-of-scope note
- `docs/specification.md` — supported dictionary categories documented with link to expansion policy
- `CONTRIBUTING.md` — v0.4 three-PR lexicon expansion workflow, offensive language contribution expectations, maintainer review requirements, and corpus-first workflow

## [0.2.0] - 2026-07-03

Expand Down
23 changes: 23 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,29 @@ New TR entry PRs **must** include:

Do not add dictionary entries without corresponding corpus coverage.

### v0.4 Turkish lexicon expansion (three-PR workflow)

Controlled Turkish dictionary expansion in v0.4 is split across three pull requests. Do not combine policy, research, and implementation in a single PR.

| PR | Purpose | Changes allowed |
|----|---------|-----------------|
| **PR 1 — Documentation and policy** | Inline taxonomy, severity guidance, Batch 1 rules | Policy docs only — no dictionary or code changes |
| **PR 2 — Research and curation** | Candidate pool, maintainer evaluation, Approved Batch 1 list | Research document only — no `data/tr.php` or fixture changes |
| **PR 3 — Implementation** | Approved entries and corpus | `data/tr.php` + fixture corpus for exactly 15 approved entries |

**Maintainer review is required.** PR 3 may only implement entries explicitly approved in PR 2 (Stage 3 — Approved Batch 1). Do not auto-generate dictionary rows from research output.

**Corpus-first workflow.** Every approved entry must have a corpus plan before implementation. PR 3 must include clean neighbor, profane, and obfuscated cases per [docs/dictionary-expansion-policy.md](docs/dictionary-expansion-policy.md) before merge.

**Offensive language in contributions.** Research and fixture PRs may reference offensive terms when necessary for moderation quality. Follow these expectations:

- Summarize usage patterns — do not paste large blocks of raw user-generated content.
- Fixture cases must be minimal, purposeful, and tied to detection or false-positive documentation.
- Offensive fixture additions require maintainer review.
- Do not scrape or store raw comment archives in the repository.

See [docs/dictionary-expansion-policy.md](docs/dictionary-expansion-policy.md) for category taxonomy, Batch 1 acceptance criteria, and the three-stage curation workflow.

## Reporting Bugs

Use the [bug report issue template](.github/ISSUE_TEMPLATE/bug_report.yml). Include:
Expand Down
165 changes: 164 additions & 1 deletion docs/dictionary-expansion-policy.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,179 @@
# Turkish Dictionary Expansion Policy

This document defines the quality gate for adding or changing Turkish dictionary entries in VerbaGuard.
This document defines the quality gate, inline taxonomy, and maintainer workflow for adding or changing Turkish dictionary entries in VerbaGuard.

For corpus format and test infrastructure, see [`tests/Fixtures/tr/README.md`](../tests/Fixtures/tr/README.md).

For dictionary author format, see [`docs/specification.md`](specification.md).

---

## Principles

1. **False positives are worse than false negatives** — see [`FOUNDATION.md`](../FOUNDATION.md).
2. **Corpus before expansion** — measurable quality ground must exist before growing the dictionary.
3. **Short terms need extra care** — tokens of three characters or fewer have higher false-positive risk in real text.
4. **Quality over volume** — expansion success is measured by corpus-backed confidence and zero false positives, not by entry count.

---

## Inline taxonomy (v0.4)

VerbaGuard v0.4 uses four supported dictionary categories. Category definitions live in this document. A separate taxonomy reference file (`docs/turkish-lexicon-taxonomy.md`) is intentionally **deferred** until the Turkish dictionary reaches approximately **100 curated entries**.

### `profanity`

- **Definition:** General profanity and vulgar slang where profanity is the dominant intent (not a personal insult).
- **Example types:** Verb roots, common standalone profanity tokens.
- **Typical severity:** `medium`–`high`
- **False-positive risk:** Medium

### `insult`

- **Definition:** Personal degradation — attacks on worth, competence, or identity.
- **Example types:** Standalone insult adjectives (prefer ≥4 characters).
- **Typical severity:** `low`–`medium`
- **False-positive risk:** High — substring collisions with innocent words are common.

### `sexual`

- **Definition:** Explicit sexual slang or sexual profanity.
- **Example types:** Sexual organ or act slang (prefer ≥4 characters, clearly standalone).
- **Typical severity:** `medium`–`high`
- **False-positive risk:** Medium — medical or biological substring risk.

### `abbreviation`

- **Definition:** High-confidence abbreviation of profanity or insult; normalized form must be unique in the dictionary.
- **Example types:** Chat abbreviations, short forms related to an existing entry.
- **Typical severity:** `low`–`medium`
- **False-positive risk:** Very high — use sparingly; only when collision risk is low.

### Explicitly excluded: `slur`

> Slur moderation is a distinct moderation domain with additional ethical, cultural, and review requirements. It is intentionally **out of scope for v0.4**. Reconsider in a later major/minor roadmap after a separate ethical, cultural, and review policy is defined.

Do not add slur-category terms in v0.4 expansion work.

---

## Severity guidance

Severity reflects moderation urgency, not term frequency. Category does **not** automatically determine severity.

| Severity | Score | Usage criteria |
|----------|-------|----------------|
| `low` | 10 | Lower aggression; precedent: existing `aq`, `mal` |
| `medium` | 25 | Clear profanity; standalone token (`amk`) |
| `high` | 50 | Severe profanity or explicit sexual slang (`siktir`, `orospu`) |

Rules:

1. Assign severity based on moderation impact, not how often a term appears online.
2. Document severity rationale in research and PR risk assessments.
3. The severity enum is fixed — see [`src/Severity.php`](../src/Severity.php).

---

## v0.4 expansion workflow (three PRs)

Turkish lexicon expansion in v0.4 follows a controlled three-pull-request sequence:

| PR | Scope | Dictionary / code changes |
|----|-------|---------------------------|
| **PR 1** | Policy and taxonomy documentation | None |
| **PR 2** | Research and curation | None |
| **PR 3** | Approved entries + corpus | `data/tr.php` + fixtures |

PR 3 cannot start until maintainer approval of the Batch 1 entry list from PR 2.

---

## Batch 1 acceptance policy

Batch 1 adds exactly **15** high-confidence Turkish entries in a single implementation PR (PR 3).

### Target distribution (flexible — planning guidance only)

These numbers are planning guidance, not hard quotas. Research may justify a different category mix when supported by evidence.

| Category | Target (flexible) |
|----------|-------------------|
| `profanity` | ≈ 6 |
| `insult` | ≈ 3 |
| `sexual` | ≈ 4 |
| `abbreviation` | ≈ 2 |
| `slur` | **0** |

Maintainer approval may adjust category counts based on real-world frequency, false-positive risk, normalized collisions, corpus quality, and research evidence. Total implementation size remains exactly **15 entries**.

### Selection rules

- Prefer **≥4 character** terms.
- **No new ≤2 character** entries in Batch 1.
- Avoid ambiguous common words unless strongly justified with clean neighbor coverage.
- Abbreviations: high-confidence only, low normalized-collision risk.
- Prefer **broad online usage** over niche slang.
- Every candidate must have a **corpus plan** before PR 3 approval.
- Do not copy large amounts of raw offensive user-generated content into the repository.

### Entry acceptance criteria

An entry may enter PR 3 only when all of the following hold:

1. Listed in PR 2 research as **Approved Batch 1** (selected from the Recommended group).
2. Canonical `term` with a unique normalized key (build-time check via `DictionaryBuilder`).
3. Category ∈ {`profanity`, `insult`, `sexual`, `abbreviation`}.
4. **Not** ≤2 characters (Batch 1 rule).
5. Not an ambiguous common word without strong justification and clean neighbors.
6. Corpus complete per minimum table below.
7. Aggregate quality gate passes (FP = 0, FN = 0, coverage = 100%).
8. PR risk assessment block filled (template below).

**Automatic reject:** slur-category terms, new ≤2 character entries, normalized key collision, false-positive gate failure.

---

## Maintainer review workflow (PR 2 curation)

Research should maximize **discovery**. Implementation should maximize **quality**. The candidate pool is larger than the implementation batch.

### Stage 1 — Candidate pool

- Collect approximately **30–40 candidate terms** through research.
- Summarize usage patterns at a high level — do not archive raw user comments in the repository.

### Stage 2 — Maintainer evaluation

Classify every candidate as **Recommended**, **Deferred**, or **Rejected**. Each classification must include a short rationale.

| Status | Meaning |
|--------|---------|
| **Recommended** | Strong fit for Batch 1; passes selection rules |
| **Deferred** | Potentially valid; postponed (risk, collision, evidence gap) |
| **Rejected** | Does not meet acceptance criteria |

### Stage 3 — Approved Batch 1

- Maintainer selects exactly **15 entries** from the **Recommended** group only.
- Final list includes category, severity, false-positive/collision assessment, and corpus plan per entry.
- **PR 3 may only use these 15 approved entries.**

### Research output format (per candidate)

| Field | Description |
|-------|-------------|
| Canonical term (proposed) | Maintainer-facing; high-level in research doc |
| Category | profanity / insult / sexual / abbreviation |
| Likely severity | low / medium / high + rationale |
| FP risk | low / medium / high |
| Normalized collision risk | vs existing entries + other candidates |
| Corpus plan | min profane / obfuscated / clean counts |
| Classification rationale | Reason for Recommended / Deferred / Rejected |

### Approval gate

PR 2 is complete when the research document includes all three stages and exactly **15 Approved Batch 1** entries are signed off. PR 3 cannot begin until this gate passes.

---

Expand Down
19 changes: 18 additions & 1 deletion docs/specification.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ Each author row contains only user-written canonical fields:
| Field | Description |
|------------|-----------------------------------------------|
| `term` | Canonical dictionary term |
| `category` | Semantic category, e.g. `profanity`, `insult` |
| `category` | Semantic category (see [Supported categories](#supported-dictionary-categories)) |
| `severity` | One of `clean`, `low`, `medium`, `high` |

Do **not** include `normalized` in author rows. It is derived at dictionary build time.
Expand Down Expand Up @@ -135,6 +135,23 @@ At build time, each `term` is passed through `normalizeKey` to produce the deriv
- `Dictionary::fromArray()` removed — use `Dictionary::fromRows()` instead.
- Author dictionary rows no longer accept a `normalized` field.

### Supported dictionary categories

The `category` field is author-defined metadata carried on each dictionary entry. VerbaGuard v0.4 recognizes four supported categories for Turkish expansion:

| Category | Description |
|----------|-------------|
| `profanity` | General profanity and vulgar slang |
| `insult` | Personal degradation or attack on worth/competence |
| `sexual` | Explicit sexual slang or sexual profanity |
| `abbreviation` | High-confidence abbreviation of profanity or insult |

Category definitions, severity guidance, false-positive risk notes, and expansion rules are documented in [`docs/dictionary-expansion-policy.md`](dictionary-expansion-policy.md).

**Out of scope for v0.4:** the `slur` category. Slur moderation requires a separate ethical, cultural, and review policy and is not a supported category in current expansion work.

A standalone taxonomy reference document is intentionally deferred until the Turkish dictionary reaches approximately 100 curated entries.

---

## Normalization stages
Expand Down
Loading