🐉 EsDrac — the first LLM that speaks Mallorcan

EsDrac-v1-7B is the first language model that really speaks Mallorcan (mallorquí) — the Balearic variant of Catalan, with the salat article (es, sa, ses) — not just standard Catalan. It also handles Spanish and English, keeps Qwen's agentic / tool-calling ability, and is de-censored on Chinese-politics topics.

Built on Qwen2.5-7B-Instruct (Apache 2.0): abliterated → LoRA fine-tune on a hand-built Mallorcan corpus → merged → GGUF. This repo holds the full pipeline that produced it, plus a free demo.

Try it (free, ~2 min)

1. Click Open in Colab above.
2. Runtime → Change runtime type → T4 GPU (free).
3. Runtime → Run all.
4. The last cell prints a public gradio.live link you can chat with and share.

Links

• 🐉 Model: https://huggingface.co/natzx94/EsDrac-v1-7B
• 📦 GGUF (Ollama / LM Studio / llama.cpp): https://huggingface.co/natzx94/EsDrac-v1-7B-GGUF
• 📚 Dataset: https://huggingface.co/datasets/natzx94/EsDrac-mallorqui

How EsDrac was built

Qwen2.5-7B-Instruct
   │  1. abliteration        (abliteration/) — remove the refusal direction
   ▼
de-censored base
   │  2. dataset             (data/, corpus_mallorqui/, ds_mallorqui/)
   │                          rondaies (public domain) + modernised + synthetic mallorquí
   │  3. LoRA fine-tune      (training/) — r=128, RAM-safe, ~5 h on 16 GB VRAM
   ▼
EsDrac-v1-7B
   │  4. eval                (eval/) — mallorquí, idiomas, política, tool-calling
   │  5. GGUF export         (gguf/) — for Ollama / llama.cpp
   ▼
published to Hugging Face + Gradio Space (space/)

Key lesson: the ceiling for generative Mallorcan was model size, not data. A 3B merge could not compose real salat sentences; the 7B LoRA can.

Repository structure

Folder	What's inside
abliteration/	Remove the refusal direction from the base model (abliterate_esdrac.py) + debias data.
data/	Dataset builders: download/modernise the rondaies, generate synthetic mallorquí, mix recovery data.
corpus_mallorqui/	Source texts — Antoni Maria Alcover's Rondaies (public domain) and modernised variants.
ds_mallorqui/	Final instruction dataset (esdrac_mallorqui.jsonl) + card.
training/	LoRA training (train_esdrac_lora.py), auto-resume scripts, HF upload.
agentic/	ReAct agent harness and agentic-trajectory dataset builder (tool-calling training).
eval/	Evaluators: Mallorcan quality, multilingual retention, politics de-censoring, tool-calling.
gguf/	Convert the merged model to GGUF for Ollama / llama.cpp.
space/	The Hugging Face Gradio Space app.
EsDrac_demo.ipynb	Free Colab demo (T4).
MODELCARD_esdrac.md	Full model card (mallorquí).

Reproduce

Each stage is a standalone script; you need a CUDA GPU (built on an RTX 5070 Ti, 16 GB) and transformers, peft, trl, datasets. Rough order:

python abliteration/abliterate_esdrac.py      # 1. de-censor the base
python data/build_mallorqui_dataset.py        # 2. build the dataset
python training/train_esdrac_lora.py          # 3. LoRA fine-tune
python eval/eval_esdrac.py                     # 4. evaluate
python gguf/convert_hf_to_gguf.py ...          # 5. export GGUF

License

Derived from Qwen2.5-7B-Instruct — Apache 2.0. Distributed under the same license. The Rondaies corpus is public domain (Alcover, early 1900s).

Fet a Mallorca. 🐉