🇹🇳 Za3tar: The Tunisian Dialect LLM

Za3tar is a Large Language Model (LLM) trained completely from scratch to speak, understand, and generate authentic Tunisian Derja (Dialect).

Unlike many projects that simply fine-tune existing English models (like Llama or GPT), Za3tar was pre-trained on a custom dataset of Tunisian text using a modern transformer architecture optimized for low-resource languages.

---

🚀 Features

• From Scratch Architecture: Not a Llama finetune. A custom GPT-style model built layer-by-layer.
• Modern Transformer Tech: Implements RoPE (Rotary Embeddings), RMSNorm, SwiGLU, and Flash Attention for state-of-the-art performance.
• Custom Tokenizer: A BPE (Byte Pair Encoding) tokenizer trained specifically on Tunisian script (Arabic letters), optimizing vocabulary for the dialect.
• Hybrid Data Strategy: Trained on a mix of real-world scraped data (YouTube, Social Media) and high-quality synthetic data generated via Mistral Large to ensure logic and politeness.
• Optimized Deployment: Lightweight architecture capable of running locally on CPU.

---

🧠 Model Architecture

We follow the Chinchilla Scaling Laws for a compute-optimal model size relative to the dataset.

Hyperparameter	Value
Parameters	~100 Million
Context Window	512 Tokens
Embedding Dim	768
Layers	12
Heads	12
Vocab Size	10,000 (Custom BPE)
Attention	Causal Self-Attention with RoPE

---

📂 Project Structure

Za3tar/
├── data/                       # Raw data folder
├── app.py                      # Streamlit Inference Interface
├── pretraining.py              # Stage 1: Main Pre-training script
├── sft_trainer.py              # Stage 2: Supervised Fine-Tuning script
├── tokenizer.py                # Script to train BPE tokenizer
├── preprocess_data.py          # Converts text to binary for training
├── sft_data_maker.py           # API script to generate synthetic Q&A
├── identity-fuse.py            # Balances identity data (The "Golden Ratio")
├── modern_gpt_model.pth        # Saved Pre-trained Weights
├── tunisian_chatbot_final.pth  # Saved Final SFT Weights
├── vocab.json                  # Tokenizer Vocabulary
├── merges.txt                  # Tokenizer Merges
└── requirement.txt             # Python Dependencies

---

🛠️ Installation

1. Clone the repository:

   git clone https://github.com/YourUsername/Za3tar.git
   cd Za3tar

2. Create the Virtual Environment:

   python -m venv za3tar_env
   # Windows:
   .\za3tar_env\Scripts\activate
   # Mac/Linux:
   source za3tar_env/bin/activate

3. Install Dependencies:

   pip install -r requirement.txt

   (Note: If you have an NVIDIA GPU, ensure you install the CUDA version of PyTorch).

---

⚙️ Training Workflow (From Scratch)

This project follows a strict pipeline from data creation to final inference.

1. Data Preparation & Tokenization

First, we train the BPE tokenizer on raw Tunisian text to create vocab.json and merges.txt.

python tokenizer.py

Then, we convert the raw text data into a binary format (tokenized_data.bin) for efficient training.

python preprocess_data.py

2. Pre-Training (The "Brain")

We train the model on the massive dataset to learn the structure of the language. This produces modern_gpt_model.pth.

python pretraining.py

3. Data Processing for SFT

We generate high-quality instruction data to teach the model how to be helpful.

python sft_data_maker.py
python data_fuser.py

4. Supervised Fine-Tuning (The "Manners")

We fine-tune the pre-trained model on the balanced dataset to teach it how to answer questions politely. This produces tunisian_chatbot_final.pth.

python sft_trainer.py

5. Identity Fusion (The "Persona")

We balance the general chat data with specific identity data so the model knows its name ("Za3tar") without overfitting or forgetting general knowledge.

python identity-fuse.py

---

💬 Inference (Running the App)

Once training is complete, you can chat with Za3tar using the web interface.

streamlit run app.py

• The app runs on CPU by default.
• It implements a stopping mechanism to prevent hallucinations.

---

You can try Za3tar directly in your browser without installing anything!

👉 Click here to chat with Za3tar

⚠️ Important: You must use Arabic letters (Arabic Script) when chatting. The model was trained strictly on Tunisian script and will not understand Latin characters (Arabizi).

📜 Credits & Acknowledgments

• Lead Developer: Mohamed Ahmed Krichen
• Data Sources: Special thanks to all data makers on Hugging Face whose Tunisian datasets made the foundation of this project possible.
• Synthetic Data: Powered by Mistral AI & Groq.
• Inspiration: Andrej Karpathy's NanoGPT and Llama 2 architecture.

---

Made with ❤️ in Tunisia 🇹🇳