LLM Engineering Platform

🔗 Dependencies

Local dependencies

Tool	Version	Purpose	Installation Link
pyenv	≥2.3.36	Multiple Python versions (optional)	Install Guide
Python	3.11	Runtime environment	Download
Poetry	>= 1.8.3 and < 2.0	Package management	Install Guide
Docker	≥27.1.1	Containerization	Install Guide
AWS CLI	≥2.15.42	Cloud management	Install Guide
Git	≥2.44.0	Version control	Download

Cloud services

Service	Purpose
HuggingFace	Model registry
Comet ML	Experiment tracker
Opik	Prompt monitoring
ZenML	Orchestrator and artifacts layer
AWS	Compute and storage
MongoDB	NoSQL database
Qdrant	Vector database
GitHub Actions	CI/CD pipeline

🗂️ Project Structure

.
├── code_snippets/       # Standalone example code
├── configs/             # Pipeline configuration files
├── llm_engineering/     # Core project package
│   ├── application/    
│   ├── domain/         
│   ├── infrastructure/ 
│   ├── model/         
├── pipelines/           # ML pipeline definitions
├── steps/               # Pipeline components
├── tests/               # Test examples
├── tools/               # Utility scripts
│   ├── run.py
│   ├── ml_service.py
│   ├── rag.py
│   ├── data_warehouse.py

llm_engineering/ is the main Python package. It follows Domain-Driven Design (DDD) principles:

• domain/: Core business entities and structures
• application/: Business logic, crawlers, and RAG implementation
• model/: LLM training and inference
• infrastructure/: External service integrations (AWS, Qdrant, MongoDB, FastAPI)

Code imports flow: infrastructure → model → application → domain

pipelines/: ZenML ML pipelines — entry point for all ML workflows.

steps/: Reusable ZenML step components (data loading, preprocessing, etc.).

tools/: Utility scripts:

• run.py: Entry point to run ZenML pipelines.
• ml_service.py: Starts the REST API inference server.
• rag.py: Demonstrates RAG retrieval module usage.
• data_warehouse.py: Export/import data from MongoDB via JSON files.

configs/: ZenML YAML configuration files for pipelines and steps.

code_snippets/: Independent code examples that run standalone.

---

💻 Installation

1. Clone the Repository

git clone https://github.com/PacktPublishing/LLM-Engineers-Handbook.git
cd LLM-Engineers-Handbook 

2. Set Up Python Environment

The project requires Python 3.11.

Option A: Global Python

python --version  # Should show Python 3.11.x

Option B: pyenv (recommended)

pyenv --version           # Should show pyenv 2.3.36 or later
pyenv install 3.11.8
python --version          # Should show Python 3.11.8

The project includes a .python-version file that automatically sets the correct Python version in the project directory.

3. Install Dependencies

poetry --version          # Should show Poetry 1.8.3 or later
poetry env use 3.11
poetry install --without aws
poetry run pre-commit install

4. Activate the Environment

poetry shell

Run project commands via Poe the Poet:

poetry poe <command>

🔧 Troubleshooting Poe the Poet

If poethepoet has issues, run commands directly via Poetry:

1. Look up the command in the [tool.poe.tasks] section of pyproject.toml
2. Run it with poetry run <actual-command>

5. Local Development Setup

Copy the example env file and fill in your credentials:

cp .env.example .env

OpenAI

OPENAI_API_KEY=your_api_key_here

→ How to get an OpenAI API key

Hugging Face

HUGGINGFACE_ACCESS_TOKEN=your_token_here

→ How to create a Hugging Face token

Comet ML & Opik

COMET_API_KEY=your_api_key_here

→ Opik getting started

6. Deployment Setup

For cloud deployments, set these additional env vars. For local-only work, the defaults work out of the box.

MongoDB

DATABASE_HOST=your_mongodb_url

→ Create a free MongoDB cluster

Qdrant

USE_QDRANT_CLOUD=true
QDRANT_CLOUD_URL=your_qdrant_cloud_url
QDRANT_APIKEY=your_qdrant_api_key

→ Create a free Qdrant cluster

AWS

AWS_REGION=eu-central-1
AWS_ACCESS_KEY=your_aws_access_key
AWS_SECRET_KEY=your_aws_secret_key

Additional configuration options are available in llm_engineering/settings.py. Any variable in the Settings class can be set via .env.

---

🏗️ Infrastructure

Local infrastructure

Hosts MongoDB and Qdrant via Docker, plus a local ZenML server.

Requires Docker >= v27.1.1

# Start all local infrastructure
poetry poe local-infrastructure-up

# Stop all local infrastructure
poetry poe local-infrastructure-down

macOS only: Before starting the server, run:

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

This is done automatically when using Poe the Poet.

Start the inference REST API:

poetry poe run-inference-ml-service

ZenML

• Dashboard: localhost:8237
• Username: default / Password: (empty)

Qdrant

• REST API: localhost:6333
• Dashboard: localhost:6333/dashboard

MongoDB

• URI: mongodb://llm_engineering:llm_engineering@127.0.0.1:27017
• Database: twin
• Username: llm_engineering / Password: llm_engineering

Cloud infrastructure

First, reinstall dependencies with the AWS group:

poetry install --with aws

AWS SageMaker

Create a restricted SageMaker IAM user:

poetry poe create-sagemaker-role

Create the SageMaker execution role:

poetry poe create-sagemaker-execution-role

Add the generated AWS_ARN_ROLE to your .env file, then update AWS_ACCESS_KEY and AWS_SECRET_KEY with the values from sagemaker_user_credentials.json.

Training

poetry poe run-training-pipeline
poetry poe run-evaluation-pipeline

Results are visible in Comet ML. Evaluation outputs are saved as *-results datasets on Hugging Face.

Inference

# Deploy endpoint
poetry poe deploy-inference-endpoint

# Test endpoint
poetry poe test-sagemaker-endpoint

# Delete endpoint
poetry poe delete-inference-endpoint

GitHub Actions CI/CD

Fork the repository and add the following as Actions secrets:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_ECR_NAME
• AWS_REGION

---

⚡ Pipelines

All ML pipelines are orchestrated by ZenML. To inspect runs:

1. Open the ZenML dashboard
2. Go to Pipelines
3. Select a pipeline → select a run → click a step or artifact for details

Data pipelines

# Collect raw data
poetry poe run-digital-data-etl

# Feature engineering
poetry poe run-feature-engineering-pipeline

# Generate instruct dataset
poetry poe run-generate-instruct-datasets-pipeline

# Generate preference dataset
poetry poe run-generate-preference-datasets-pipeline

# Run all of the above in one pipeline
poetry poe run-end-to-end-data-pipeline

Chrome required for LinkedIn and Medium crawlers (Selenium). If unavailable, use the Docker option:

poetry poe run-docker-end-to-end-data-pipeline

To add more data sources, edit configs/digital_data_etl_[author_name].yaml and add URLs to the links field.

Utility pipelines

# Export data warehouse to JSON
poetry poe run-export-data-warehouse-to-json

# Import data warehouse from JSON
poetry poe run-import-data-warehouse-from-json

# Export ZenML artifacts to JSON
poetry poe run-export-artifact-to-json-pipeline

The export pipeline outputs the following to the output/ folder:

• cleaned_documents.json
• instruct_datasets.json
• preference_datasets.json
• raw_documents.json

Configurable via configs/export_artifact_to_json.yaml.

Training pipelines

Requires AWS SageMaker to be configured (poetry install --with aws + AWS env vars).

poetry poe run-training-pipeline
poetry poe run-evaluation-pipeline

Inference pipelines

# Test RAG retrieval
poetry poe call-rag-retrieval-module

# Start REST API server
poetry poe run-inference-ml-service

# Call REST API with a test query
poetry poe call-inference-ml-service

Monitor prompt traces on Opik.

Inference requires the LLM microservice deployed to AWS SageMaker.

Linting & formatting

poetry poe lint-check
poetry poe lint-fix

poetry poe format-check
poetry poe format-fix

poetry poe gitleaks-check

Tests

poetry poe test

---

🏃 Run end-to-end

Data

poetry poe run-digital-data-etl
poetry poe run-feature-engineering-pipeline
poetry poe run-generate-instruct-datasets-pipeline
poetry poe run-generate-preference-datasets-pipeline

Training

Requires AWS SageMaker setup.

# SFT fine-tuning
poetry poe run-training-pipeline

# DPO fine-tuning: set finetuning_type=dpo in configs/training.yaml, then:
poetry poe run-training-pipeline

# Evaluate
poetry poe run-evaluation-pipeline

Inference

Requires AWS SageMaker setup.

poetry poe call-rag-retrieval-module
poetry poe deploy-inference-endpoint
poetry poe test-sagemaker-endpoint
poetry poe run-inference-ml-service
poetry poe call-inference-ml-service

---