Session 12: split the bot into MLflow, classifier, LLM and orchestrator services

[mkhlndrv]

Session 12 homework: turned the quick-start into a small microservice setup. The
whole thing starts with docker compose up --build.

Services

• mlflow — tracking server and model registry (SQLite + a local artifact store).
• classifier — FastAPI service. On first boot it registers the zero-shot bot
  detector in MLflow as youarebot-classifier@champion, then loads that champion
  from the registry and serves POST /predict.
• llm — llama.cpp OpenAI-compatible server (Qwen2.5-0.5B) on port 11434,
  exposing POST /v1/chat/completions.
• orchestrator — the public gateway. It only forwards: /predict to the
  classifier and /get_message to the llm. It runs no model itself.

Each service is its own folder under services/ with its own Dockerfile and
requirements. Inside the compose network they call each other by service name
(classifier:8000, llm:11434, mlflow:5000), never localhost.

How to run and test

docker compose up --build
# /predict -> probability in [0,1] from the classifier
curl -s http://localhost:8000/predict -H "Content-Type: application/json" \
  -d '{"text":"hello there","dialog_id":"<uuid4>","id":"<uuid4>","participant_index":0}'
# /get_message -> real reply from the LLM
curl -s http://localhost:8000/get_message -H "Content-Type: application/json" \
  -d '{"dialog_id":"<uuid4>","last_msg_text":"Hi!","last_message_id":"<uuid4>"}'

Full instructions and example responses are in the README.

Checks

I ran the whole stack locally and confirmed:

• /predict returns a real probability in [0,1] (0.28 for a human-like line,
  0.75 for an "as an AI language model" line, so the model actually discriminates).
• /get_message returns a real LLM answer.
• The orchestrator logs show it forwarding to classifier:8000 and llm:11434,
  it does not run any model.
• MLflow ends up with the youarebot-classifier experiment and the champion
  registered model.

Notes

• First boot is slow because it downloads the classifier model (~560 MB) and the
  GGUF weights (~400 MB).
• The MLflow UI is published on host port 5001 (host 5000 is taken by AirPlay
  on macOS); internally it is still mlflow:5000.
• No secrets are committed. I also removed the old portforward_key (an SSH
  private key that was in the repo) and the SSH-tunnel run scripts that used it.
  Since it was committed before, that key should be rotated regardless.