AI Helpdesk Automation

Production-oriented demo of an AI helpdesk for a visual-effects studio. A local Qwen 3 8B routing agent classifies free-form support requests, n8n orchestrates the approved scenario, and a FastAPI action service performs a small allowlisted set of infrastructure operations through SSH, LDAP, and the studio account store.

The system was developed as a bachelor's thesis and introduced into the production workflow of a VFX studio. This repository publishes the author's components, sanitized integration artifacts, reproducible evaluation, and a safe standalone demo. The VFX studio's Telegram and Django systems remain private and are represented by documented contracts and mocks.

Results

Metric	Result	Evidence type
Current Qwen 3 routing accuracy	192/200 (96.0%)	Frozen synthetic holdout
Current Qwen 3 macro F1	0.959	Frozen synthetic holdout
Qwen 3.5 candidate accuracy	167/200 (83.5%)	Same frozen holdout
Unsafe-action false positives	7 current / 33 candidate	Same frozen holdout
Second-annotator label agreement	98/100 (98.0%), κ=0.9733	Blind stratified holdout subset
Original public Qwen result	38/40 (95.0%)	Development/evaluation set
Historical routing accuracy	37/40 (92.5%)	Separate internal thesis set

The locked holdout contains 200 synthetic requests, 50 per route. The first production-prompt run is primary; no best-of-three result is selected. A separate post-hoc stage analyzes output-contract failures, prompt components, temperature sensitivity and paraphrase consistency without changing the frozen data, labels or production prompt. The Qwen 3.5 candidate is evaluation-only and was not promoted.

A second annotator manually reviewed a blind fixed 100-case stratified subset and agreed with the frozen labels on 98/100 cases (Cohen's κ 0.9733, macro F1 0.9802). The annotator worked without expected labels, model predictions, correctness flags, metadata or provenance, and was not involved in dataset authoring, prompt design or model runs. The two disagreements remain unadjudicated and neither the frozen labels nor prior model results were changed. See the human-review report.

The original development/evaluation regression remains 38/40 (95.0%), macro F1 0.950, versus 26/40 (65.0%), macro F1 0.669 for the author-written keyword baseline. It is reported separately and is not pooled with the frozen holdout.

The datasets, raw predictions, hashes, statistics and runners are versioned under eval/. Start with the frozen holdout report and the post-hoc diagnostic report. Reproduction and evidence roles are documented in docs/reproducibility.md.

Operational impact

The following figures are operational expert estimates documented in the thesis, not controlled-experiment measurements:

• about 100 helpdesk requests per month, including about 40 routine cases;
• 32–34 routine cases handled automatically;
• 15–20 minutes of manual handling per routine case reduced to about 30–45 seconds;
• about 8–11 engineer-hours saved directly per month;
• about 18–26 hours of total monthly effect when estimated avoided downtime is included.

The deployment itself is confirmed by a formal implementation act. The wider 18–26-hour estimate depends partly on assessed avoided downtime and should not be read as directly measured engineer time.

Architecture

flowchart LR
    U["VFX studio employee"] --> B["Telegram bot\nprivate external system"]
    B --> D["Django employee context\nprivate external system"]
    B --> N["n8n workflow"]
    D -->|"employee and workstation context"| N
    N --> O["Ollama / Qwen 3 8B\nrouting agent"]
    O -->|"one allowlisted tool or abstain"| N
    N --> A["FastAPI action service"]
    A --> L["LDAP directory"]
    A --> J["Account store"]
    A --> S["Workstations over SSH"]
    A --> C["Client configuration store"]
    A -->|"structured result"| N
    N -->|"solved or escalated"| B

Loading

The language model never generates shell commands. It can select only one of three declared tools, while the backend owns fixed command templates. Ambiguous or unsupported requests are escalated to an engineer.

Detailed component and trust-boundary documentation is in docs/architecture.md.

Published scope

• FastAPI action service and CLI;
• sanitized n8n workflow with no production URLs or secrets;
• Ollama model configuration and versioned routing prompt;
• JSON contracts for external employee and bot context;
• automated unit, contract, security, and evaluation tests;
• safe demo adapters that do not access VFX studio infrastructure;
• Docker packaging and locked Python dependencies.

Not published:

• the VFX studio Telegram bot;
• the VFX studio administration portal;
• employee data, credentials, host inventory, internal addresses, and VPN configuration files.

Those systems predate this component and contain unrelated proprietary logic. Their exact integration boundary is documented in docs/integrations.md.

Quick start for development

Requirements: Python 3.12 and uv.

cd ai-helpdesk-automation
cp .env.example .env
uv sync --locked --extra dev
uv run pytest -q --cov

The production API expects LDAP, SSH, and storage settings from .env. Never commit real values. The standalone demo uses isolated mock adapters and is documented in docs/demo.md.

The production compose file uses ./runtime only as a safe local default. Deployments should replace that mount with the approved account store and secret/configuration mounts for their environment.

API surface

Method	Path	Purpose
POST	/activate	Create an account in the account store and LDAP
POST	/deactivate	Remove the account-store entry and lock LDAP
POST	/reactivate	Restore an existing account
POST	/nomachine/start	Start the remote-access service
POST	/nomachine/stop	Stop the remote-access service
POST	/nomachine/restart	Restart the remote-access service
POST	/nomachine/status	Read the remote-access service state
POST	/network-tunnel/stop	Stop the protected-network client
POST	/network-tunnel/status	Read the protected-network client state
POST	/host/restart	Schedule a workstation reboot
GET	/client-config/{workstation}	Download an allowlisted client config

FastAPI exposes the live OpenAPI contract at /docs in development. Production deployments must place the API behind authenticated network access and restrict interactive documentation.

Quality gates

The CI pipeline runs:

• locked dependency installation;
• Ruff formatting and linting;
• mypy type checking;
• pytest with branch coverage (minimum 85%);
• dependency vulnerability audit;
• wheel/source-distribution build;
• Docker image build.

Repository test verification on 26 July 2026: 169 tests passed, 87.72% combined line/branch coverage. The previously verified Ruff, mypy, dependency audit, packaging, Compose, and container-image gates remain unchanged.

This project was built with heavy use of AI coding agents. The division of labour, the review model, and the controls that make agent output checkable are documented in docs/ai-assisted-development.md.

AI evaluation

The original 40-case set remains a development/evaluation regression suite. The headline comparison uses a separately frozen 200-case synthetic holdout with preassigned labels, provenance and overlapping difficulty tags. Reports include per-class metrics, confusion matrices, Wilson intervals, paired McNemar tests, stability, latency, safety errors and full prediction records.

The exploratory post-hoc package adds bootstrap intervals, output-contract analysis, a fixed difficult subset and a 100-case paraphrase challenge. These diagnostics are explicitly separated from confirmatory holdout results.

See eval/README.md for dataset roles and eval/posthoc/README.md for the extended evaluation.

Security model

This service can perform privileged infrastructure actions. The production boundary therefore requires authentication, role checks, an explicit host inventory, pinned SSH host keys, encrypted LDAP transport, and structured audit events. The public demo cannot contact real LDAP or SSH targets.

Threats, controls, residual risks, and deployment requirements are documented in docs/security.md.

Limitations

• A workstation that is fully unreachable over the network cannot be rebooted through SSH and is escalated to an engineer.
• Only explicitly approved scenarios are automated; unsupported and ambiguous requests require human diagnosis.
• The routing datasets are synthetic and domain-specific. Reported accuracy should not be generalized beyond this workflow or treated as direct production accuracy.
• Production integrations are intentionally not reproducible outside the VFX studio; the public demo validates contracts and orchestration with mocks.

Author contribution

The author designed and implemented the action service, LDAP/account-store and SSH integrations, routing prompt, n8n scenario, testing methodology, production integration, and operational evaluation. The pre-existing VFX studio Telegram bot and Django portal are external systems and are not claimed as the author's work.