Skip to content

COMMANDS.md

Latest commit

 

History

History
1160 lines (864 loc) · 28.2 KB

COMMANDS.md

File metadata and controls

1160 lines (864 loc) · 28.2 KB

LinTO Deploy - Command Reference

This document provides comprehensive documentation for all linto CLI commands with real-world usage scenarios.

Quick Reference

Command Description
linto wizard Interactive profile creation wizard
linto list List all deployment profiles
linto show <profile> Display profile configuration details
linto render <profile> Generate Helm values files without deploying
linto deploy <profile> Generate values and deploy to Kubernetes cluster
linto status <profile> Show deployment status with resource metrics
linto logs <profile> <service> View service logs
linto redeploy <profile> Force restart deployments
linto destroy <profile> Remove deployment from cluster
linto version Show version information
linto profile set-kubeconfig <profile> <file> Set kubeconfig for a profile
linto kubeconfig export <profile> Export kubeconfig for manual kubectl/helm
linto exec <profile> <service> Interactive shell access to pods
linto port-forward <profile> <service> Port forwarding to services (alias: pf)
linto backup <profile> Database backup (MongoDB/PostgreSQL)

Cluster Access

Profiles are self-contained and include cluster connection settings. Use linto kubeconfig export if you need to run kubectl/helm commands manually:

# Export to file
linto kubeconfig export prod -o /tmp/prod.yaml
KUBECONFIG=/tmp/prod.yaml kubectl get pods -n linto

# Or merge into ~/.kube/config
linto kubeconfig export prod --merge
kubectl config use-context prod

Installation & Setup

Prerequisites

  1. kubectl - Kubernetes command-line tool

    kubectl version --client

  2. helm (v3.x) - Kubernetes package manager

    helm version

  3. Python 3.11+

    python3 --version

  4. Kubernetes cluster access

    kubectl cluster-info
kubectl get nodes

Installation

# Clone the repository
git clone https://github.com/linto-ai/linto-deploy.git
cd linto-deploy

# Install with uv (recommended)
uv sync
uv run linto --help

# Or install with pip
python -m venv .venv
source .venv/bin/activate
pip install -e .
linto --help

Shell Completion

Enable tab completion for better productivity:

# Bash
linto --install-completion bash
source ~/.bashrc

# Zsh
linto --install-completion zsh

# Fish
linto --install-completion fish

Scenarios

Scenario 1: First Deployment (New User)

A step-by-step guide for deploying LinTO for the first time.

Step 1: Create a profile with the wizard

linto wizard

The wizard will guide you through:

  • Profile name (e.g., production, dev, staging)
  • Domain configuration (e.g., linto.example.com)
  • Kubernetes namespace
  • Storage configuration
  • Service selection (Studio, STT, Live Session, LLM)
  • GPU configuration
  • TLS settings
  • Admin account

Step 2: Review generated configuration

linto render production

This creates Helm values files in .linto/render/k3s/production/values/.

Review the generated files:

ls -la .linto/render/k3s/production/values/
# studio-values.yaml
# stt-values.yaml
# live-values.yaml  (if Live Session enabled)
# llm-values.yaml   (if LLM enabled)

Step 3: Deploy to cluster

linto deploy production

Step 4: Verify deployment

linto status production

Expected output:

Profile: production (k3s)
Domain: linto.example.com
Namespace: linto

                          Services
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Service                   ┃ Status  ┃ CPU       ┃ URL                             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ pod/studio-api-xxx        │ Running │ 50m/2     │ https://linto.example.com/cm-api│
│ pod/studio-frontend-xxx   │ Running │ 10m/500m  │ https://linto.example.com/      │
│ pod/studio-websocket-xxx  │ Running │ 5m/500m   │ https://linto.example.com/ws    │
└───────────────────────────┴─────────┴───────────┴─────────────────────────────────┘

Scenario 2: Daily Operations

Commands for routine monitoring and maintenance.

Check deployment health

# Full status with resource metrics
linto status production

# Compact view (hide CPU/Memory/GPU columns)
linto status production --compact

# Continuous monitoring (refreshes every 5 seconds)
linto status production --follow

# Custom refresh interval (10 seconds)
linto status production --follow --interval 10

View service logs

# View last 100 lines from studio-api
linto logs production studio-api

# Stream logs in real-time
linto logs production studio-api --follow

# View last 500 lines
linto logs production studio-api --tail 500

Scenario 3: Updating a Deployment

When you need to modify configuration or update images.

Option A: Update configuration

  1. Edit the profile JSON directly:

    # Profile location
cat .linto/profiles/production.json

  2. Or re-run the wizard (creates new profile):

    linto wizard

  3. Regenerate Helm values:

    linto render production

  4. Apply changes:

    linto deploy production

Option B: Force image update (for latest-* tags)

When using floating tags like latest-unstable:

# Restart all deployments to pull latest images
linto redeploy production

# Restart only a specific chart
linto redeploy production linto-studio

Scenario 4: Managing Multiple Environments

Best practices for dev, staging, and production environments.

List all profiles

linto list

Output:

                    Profiles
┏━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓
┃ Name        ┃ Backend ┃ Domain                ┃ Services            ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩
│ dev         │ k3s     │ localhost             │ studio, stt         │
│ staging     │ k3s     │ staging.linto.ai      │ studio, stt, live   │
│ production  │ k3s     │ linto.example.com     │ studio, stt, live   │
└─────────────┴─────────┴───────────────────────┴─────────────────────┘

View profile details

linto show staging

Output:

                 Profile: staging
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Setting        ┃ Value                            ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Name           │ staging                          │
│ Backend        │ k3s                              │
│ Domain         │ staging.linto.ai                 │
│ Image Tag      │ latest-unstable                  │
│ TLS Mode       │ acme                             │
│ GPU Mode       │ exclusive                        │
│ GPU Count      │ 2                                │
│ Studio         │ enabled                          │
│ STT            │ enabled                          │
│ Live Session   │ enabled                          │
│ LLM            │ disabled                         │
│ Admin Email    │ admin@linto.ai                   │
│ Namespace      │ linto-staging                    │
│ Storage Class  │ local-path                       │
└────────────────┴──────────────────────────────────┘

Recommended profile naming

  • dev - Local development
  • staging - Pre-production testing
  • production or prod - Production environment
  • demo - Demonstration environment

Scenario 5: Troubleshooting

Diagnosing and fixing common issues.

Check pod states

linto status production

Look for:

  • Pending - Pod waiting for resources or scheduling
  • Error - Pod crashed or failed to start
  • CrashLoopBackOff - Pod repeatedly crashing

View detailed logs

# Check for errors in studio-api
linto logs production studio-api --tail 500

# Follow logs to see real-time errors
linto logs production studio-api --follow

Common issues

Pods stuck in Pending:

# Check node resources
kubectl describe nodes

# Check pod events
kubectl describe pod <pod-name> -n <namespace>

Image pull errors:

# Force restart to pull fresh images
linto redeploy production

# Check image pull status
kubectl describe pod <pod-name> -n <namespace> | grep -A5 "Events"

Certificate issues (ACME):

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

# Check certificate status
kubectl get certificates -n <namespace>

Scenario 6: Cleanup & Removal

Removing deployments safely.

Remove deployment (keep data)

linto destroy production

This will:

  1. Backup TLS certificates (if ACME)
  2. Uninstall all Helm releases
  3. Keep PVCs (persistent data)

Remove deployment with all data

# WARNING: This deletes all databases and cached models!
linto destroy production --volumes

Skip confirmation prompts

linto destroy production --force
linto destroy production --force --volumes

Remove generated files

linto destroy production --remove-files

Command Reference (Detailed)

linto wizard

Interactive profile creation wizard.

Usage:

linto wizard

Description:

Launches an interactive wizard that guides you through creating a deployment profile.

Wizard Flow:

  1. Profile name - Identifier for this deployment (alphanumeric, hyphens allowed)
  2. Domain - Hostname for accessing the deployment
  3. Backend - Currently only k3s is supported
  4. Namespace - Kubernetes namespace for deployment
  5. Storage configuration - StorageClass and host paths
  6. Service selection - Choose which services to enable:
    • Studio (Web UI, API, WebSocket)
    • STT (Speech-to-text with Whisper)
    • Live Session (Real-time streaming)
    • LLM (Language model integration)
  7. GPU configuration - None, exclusive, or time-slicing
  8. TLS configuration - Off, mkcert, ACME (Let's Encrypt), or custom
  9. Admin account - Super admin configuration
  10. Action - Save, render (generate files), or deploy

Example session:

LinTO Deployment Wizard
Configure your LinTO deployment interactively

Profile name [dev]: production
Domain [localhost]: linto.example.com

Deployment backend: Kubernetes (k3s)

Kubernetes namespace [linto]: linto-prod

Persistent Storage:
LinTO stores data in two locations:
  • Databases (MongoDB, PostgreSQL, Redis) → local disk on one node
  • Files (models, audio, exports) → shared storage (NFS recommended)

Database path on host [/home/ubuntu/linto/databases]:
Shared files path (NFS mount) [/data/linto]:
...

linto list

List all deployment profiles.

Usage:

linto list

Description:

Displays a table of all profiles stored in .linto/profiles/. Shows profile name, backend type, domain, and enabled services.

Output:

                    Profiles
┏━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Name        ┃ Backend ┃ Domain            ┃ Services        ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dev         │ k3s     │ localhost         │ studio, stt     │
│ production  │ k3s     │ linto.example.com │ studio, stt     │
└─────────────┴─────────┴───────────────────┴─────────────────┘

linto profile set-kubeconfig <profile> <file>

Set or update kubeconfig for an existing profile.

Usage:

linto profile set-kubeconfig <profile> <kubeconfig-file>

Arguments:

Argument Description
profile Name of the profile
kubeconfig-file Path to kubeconfig file (e.g., k3s.yaml)

Example:

# Copy kubeconfig from server
scp ubuntu@k3s-server:/etc/rancher/k3s/k3s.yaml ~/Downloads/

# Set it in profile
linto profile set-kubeconfig prod ~/Downloads/k3s.yaml

linto kubeconfig export <profile>

Export kubeconfig from a profile for manual kubectl/helm usage.

Usage:

linto kubeconfig export <profile> [OPTIONS]

Arguments:

Argument Description
profile Name of the profile

Options:

Option Short Description
--output -o Write to file instead of stdout
--merge Merge into ~/.kube/config

Examples:

# Output to stdout
linto kubeconfig export prod

# Write to file
linto kubeconfig export prod -o ~/.kube/linto-prod.yaml

# Merge into existing kubeconfig
linto kubeconfig export prod --merge

# Then use manually
kubectl config use-context prod
kubectl get pods -n linto

linto show <profile>

Display detailed profile configuration.

Usage:

linto show <profile>

Arguments:

Argument Description
profile Name of the profile to display

Example:

linto show production

linto render <profile>

Generate Helm values files without deploying.

Usage:

linto render <profile> [--output DIR]

Arguments:

Argument Description
profile Name of the profile

Options:

Option Short Description
--output -o Custom output directory

What it does:

  1. Loads the profile configuration
  2. Creates Helm values files for enabled services

Output location:

.linto/render/k3s/<profile>/values/
├── studio-values.yaml
├── stt-values.yaml
├── live-values.yaml    (if Live Session enabled)
└── llm-values.yaml     (if LLM enabled)

Example:

linto render production
# Output: .linto/render/k3s/production/values/

linto deploy <profile>

Deploy services to the Kubernetes cluster.

Usage:

linto deploy <profile> [--force]

Arguments:

Argument Description
profile Name of the profile to deploy

Options:

Option Description
--force Skip GPU capacity warnings

What it does:

  1. Validates prerequisites (kubectl, helm, cluster access)
  2. Ensures namespace exists (creates if needed)
  3. Installs cert-manager (if ACME TLS mode)
  4. Restores TLS certificates from backup (if available)
  5. Generates values files (if not present)
  6. Runs helm upgrade --install for each enabled chart

Example:

linto deploy production

# Sample output:
Installing/upgrading linto-studio...
linto-studio deployed successfully
Installing/upgrading linto-stt...
linto-stt deployed successfully
Deployment complete!
Access at: https://linto.example.com

linto status <profile>

Show deployment status with resource metrics.

Usage:

linto status <profile> [OPTIONS]

Arguments:

Argument Description
profile Name of the profile

Options:

Option Short Description
--compact -c Hide resource metrics (CPU, Memory, GPU)
--follow -f Continuously refresh status
--interval -i Refresh interval in seconds (default: 5)

Output columns:

  • Service name
  • Status (Running, Pending, Error)
  • CPU usage (current/limit)
  • Memory usage (current/limit)
  • GPU count (if applicable)
  • Endpoint URL

Examples:

# Full status
linto status production

# Compact view
linto status production --compact

# Continuous monitoring
linto status production --follow

# Custom refresh interval
linto status production --follow --interval 10

linto logs <profile> <service>

View logs from a specific service.

Usage:

linto logs <profile> <service> [OPTIONS]

Arguments:

Argument Description
profile Name of the profile
service Service/deployment name (tab-completion supported)

Options:

Option Short Description
--follow -f Stream logs in real-time
--tail -n Number of lines to show (default: 100)

Service names:

  • studio-api, studio-frontend, studio-websocket
  • stt-all-whisper-v3-turbo, stt-whisper-workers, diarization-pyannote
  • session-api, session-scheduler, session-transcriber
  • llm-gateway-api, llm-celery-worker, llm-gateway-frontend

Examples:

# View recent logs
linto logs production studio-api

# Follow logs
linto logs production studio-api --follow

# View last 500 lines
linto logs production studio-api --tail 500

linto redeploy <profile> [chart]

Force restart deployments (useful for latest-* image tags).

Usage:

linto redeploy <profile> [chart] [OPTIONS]

Arguments:

Argument Description
profile Name of the profile
chart Specific chart to redeploy (optional)

Charts:

  • linto-studio
  • linto-stt
  • linto-live
  • linto-llm

What it does:

  1. Runs kubectl rollout restart for deployments
  2. Kubernetes pulls fresh images (with imagePullPolicy: Always) and recreates pods

Examples:

# Restart all deployments
linto redeploy production

# Restart only Studio services
linto redeploy production linto-studio

linto destroy <profile>

Remove deployment from cluster.

Usage:

linto destroy <profile> [OPTIONS]

Arguments:

Argument Description
profile Name of the profile to destroy

Options:

Option Short Description
--force Skip confirmation prompt
--volumes -v Also remove PVCs (persistent data)
--remove-files -r Remove generated files

What it does:

  1. Backs up TLS certificates (if ACME mode)
  2. Runs helm uninstall for each chart
  3. Optionally removes PVCs
  4. Optionally removes generated files

Warning: --volumes will permanently delete:

  • Database contents
  • Model caches
  • Audio files

Examples:

# Remove deployment, keep data
linto destroy production

# Remove deployment and all data
linto destroy production --volumes

# Skip confirmation
linto destroy production --force

# Remove everything including generated files
linto destroy production --force --volumes --remove-files

linto version

Show version information.

Usage:

linto version

Example output:

linto-deploy version 0.1.0

Configuration Reference

Profile Fields

Field Type Default Description
name string dev Profile identifier (alphanumeric, hyphens)
domain string localhost Deployment domain
backend enum k3s Deployment backend (only k3s supported)
image_tag string latest-unstable Docker image tag
tls_mode enum mkcert TLS mode: off, mkcert, acme, custom
gpu_mode enum none GPU mode: none, exclusive, time-slicing
gpu_count int 1 Number of GPUs available
studio_enabled bool true Enable Studio services
stt_enabled bool true Enable STT services
live_session_enabled bool false Enable Live Session
llm_enabled bool false Enable LLM services
k3s_namespace string linto Kubernetes namespace
k3s_storage_class string null StorageClass for PVCs
super_admin_email string admin@linto.local Admin email address

TLS Modes

Mode Description
off No TLS (HTTP only, not recommended)
mkcert Local development certificates
acme Let's Encrypt certificates (production)
custom User-provided certificates

GPU Modes

Mode Description
none CPU only, no GPU
exclusive One GPU per pod (recommended for production)
time-slicing Share GPU across pods (development/testing)

Troubleshooting Guide

Common Issues

Pods stuck in Pending

Symptoms: Pods remain in Pending state.

Diagnosis:

# Check pod events
kubectl describe pod <pod-name> -n <namespace>

# Check node resources
kubectl describe nodes

# Check PVC status
kubectl get pvc -n <namespace>

Common causes:

  • Insufficient CPU/memory on nodes
  • PVC cannot be provisioned (wrong StorageClass)
  • Node selector doesn't match any nodes
  • GPU requested but not available

Operations Commands

Commands for debugging, development, and maintenance workflows.

linto exec <profile> <service>

Execute an interactive shell inside a running pod.

Usage:

linto exec <profile> <service> [OPTIONS]

Arguments:

Argument Description
profile Name of the profile
service Service/pod name (tab-completion supported)

Options:

Option Short Description
--container -c Container name for multi-container pods
--command Command to execute (default: /bin/sh)

What it does:

  • Opens an interactive shell session inside a running pod
  • Default shell is /bin/sh
  • Supports multi-container pods via --container flag
  • Non-interactive mode available with --command flag

Examples:

# Open shell in studio-api pod
linto exec production studio-api

# Open shell in specific container
linto exec production studio-api --container nginx

# Run a single command (non-interactive)
linto exec production studio-api --command "cat /etc/hosts"

# Access a specific pod by name
linto exec production pod/linto-studio-api-54ccb7d-xyz

linto port-forward <profile> <service>

Forward local ports to cluster services for debugging and development.

Aliases: pf

Usage:

linto port-forward <profile> <service> [port_spec] [OPTIONS]
linto pf <profile> <service> [port_spec] [OPTIONS]

Arguments:

Argument Description
profile Name of the profile
service Service/pod name (tab-completion supported)
port_spec Port specification: [local_port:]remote_port (optional)

Options:

Option Short Description
--address -a Local address to bind to (default: 127.0.0.1)

Port Specification:

  • 8080:80 - Forward local port 8080 to remote port 80
  • 8080 - Forward local port 8080 to remote port 8080
  • If omitted and service has a single port, auto-detect

What it does:

  • Creates a tunnel from your local machine to a service in the cluster
  • Displays connection info with clear URL
  • Gracefully shuts down on Ctrl+C

Examples:

# Forward local 8080 to studio-api port 80
linto port-forward production studio-api 8080:80

# Same as above using the alias
linto pf production studio-api 8080:80

# Forward to MongoDB (same port locally and remotely)
linto port-forward production studio-mongodb 27017

# Bind to all interfaces (accessible from other machines)
linto port-forward production studio-api 8080:80 --address 0.0.0.0

Output:

Forwarding localhost:8080 -> linto-studio-api:80
Press Ctrl+C to stop

linto backup <profile>

Backup MongoDB and PostgreSQL databases to local files.

Usage:

linto backup <profile> [OPTIONS]

Arguments:

Argument Description
profile Name of the profile

Options:

Option Short Description
--output -o Output directory (default: .linto/backups/<profile>/<timestamp>/)
--databases -d Comma-separated list of specific databases to backup

Auto-detected Databases:

Service Database Type
linto-studio-mongodb MongoDB
linto-stt-mongodb MongoDB
linto-live-postgres PostgreSQL
linto-llm-postgres PostgreSQL

What it does:

  1. Auto-detects deployed databases from the profile
  2. Creates compressed backups using mongodump and pg_dumpall
  3. Generates a manifest file with metadata
  4. Shows progress with Rich progress bar
  5. Displays summary with file sizes

Output Structure:

.linto/backups/<profile>/<timestamp>/
├── studio-mongodb.gz           # MongoDB dump (gzip compressed)
├── stt-mongodb.gz              # MongoDB dump (gzip compressed)
├── live-postgres.sql.gz        # PostgreSQL dump (gzip compressed)
├── llm-postgres.sql.gz         # PostgreSQL dump (gzip compressed)
└── manifest.json               # Metadata (timestamps, sizes, status)

Examples:

# Backup all databases for production profile
linto backup production

# Backup to a specific directory
linto backup production --output ./backups/2024-01-18

# Backup only specific databases
linto backup production --databases studio-mongodb,live-postgres

Exit Codes:

Code Description
0 All backups successful
1 Profile not found / no databases found / all backups failed
2 Partial failure (some backups succeeded, some failed)

Certificate not issued (ACME)

Symptoms: HTTPS not working, certificate errors in browser.

Diagnosis:

# Check certificate status
kubectl get certificates -n <namespace>

# Check cert-manager logs
kubectl logs -n cert-manager deployment/cert-manager

# Check certificate request
kubectl describe certificaterequest -n <namespace>

Common causes:

  • Domain DNS not pointing to cluster
  • Port 80 not accessible (HTTP-01 challenge)
  • Rate limit exceeded

Image pull errors

Symptoms: Pods in ImagePullBackOff state.

Diagnosis:

kubectl describe pod <pod-name> -n <namespace>

Solutions:

# Force restart to pull fresh images
linto redeploy <profile>

# Check image name and tag
kubectl get pod <pod-name> -n <namespace> -o yaml | grep image

Database connection errors

Symptoms: Services failing to connect to MongoDB/PostgreSQL.

Diagnosis:

# Check database pod status
linto status <profile>

# View database logs
linto logs <profile> studio-mongodb
linto logs <profile> session-postgres

Common causes:

  • Database pod not ready yet
  • Incorrect password in configuration
  • PVC issues

Services not accessible

Symptoms: Cannot access the web interface or API.

Diagnosis:

# Check ingress
kubectl get ingress -n <namespace>

# Check traefik logs (ingress controller)
kubectl logs -n kube-system -l app.kubernetes.io/name=traefik

# Verify DNS resolution
nslookup <domain>

Common causes:

  • DNS not configured
  • Firewall blocking ports 80/443
  • TLS certificate issues
  • Ingress misconfiguration