Skip to content

RUNBOOK.md

Latest commit

 

History

History
420 lines (307 loc) · 14.1 KB

RUNBOOK.md

File metadata and controls

420 lines (307 loc) · 14.1 KB
Raw

RUNBOOK — bookstack-api

Service: BookStack Knowledge Base Production URL: https://docs.basicconsulting.no Oncall: Alem Basic (alem@alai.no, +47 404 74 251) Last reviewed: 2026-04-29 Next review trigger: After first P0/P1 incident, or 2026-07-01 (quarterly)

1. Service Overview

BookStack is ALAI's central knowledge base, hosting runbooks, architecture docs, ADRs, client contract summaries, and agent training materials. It runs on an Azure VM (docs.basicconsulting.no) with a MySQL backend. Local development uses Docker Compose.

Critical dependencies: Azure VM uptime, MySQL database, filesystem storage for attachments.

Users: ALAI agents (Skillforge, John), CEO, future client documentation (if shared externally).

SLO: Internal tool — 90% availability, P95 latency <5s. No paid customers. Downtime impacts ALAI operations (agents cannot query/create docs), but does NOT block client-facing services.

Reference: See BUILD-BLUEPRINT.md for architecture, data model, tech stack.

2. Dependencies

Required Services

Service Purpose Impact if down
Azure VM Hosting platform Total outage — site unreachable
MySQL 8.4 Database Total outage — cannot read/write pages
Filesystem Image/attachment storage Partial — new uploads fail, existing pages load
DNS (docs.basicconsulting.no) Domain resolution Total outage — site unreachable

External Integrations (Optional)

Service Purpose Impact if down
SMTP server Email notifications Degraded — no email alerts for page updates
OAuth providers (Google, Azure, etc.) SSO login Degraded — SSO users cannot login; local users unaffected

Credentials

  • Azure VM SSH: TBD — retrieve from Vaultwarden bookstack-azure-vm item (CEO must confirm access)
  • MySQL root password: TBD — retrieve from Azure VM /var/www/bookstack/.env (DB_PASSWORD)
  • BookStack API token: Vaultwarden item bookstack-api-token (token format: {id}:{secret})

3. Deploy Procedure

Local Development Deploy

cd /Users/makinja/projects/bookstack-api

# Start containers
docker-compose up -d

# Wait for MySQL init
sleep 10

# Install dependencies (first time only)
docker-compose exec app composer install
docker-compose exec node npm install

# Run migrations
docker-compose exec app php artisan migrate --force

# Build assets
docker-compose exec node npm run build

# Access: http://localhost:8080
# Default admin: admin@admin.com / password

Production Deploy (Azure VM)

⚠️ PREREQUISITE: Azure VM SSH access (credentials TBD — escalate to CEO).

ZAKON PI2 Pre-Flight:

# 1. Verify site is currently accessible
curl -sI https://docs.basicconsulting.no | head -5
# Expected: HTTP/2 200

# 2. Check recent commits
cd /Users/makinja/projects/bookstack-api
git log -5 --oneline

# 3. Backup database (on Azure VM)
ssh bookstack@docs.basicconsulting.no "mysqldump -u root -p bookstack > /tmp/bookstack-backup-$(date +%Y%m%d-%H%M%S).sql"

Deploy Steps:

# SSH to Azure VM
ssh bookstack@docs.basicconsulting.no

# Navigate to app directory
cd /var/www/bookstack  # TBD — confirm actual path

# Pull latest code
git fetch origin
git checkout release  # BookStack stable branch
git pull origin release

# Install PHP dependencies (production mode)
composer install --no-dev --optimize-autoloader

# Build frontend assets
npm run production

# Run migrations (if schema changed)
php artisan migrate --force

# Clear Laravel cache
php artisan cache:clear
php artisan config:cache
php artisan route:cache
php artisan view:cache

# Restart PHP-FPM (if needed)
sudo systemctl restart php8.2-fpm

ZAKON PI2 Post-Deploy Verification:

# 1. Verify site responds
curl -sI https://docs.basicconsulting.no | grep "200"

# 2. Test API endpoint
curl -s -H "Authorization: Token YOUR_TOKEN_HERE" https://docs.basicconsulting.no/api/shelves | jq '.data | length'
# Expected: non-zero number (shelf count)

# 3. Check error logs (last 20 lines)
ssh bookstack@docs.basicconsulting.no "tail -20 /var/www/bookstack/storage/logs/laravel.log"
# Expected: No new CRITICAL or ERROR entries since deploy timestamp

# 4. Visual smoke test
# Open https://docs.basicconsulting.no in browser
# Verify: login page loads, can login, can view a page

Evidence: Capture pre-flight + post-deploy output to /tmp/bookstack-deploy-YYYYMMDD.txt.

4. Common Failure Modes

Alert / Symptom Diagnosis Fix Escalation
Site unreachable (HTTP timeout) Azure VM down or network issue 1. Ping VM: ping docs.basicconsulting.no
2. Check Azure portal for VM status
3. Restart VM if stopped		 CEO → Azure support (if VM unresponsive)
500 Internal Server Error Laravel crash (syntax error, missing env var, DB connection failure) 1. SSH to VM
2. Check logs: tail -50 /var/www/bookstack/storage/logs/laravel.log
3. Fix error (missing .env var, DB credentials, etc.)
4. Restart PHP-FPM: sudo systemctl restart php8.2-fpm		 Codecraft (if code fix needed)
Database connection refused MySQL service stopped 1. SSH to VM
2. Check MySQL: sudo systemctl status mysql
3. Start: sudo systemctl start mysql
4. Verify: mysql -u root -p -e "SELECT 1;"		 CEO → Azure support (if MySQL won't start)
Pages load, but images missing Filesystem full or /storage/uploads permissions wrong 1. Check disk: df -h
2. Check permissions: ls -ld /var/www/bookstack/storage/uploads
3. Fix perms: chown -R www-data:www-data /var/www/bookstack/storage		 CEO (if disk expansion needed)
Login fails (all users) Session storage issue or .env APP_KEY changed 1. Check .env file: grep APP_KEY /var/www/bookstack/.env
2. Regenerate sessions: php artisan session:clear
3. Restart PHP-FPM		 Securion (if session hijacking suspected)
API returns 401 Unauthorized Invalid or expired API token 1. Verify token in Vaultwarden: bw get item "bookstack-api-token"
2. Regenerate token via BookStack UI: Settings → API Tokens
3. Update Vaultwarden item with new token		 Securion (token rotation)
Slow page loads (>10s) Database query bottleneck or full-text search index corruption 1. Check MySQL slow query log
2. Rebuild search index: php artisan bookstack:regenerate-search
3. Optimize DB: php artisan bookstack:cleanup-images		 Codecraft (query optimization)

5. Rollback Procedure

Scenario: New deploy caused critical bug (500 errors, data loss, auth broken).

Rollback steps (< 5 minutes):

# 1. SSH to Azure VM
ssh bookstack@docs.basicconsulting.no

cd /var/www/bookstack

# 2. Identify last known-good commit
git log --oneline -10
# Note the commit hash BEFORE the bad deploy

# 3. Revert to previous commit
git checkout COMMIT_HASH_HERE

# 4. Reinstall dependencies (in case composer.lock changed)
composer install --no-dev --optimize-autoloader

# 5. Rebuild assets
npm run production

# 6. Rollback database (if migration was run)
php artisan migrate:rollback --step=1

# 7. Clear cache
php artisan cache:clear
php artisan config:cache

# 8. Restart PHP-FPM
sudo systemctl restart php8.2-fpm

# 9. Verify site loads
curl -sI https://docs.basicconsulting.no | grep "200"

Database restore (if data corruption):

# Find latest backup
ls -lh /tmp/bookstack-backup-*.sql | tail -5

# Restore
mysql -u root -p bookstack < /tmp/bookstack-backup-YYYYMMDD-HHMMSS.sql

# Restart PHP-FPM
sudo systemctl restart php8.2-fpm

Escalation: If rollback fails, escalate to CEO (alem@alai.no). Consider migrating to new Azure VM as last resort.

6. Database Access

Local Dev (Docker)

# Access MySQL container
docker-compose exec db mysql -u bookstack-test -pbookstack-test bookstack-dev

# Backup local DB
docker-compose exec db mysqldump -u bookstack-test -pbookstack-test bookstack-dev > /tmp/bookstack-local-backup.sql

Production (Azure VM)

# SSH to VM
ssh bookstack@docs.basicconsulting.no

# Access MySQL
mysql -u root -p bookstack

# Common queries:
# List all shelves:
SELECT id, name, slug FROM shelves;

# List recent pages:
SELECT id, name, updated_at FROM pages ORDER BY updated_at DESC LIMIT 10;

# Search pages by keyword:
SELECT id, name FROM pages WHERE name LIKE '%runbook%' OR html LIKE '%runbook%';

Backup verification:

# Check latest backup exists
ls -lh /tmp/bookstack-backup-*.sql | tail -1

# Verify backup is valid SQL
head -20 /tmp/bookstack-backup-YYYYMMDD-HHMMSS.sql
# Expected: MySQL dump header, CREATE TABLE statements

Automated backups: TBD — no cron job observed. ESCALATE to Proveo for backup automation (MC task).

7. Log Access

Local Dev (Docker)

# Laravel logs (app container)
docker-compose exec app tail -f /app/storage/logs/laravel.log

# Web server logs (nginx or Apache inside app container)
docker-compose exec app tail -f /var/log/apache2/error.log  # TBD — confirm web server

Production (Azure VM)

# SSH to VM
ssh bookstack@docs.basicconsulting.no

# Laravel application logs
tail -f /var/www/bookstack/storage/logs/laravel.log

# PHP-FPM logs
sudo tail -f /var/log/php8.2-fpm.log

# Web server logs (nginx or Apache)
sudo tail -f /var/log/nginx/error.log  # TBD — confirm web server type
sudo tail -f /var/log/apache2/error.log

Log rotation: TBD — verify if logrotate is configured for Laravel logs.

8. Escalation Contacts

Role Contact When to escalate
Primary oncall Alem Basic (alem@alai.no, +47 404 74 251) All P0/P1 incidents
Fallback (ALAI) Edita Selimovic (edita@alai.no, phone TBD) If Alem unreachable >30 min
Code fixes Codecraft agent (via John orchestrator) 500 errors, bugs, query optimization
Security issues Securion agent (via John) Unauthorized access, token compromise
Infrastructure CEO → Azure support VM unresponsive, disk full, network down

P0 definition: Production BookStack unreachable for >5 minutes, affecting ALAI operations (agents cannot query docs).

P1 definition: Degraded service (slow, partial outage), login broken for some users, API endpoints failing.

9. Admin Procedures

Create New User (Web UI)

  1. Login as admin (admin@admin.com or via SSO)
  2. Settings → Users → Create New User
  3. Enter name, email, password
  4. Assign role (Admin, Editor, Viewer)
  5. Save

Create New User (CLI)

ssh bookstack@docs.basicconsulting.no
cd /var/www/bookstack
php artisan bookstack:create-admin --email=newadmin@alai.no --name="New Admin"
# Prompts for password

Rotate API Token

  1. Login to BookStack UI
  2. Settings → Access Settings → API Tokens
  3. Find old token, click "Regenerate"
  4. Copy new token
  5. Update Vaultwarden: bw edit item bookstack-api-token
  6. Paste new token in notes field
  7. Update any scripts using old token (e.g., Skillforge agent)

Clear Cached Data

ssh bookstack@docs.basicconsulting.no
cd /var/www/bookstack
php artisan cache:clear       # Application cache
php artisan config:cache      # Rebuild config cache
php artisan route:cache       # Rebuild route cache
php artisan view:cache        # Rebuild Blade template cache

Regenerate Search Index

ssh bookstack@docs.basicconsulting.no
cd /var/www/bookstack
php artisan bookstack:regenerate-search
# Rebuilds fulltext index for all pages (may take 1-5 min for large wiki)

Cleanup Old Revisions (Disk Space)

ssh bookstack@docs.basicconsulting.no
cd /var/www/bookstack
php artisan bookstack:cleanup-images  # Remove orphaned images
# TBD — check for revision cleanup command (not in standard BookStack)

10. Monitoring & Dashboards

Current state: NO monitoring dashboards (BetterStack or Grafana not configured for BookStack).

Recommended (future MC task):

  1. Uptime monitoring: BetterStack HTTP check on https://docs.basicconsulting.no (every 5 min)
  2. Alert destination: Email to alem@alai.no + Telegram alert via PA layer
  3. Metrics to track:
    • HTTP response time (P95 latency)
    • Database query time
    • Disk usage on Azure VM
    • Error rate (Laravel logs ERROR/CRITICAL count per hour)

Manual health check (interim solution):

# From any machine
curl -sI https://docs.basicconsulting.no | grep "200"
# Expected: HTTP/2 200 OK

# Check latency
time curl -s https://docs.basicconsulting.no > /dev/null
# Expected: <2 seconds

ESCALATE to Proveo: Set up BetterStack monitoring for BookStack (MC task, priority M).

11. Known Issues & Workarounds

Issue: Azure VM credentials unknown

Impact: Cannot deploy, cannot rollback, cannot access logs. Workaround: Escalate to CEO (alem@alai.no) for Azure VM SSH access. Add credentials to Vaultwarden under bookstack-azure-vm. Permanent fix: Document Azure VM access in this runbook after CEO provides credentials.

Issue: No automated database backups

Impact: Risk of data loss if VM fails. Workaround: Manual backup before each deploy (see Section 3). Permanent fix: Add cron job on Azure VM to run mysqldump daily, upload to Azure Blob Storage or Backblaze B2. MC task TBD.

Issue: No CI/CD pipeline

Impact: Deploys are manual, error-prone, slow. Workaround: Follow this runbook step-by-step, capture evidence. Permanent fix: Create GitHub Actions workflow or Azure DevOps pipeline (post-2026-05-04, low priority).

12. Next Review

Trigger events for runbook update:

  1. First P0 or P1 incident (capture postmortem, update failure modes)
  2. Azure VM credentials obtained (update Section 2, Section 3, Section 6)
  3. Monitoring dashboards deployed (update Section 10)
  4. Quarterly review: 2026-07-01

Owner: Skillforge (runbook maintenance) + Proveo (validation after incidents)

Sources: BUILD-BLUEPRINT.md, ALAI-PROJECT-BLUEPRINT.md Section 4.7, docker-compose.yml, CLAUDE.md

Evidence of gaps: Multiple TBD items flagged. This is a first-draft runbook based on code archaeology. Requires real-world incident response to validate and refine.