Skip to content

Latest commit

 

History

History
280 lines (181 loc) · 15.9 KB

FAQ.md

File metadata and controls

280 lines (181 loc) · 15.9 KB

Devops-Glue CD — FAQ

Custom Monitoring

Q: Test results show all values as "—" but the command outputs data?

Cause: field_key doesn't match the output headers.

Diagnosis: Expand "Raw Output" to see actual headers, check the diagnostic panel.

Solution:

  • For non-English locales, headers may be in the local language. Either change field_key to match, or prefix the command with LANG=C for English output: LANG=C df -h --type=ext4 --type=xfs
  • Values with unit suffixes (e.g. 20G, 300M) are automatically stripped — no manual handling needed

Q: My command outputs a single number. How do I configure it?

No metrics needed. Choose "Auto Detect" format. The system automatically extracts the first number. Example: ps aux | wc -l

Q: Output has no header row?

Prepend a header with echo:

echo "col1 col2 col3"; some-command-here

Q: Output rows vary (e.g. lscpu)?

Filter with grep/awk to get stable rows:

lscpu | grep -E '^CPU\(s\)|^Thread|^Core|^Socket'

Q: Can I monitor multiple metrics from one command?

Yes. Add multiple metrics under a single monitor item, each targeting a different column. E.g. for df -h, add Use%, Avail, and Used as separate metrics.


Database

Q: CD reports "not enabled" even though the database looks configured?

The enablement state is read only through Glue's contract endpoint GET /api/integration/services/cd; when it cannot be reached, CD fails closed as "contract unreachable" (503 errors.ci_contract_unreachable). Check CI_API_URL / CI_API_TOKEN, and that DB_PATH (SQLite) / DB_* (MySQL) points to the database holding the CD tables.

Q: Login fails with "CD is not enabled: enable Devops-CD in the Devops-Glue console under 'API Integration Services'"?

Whether CD is enabled is decided by the switch in the Devops-Glue console (CD reads it via the contract endpoint GET /api/integration/services/cd): enabled<>1 counts as not enabled, CD performs zero DDL, and login, every request carrying a session, the public webhook receiver and the WebShell handshake all return 503 (errors.cd_not_enabled).

When the contract endpoint cannot be reached (older Glue, network/auth failure), CD fails closed as "contract unreachable" (503 errors.ci_contract_unreachable) — never treated as "not enabled".

Fix: turn Devops-CD on in the Devops-Glue console under "API Integration Services → Enable" — CD picks it up within one contract cache window (CI_CONTRACT_TTL, 30s by default — no config change, no restart; set CI_CONTRACT_TTL=0 to make it immediate).

The message box shows only that one sentence: the login error box and the toast now share one path — both resolve error_key=errors.cd_not_enabled through the frontend i18n, so switching to EN really shows English instead of the backend's Chinese fallback; the full action guidance travels in the response's detail.hint and in the CD startup log instead of being appended to the message.

Q: Login fails with "CD is not deployed: the CD tables are incomplete (missing: ...)"?

This means CD is enabled, but the inventory CD_REQUIRED_TABLES (14 tables today; that list is the one source of truth, and one missing table means "not deployed") is incomplete. The fix depends on the provision value on the contract row:

  • provision='auto' (CI runs in auto-migrate mode): CD creates its tables automatically — restarting CD heals a missing table; that is exactly what the error message says;
  • provision='manual' (CI runs in manual schema mode): CD issues zero DDL (not even ALTER), so an administrator must import the DDL script to complete the inventory — it is idempotent, safe to re-run, and should be re-run after every upgrade to pick up new tables/columns:
mysql -u root -p devops_glue < database/init_mysql.sql

Also note: the gate covers login, requests carrying a session, the public webhook receiver and the WebShell handshake ("even if login were possible, no operation is allowed"). Disabling the integration service only freezes admission — it never deletes data (the CD tables and records stay as they are); to wipe everything you must DROP the cd_* tables yourself (which also discards session/approval data).

SQLite mode (dev / single instance): with provision='auto' the CD tables are created together with the DB file, so nothing is normally missing; if you manually DROP TABLEd one, the gate immediately answers 503 (it reports, it does not repair) and restarting the service re-creates them and heals it. If CI runs in manual mode while CD uses SQLite, CD does not create tables either — switch CI back to auto-migrate and re-enable, or have an operator create them (the message says exactly that and never tells you to import the MySQL script).

Don't confuse the other 503: errors.db_unavailable (the SQLite file is read-only / the disk is full / the file is locked, the DB file cannot be opened, MySQL hands out no connection) is an environment problem — check connectivity/configuration, the data directory's write permission and free disk space, and do not re-run the DDL script, which cannot fix it.

Q: SQLite mode — container deployment shows empty pages?

Mount the database directory as a shared volume in docker-compose.yml:

volumes:
  - config/data:/shared_data

And set .env: DB_PATH=/shared_data/data.db

Q: SQLite or MySQL for production?

MySQL is strongly recommended. SQLite experiences lock contention when CD and PHP API write concurrently.


Image Registry (Harbor / standard registry:2)

Q: Registry repository list is empty?

  1. Verify .env settings: REGISTRY_USER, REGISTRY_PASSWORD (the registry address and type are not local CD config — they come from the Glue contract endpoint)
  2. The registry_type delivered by the Glue contract (GET /api/integration/services/cd, Glue ≥ 2.11) selects the backend: harbor (Harbor API must be enabled, v1.x or v2.x, auto-detected) or registry (standard OCI Distribution / registry:2, /v2/ endpoints only); when the contract has no such field the Image Registry page shows "unknown" with a hint (common cause: the service credential does not have the Harbor read permission enabled in Glue "API Management"), while the backend keeps the harbor fallback
  3. When the contract has no registry_url (endpoint unreachable / Glue too old) CD reports "no image registry address" — make sure CI_API_URL / CI_API_TOKEN can reach that endpoint
  4. Click "Sync" in the "Image Registry" panel for the initial full sync
  5. Check connectivity: for Harbor run curl https://hub.xxx.com/api/v2.0/health on the CD server; for a standard registry use curl https://reg.xxx.com/v2/ (200 or 401/403 both count as reachable)

Q: Why are vulnerability scanning, image size and push time missing with a standard registry:2?

The standard OCI Distribution /v2/ API only exposes the tag list — no vulnerability scanning, image size or push time. With a registry type those features are unavailable, so the frontend hides the scan result column, the Report/Scan buttons and the Size/Push-time columns. This is expected behaviour, not a fault.

Q: Deployment fails with server gave HTTP response to HTTPS client / a plain-HTTP registry cannot be pulled?

When the contract registry_url is http://... (plain HTTP), dockerd tries HTTPS only for registries it does not allow, hence this error. The docker CLI has no "skip TLS verification" flag (--tls-verify belongs to podman / skopeo) and insecure-registries can only be configured on the target host's dockerd — CD cannot express it for the target.

  1. Default CD behaviour (checked automatically before the pull, reported in the deploy log): CD reads /etc/docker/daemon.json (plus daemon.json.d/*.json) on the target and checks whether that host[:port] is listed in insecure-registries (matching host / host:port / CIDR entries, same semantics as dockerd). If it is missing, the deploy is aborted with copy-pasteable fix commands, and CD does not fall back to Docker Hub (which would otherwise deploy a same-named public image as if it were yours).
  2. Run the commands from the output on the target host (root): back up and merge insecure-registries → systemctl reload docker (fall back to restart docker, which restarts that host's containers) → verify with docker pull <registry>/<repo>:<tag>.
  3. To let CD perform step 2 automatically, set REGISTRY_INSECURE_AUTO=true (default false: detect only, never touch the target).
  4. Better still: put TLS in front of the registry and switch the Glue contract registry_url to https://... — no CD configuration needed at all.
  5. The image name written to .env must stay scheme-less: host[:port]/repo:tag; a leading http:// makes docker fail with invalid reference format.
  6. The old REGISTRY_VERIFY_SSL (formerly HARBOR_VERIFY_SSL) switch is deprecated with no consumers left: it only affected CD's own TLS verification and never the target dockerd pull, so setting it has no effect.

For the K8S-family deployers (ArgoCD / FluxCD / native K8s) the pull is done by the node kubelet / containerd, so a plain-HTTP registry must be allowed in the node's containerd certs.d / insecure configuration; CD does not intervene there (the detection and auto-fix above cover only the compose remote deployer, where CD runs docker pull itself).

Q: Sync is too slow?

Full sync pulls all projects, repositories, and tags. Set REGISTRY_SYNC_INTERVAL (in minutes, default 30, 0 disables) for periodic incremental sync, or trigger sync manually with a project parameter.


SSH Connectivity

Q: Server connection timeout after adding?

  1. Verify SSH port (default 22) is open on the target
  2. Check firewall rules — is the CD server's IP whitelisted?
  3. Use Web Shell to test manually: ssh -p <port> <user>@<host>

Q: SSH key authentication fails?

  • Ensure the public key is in ~/.authorized_keys on the target
  • Key format must be PEM (-----BEGIN RSA PRIVATE KEY-----), not OpenSSH format
  • Check key permissions: chmod 600 ~/.ssh/id_rsa

Q: My server uses a non-standard SSH port?

Enter the actual port in server settings. The system uses this port for all SSH connections.


K8s Monitoring

Q: kubectl top nodes returns no data?

The K8s cluster needs metrics-server installed:

kubectl get deployment metrics-server -n kube-system

Install via metrics-server docs if missing.

Q: K8s 1.24+ cluster — docker stats unavailable?

Expected behavior. K8s 1.24 removed dockershim. Docker-level monitoring is unavailable, but pod-level monitoring (kubectl top pods) continues to work.


Deployment

Q: Docker Compose deployment fails with image pull error?

  1. Test Harbor access: docker login hub.xxx.com
  2. Verify the image tag was pushed to Harbor during CI build
  3. Confirm the project tag exists in the CI panel

Q: Deployment logs show garbled characters?

The backend auto-detects GBK/UTF-8. For unusual server encodings, add export LANG=en_US.UTF-8 to the deployment command.

Q: Container still running after "Stop"?

SSH mode is pure passthrough — stop commands are user-defined. Verify the "Stop Command" in server settings (e.g. docker stop <container> or docker compose down).

Q: CD reports "Connection refused" when connecting to CI under Docker Compose?

Cause: Incorrect CI_API_URL. When CD and CI are in the same docker-compose stack, containers communicate via service name + internal port, not the host-mapped port.

Correct configuration: Uncomment and edit in docker-compose.yml under cd-service → environment:

environment:
  # ── CI API integration (enable when pairing with PHP Devops-Glue) ──
  # Use the service name for container-to-container communication, NOT localhost
  CI_API_URL: http://devops-glue
  CI_API_TOKEN: dg_xxx      # API token (dg_ prefix, service account / third-party)
  # Fallback account login when token not set:
  # CI_ADMIN_USER: root
  # CI_ADMIN_PASS: your_root_password

Reminder: After changing the config, you must run docker-compose up -d to recreate the container. restart does not refresh environment variables.


Frontend

Q: npm run build produces a blank page?

Since v1.1.1 the SPA route re-reads static/index.html from disk on every request (HTML is never cached), so no service restart is needed after a rebuild. If a blank page still appears: hard-refresh the browser (Ctrl+F5) to bypass the cached HTML/assets, and confirm the build actually wrote to static/ (run npm run build inside frontend/).

Q: Dev mode (npm run dev) shows 404?

Vite dev server runs on port 5173. vite.config.js proxies /api, /static and /ws to the CD backend on port 8000 — start the backend with PORT=8000 (or adjust the proxy target to your backend port):

server: {
  port: 5173,
  proxy: {
    '/api': 'http://localhost:8000',
    '/static': 'http://localhost:8000',
    '/ws': { target: 'ws://localhost:8000', ws: true }
  }
}

Permissions

Q: Some sidebar menu items are missing after login?

Permissions are managed by CI's roles / permissions / role_permissions tables. CD only reads, never writes.

Diagnosis:

  1. Verify your role has the corresponding perm_key in CI's role_permissions table
  2. super_admin role has implicit access to all cd.* permissions — no separate assignment needed
  3. See admin manual for full permission list

Without CI admin UI, insert directly in the database:

-- View current role permissions
SELECT r.name, GROUP_CONCAT(rp.perm_key) AS perms
FROM roles r
LEFT JOIN role_permissions rp ON r.id = rp.role_id
GROUP BY r.id;

-- Grant all CD permissions to admin role
INSERT INTO role_permissions (role_id, perm_key)
SELECT r.id, p.perm_key
FROM roles r, permissions p
WHERE r.name = 'admin' AND p.perm_key LIKE 'cd.%';

Q: Error "Permission denied: cd.xxx required"?

Your role lacks that permission key. Ask an admin to assign it in CI, or use a super_admin account.

Q: CI hasn't created roles/permissions tables yet?

CD degrades gracefully: when tables don't exist, get_current_user returns an empty permission list. The super_admin role still has implicit full access (via the role field), so super admins are unaffected. Normal roles will work after tables are created.

Q: Added a new permission point but checks don't work?

New perm_key entries must be inserted into CI's permissions table AND assigned to roles in role_permissions. Otherwise require_perm() will deny all users. Always synchronize new permission keys with CI.


Performance

Q: High memory usage?

  • Reduce monitor cache TTL via env vars: lower MONITOR_CACHE_* values
  • Harbor sync is a background thread; increase REGISTRY_SYNC_INTERVAL to reduce frequency

Q: Log table growing too fast?

cd_deploy_logs stores every deployment's output. The CD Service does not have built-in log cleanup — this is an ops responsibility.

Manual cleanup (SQL):

DELETE FROM cd_deploy_logs WHERE created_at < DATE_SUB(NOW(), INTERVAL 30 DAY);

System cron (Linux crontab example):

# Clean up deployment logs older than 30 days, daily at 2 AM
0 2 * * * mysql -u root -p'password' devops_glue -e "DELETE FROM cd_deploy_logs WHERE created_at < DATE_SUB(NOW(), INTERVAL 30 DAY);"

Background tasks: The CD Service runs the following built-in background threads (intervals configured via config/app.env):

  • Harbor registry sync: REGISTRY_SYNC_INTERVAL (default 30 minutes, 0 disables)
  • Alert checking: ALERT_CHECK_INTERVAL (default 60s)
  • Scheduled publish executor: drains approved deployment approval tickets whose scheduled_at time has arrived
  • Deploy heartbeat: refreshes cd_deploy_logs.heartbeat_at so interrupted deploys can be detected and recovered on restart