Draft: Meta Commit II - far too large.#10
Open
DarthPJB wants to merge 179 commits into
Open
Conversation
added 2 commits
July 10, 2026 09:22
DarthPJB
changed the title
added 27 commits
July 11, 2026 09:19
…3100→9100 Scraped 4 imperative dashboards from Grafana database on local-nas: - disk-usage.json — LINDA memory/ZFS/disk usage monitoring - failstate-overview.json — Fleet-wide failed systemd units - cpu-monitor-update.json — CPU frequency heatmaps per machine - cpu-monitor-disk-regex-fix.json — CPU/disk/ZFS combined view All dashboards had stale port 3100 references (node_exporter standardized on 9100 via environments/metrics.nix). Fixed all 54 occurrences. These dashboards were previously only in the Grafana SQLite database (imperative state). Now they're in the project and will be provisioned declaratively on next deployment, replacing the imperative copies.
…b.json Merged cpu-monitor-update.json and cpu-monitor-disk-regex-fix.json into a single fleet-cpu-disk.json dashboard with the best of both: - System Statuses state-timeline with 9 hostname mappings - Data Throughput (ethtool) and Energy Usage (hwmon/nvidia) - Disk RW Access as timeseries (better than heatmap) - CPU heatmaps: LINDA, cortex-alpha, terminal-zero, terminal-nx-01, data-storage, remote systems, ARM systems, local systems Removed noob.json (uid: jof8tnw) — fully superseded by the merged dashboard. Metric validation against live Prometheus (23/24 pass): - All metrics return data except node_power_supply_energy_watthour (expected — no battery/UPS hardware on fleet machines) Final dashboard count: 9 (from original 7 + 4 scraped - 3 removed)
Review against live Prometheus found: - disk-usage.json: ZFS panel had duplicate queries (A and B identical); fixed B to show writes. Renamed uid to 'linda-system', title to 'LINDA System Metrics', added tags. - failstate-overview.json: Panel 'Disk Read' actually showed writes; renamed to 'Disk Write Activity'. Fixed uid to 'failstate-overview', added tags. - fleet-cpu-disk.json: Removed 4 panels targeting DOWN instances (cortex-alpha, terminal-zero, terminal-nx-01, remote systems) and empty 'CPU - General' row. Kept panels for UP instances only. Removed old CPU monitor dashboards from Grafana DB on local-nas (jokkz2m, jo5b5zx) and restarted Grafana. Review reports written to documentation/2026-07-11-GRAFANA-DASHBOARD-REVIEW/
…ce redesign Exporter (nixos-deployment-exporter.nix): - Added derivation_path and outPath to build metadata - Added derivation_path label to nixos_flake_info metric Dashboard fixes: - disk-health.json: Fixed SMART pending sectors attribute name (Current_Pending_Sector_Ct → Current_Pending_Sector) - service-health.json: Removed noisy Active Services panel; all Key Services now only show failures (state=failed); fixed SSH to check sshd.service|sshd.socket instead of just sshd.service - fleet-deployment.json: Added Derivation Path table panel showing nixos_derivation_path from nixos_flake_info - storage-io.json: Added ZFS Pool Usage gauge panel; filtered filesystem usage to exclude ZFS child datasets sharing parent pool space - network-wireguard.json: Hide expected-down interfaces (wireg0) and only show interfaces that are UP in status panel - disk-usage.json: Fixed schema version 41→42
…toplevel Use self.outPath (flake output path) instead of config.system.build.toplevel which creates a circular dependency since the exporter is part of the system.
…d. nix wins, db stays pure ♡
…ableDelete); remove non-standard purge hack
…n file per user instruction. Return to pre-session state.
Correct updateIntervalSeconds, enable provision, allowUiUpdates=false, disableDeletion=false, prune datasources. Remote state follows Nix only.
Bump every dashboard checksum (version++, editable=false, nix-provisioned tag) so deploy must overwrite SQLite by UID. Poll interval 300s (5m).
Dashboards: Disk RW rate(), remote CPU/vCPU, activation dateTime, system path table, SMART pending regex, UP-only network, SSH active health, bargauge FS usage, ZFS dedup folded into storage-io, LINDA portrait. Exporter: write system_path at activation + oneshot; nixos_system_info; derivation_path from live closure not flake source.
print-controller/display-*: force-disable smartd+smartctl (deploy exit 17). Topology validate: skip hostname-without-mac for WireGuard-only hosts. remote-worker: pkgs.stdenv.hostPlatform.system (kills rename deprecation).
Regenerated: alpha-one, alpha-three, cortex-alpha, gaming-host-1, LINDA, local-nas, remote-builder, remote-worker, terminal-nx-01, terminal-zero. Skipped (dormant/not in nixosConfigurations): alpha-two, display-0, storage-array. Skipped (nixpkgs fail2ban eval bug): arm-builder, beta-one, display-1, display-2, print-controller.
…olden format The golden files were generated with dump-config (lib/serialize-config.nix) which produces hierarchical config sections. check-network was using generate-golden (real-topology/default.nix) which produces flat individual options. Updated check-network and checks.network-config-cortex-alpha to use dump-config for consistency.
…tion Phase 1 of topology rectification: - topology/shared.nix (from topology.nix) - topology/cortex-alpha.nix (from real-topology/) - topology/default.nix (entry point, imports shared + per-machine) - topology/external/ (for non-Nix-managed systems) - goldens/ (golden test files) - lib/golden_generator.nix (from real-topology/default.nix) - lib/golden_coverage.nix (from real-topology/coverage.nix)
Phase 2 of topology rectification: - modules/core-router.nix → topology/<machine>.nix - modules/enable-wg-topology.nix → topology/shared.nix - modules/core-router-topology.nix → topology/shared.nix - services/prometheus.nix → topology/shared.nix - lib/golden_coverage.nix → topology/shared.nix + goldens/ - flake.nix → topology/shared.nix + goldens/ + topology/default.nix - tests/test-new-architecture.nix → topology/cortex-alpha.nix All 10 active machines pass golden validation.
Phase 3 of topology rectification: - Removed topology.nix (replaced by topology/shared.nix) - Removed real-topology/ directory entirely - All topology data now in topology/ - All golden files now in goldens/ - All 10 active machines pass golden validation real-topology/ is gone forever.
Phase 5 of overlord-II: - Added mkMultiplexConfig in flake.nix (mirrors mkKnownHosts pattern) - Generates programs.ssh.matchBlocks from topology data - Added tmpfiles rules for /run/user/%i/ssh-mux socket directory - Increased MaxSessions from 2 to 20 for multiplexing support - All machines get multiplexing config automatically from topology
Phase 7 of overlord-II: - Updated AGENTS.md with new topology/ paths - Updated documentation/file_structure.md - Updated documentation/code_structure.md - All references to real-topology/ replaced with topology/ and goldens/
programs.ssh.matchBlocks is not a valid NixOS option in nixpkgs 25.11. The SSH multiplexing plan was based on a false premise. Removed: - mkMultiplexConfig function from flake.nix - programs.ssh.matchBlocks from commonModules - MaxSessions reverted from 20 to 2 - tmpfiles rules for socket directory removed SSH multiplexing plan (ssh-multiplex-topology-2026-07-03.md) needs redesign using programs.ssh.extraConfig instead.
…olations Development report covering: - Golden test status: all 18 goldens identical to v1.9-Golden tag - Flake validation: all checks pass - Directive violations: SSH multiplexing implemented without verifying matchBlocks exists - Blockers resolved: check-network/golden format mismatch - Outstanding items: SSH multiplexing needs redesign
Deployment status with derivation outpaths for all 7 deployed machines. Listed 10 undeployed active systems and 3 dormant for user review.
- Updated overlord-II-deployment-status.md with all 12 deployed machines - Documented known issues (exporter state discrepancy, matchBlocks) - Recorded tool patterns for future reference
retentionTime = "0d" — never delete historical data.
Four-agent parallel review: - tpol-xai: structural analysis — clean, no dead code - tpol-minimax: goal validation — Phase B/C largely incomplete - bellana-deepseek: engineering review — 3 functional issues found - ezri-claude-haiku: tactical review — deployment patterns correct Key findings: - Topology rectification: ✅ structurally sound - Golden integrity: ✅ preserved - Phase B transformers: ❌ skeletons, not production-ready - Phase C library split: ❌ not initiated - generate-golden app:⚠️ produces wrong format, needs fix/remove - Broken scripts:⚠️ topology-report.sh, validate-new-architecture.sh
…locks absence Verified claims: - generate-golden produces 34 keys (flat), dump-config produces 25 keys (hierarchical) - Golden files were regenerated in db90b5d from generate-golden to dump-config format - programs.ssh.matchBlocks confirmed absent from nixpkgs 25.11 SSH module - But this is Nix — we can create custom module or use extraConfig Phase B/C objectives remain incomplete.
…n.nix) arm-builder explicitly disables configuration.nix to stay minimal but still needs Determinate Nix FlakeHub token to silence auth errors. Duplicates the secrix.services.determinate-flakehub-login block.
added 28 commits
July 21, 2026 15:20
Category A — direct .system access (6 instances):
flake.nix, configuration.nix, arm-builder, input-methods, hyperland
Category B — deprecated system param to nixosSystem (5 instances):
mkX86_64, mkAarch64, arm-bootstrap, bargman-greeter-vm, beta-one
All had/have nixpkgs.hostPlatform; beta-one needed it added.
Category C — import nixpkgs { system = "..." } (3 instances):
Changed to localSystem for nixpkgs_unstable imports.
Eliminates all 'system has been renamed to stdenv.hostPlatform.system'
evaluation warnings.
…WowPackages → wineWow64Packages wireless: 4 machines (alpha-two, LINDA, terminal-zero, terminal-nx-01) wine: terratech.nix game server Goldens regenerated for affected machines.
…v1.18.4 environments/code.nix was installing pkgs_llm.opencode (1.15.10) which shadowed the opencode-fleet module's package (1.18.4 from opencode-flake) in the system PATH. Both provided /bin/opencode — the older version won the symlink. Removing the duplicate lets the fleet module's version take effect.
Agent needs full filesystem and command access to function. The upstream NixOS module hardcodes ProtectSystem=strict, ProtectHome=true, PrivateTmp=true, and restrictive ReadWritePaths — no module options exposed to control these. Override with lib.mkForce to allow free operation on the expendable alpha-three host.
The LLM-CORE opencode-fleet module requires services.opencode-fleet.user to be set because opencode reads agents from ~/.config/opencode/agents/, not /etc/opencode/agents/. - Added user = "John88" to services.opencode-fleet configuration - rootPath defaults to /speed-storage/opencode (matches local path) - terminal-zero now evaluates successfully
terminal-zero lacks AVX support required by opencode. Moving the LLM-CORE opencode-fleet module and configuration to alpha-one. - Removed LLM-CORE.nixosModules.opencode-fleet from terminal-zero - Removed services.opencode-fleet from terminal-zero/default.nix - Added LLM-CORE.nixosModules.opencode-fleet to alpha-one - Added services.opencode-fleet with user = "John88" to alpha-one Both machines evaluate successfully after changes.
…to 5 The --max 3 flag passed to nix-collect-garbage is not a valid option and causes nix-gc.service to fail across 9 machines. Removed it entirely. Instead, use boot.loader.systemd-boot.configurationLimit = mkDefault 5 as the fleet-wide generation retention default. LINDA keeps its explicit override at 1 for space-constrained NVMe.
Full implementation plan for serving signed /nix/store paths from remote-builder via cortex-alpha's nginx reverse proxy + split-horizon DNS. Explicitly blocked on topology implementation completion — all nginx proxy and dnsmasq entries flow through the topology system. Execution follows the infrastructure-2 pattern (nix-serve + reverse proxy + secrix keys) adapted for WireGuard-gated access instead of Tailscale. No public internet exposure. No changes to topology transformers/generators.
Added full opencode-fleet configuration modeled after LINDA: - secrix secret declarations for MCP tokens (github, gitlab, openrouter, mimo) - MCP servers: git, filesystem, time, sqlite, playwright, github, gitlab, prometheus - Providers: openrouter, xiaomi-token-plan-sgp Using shared tokens for now; alpha-one specific tokens can be added when re-encrypted secret files are available.
Added secrix secret declaration for general-opencode-key and configured opencode-go provider to use it.
Added secrix secret declaration for general-xai-key and configured xai provider to use it. All four providers now configured: - openrouter (openrouter-master-token) - opencode-go (general-opencode-key) - xiaomi-token-plan-sgp (mimo-token-plan-ai-key) - xai (general-xai-key)
… artifact actions - Remove unused self from ci.nix signature - Parameterize workflowAttrPath in generateWorkflowScript - Add concurrency controls (cancel-in-progress) to prevent queue buildup - Add push.branches = [main] for head-only trigger strategy - Remove upload-artifact from deploy-prep job - Regenerate golden and workflow YAML
- arm-builder and beta-one are cross-compiled (buildPlatform=x86_64-linux) - display-1, display-2, print-controller are native aarch64 builds - Separate jobs: build-arm-native (aarch64 runner) and build-arm-cross (x86_64 runner) - Cross-compiled jobs use x86 parallelism settings (max-jobs auto, cores 0) - Native ARM jobs keep constrained settings (max-jobs 2, cores 2)
- validate-ci-workflow now runs actionlint for comprehensive GitHub Actions validation - Fix SC2086: quote shell variables in deploy-prep build step
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This represents a large medium of work intedend to rectify a great deal of system functionality.
Rectification will allow for viable framework code compaction in overlord-III
Refer to internal documentation.
Also; self host your systems.