IBRL - Beacon & Policy VM for Hypercore Cluster - #20
Merged
mmchougule merged 6 commits intoNov 11, 2025
Conversation
Integrated IBRL (Incentivized Bandwidth Resource Layer) beacon functionality into Hypercore to enable path-aware, economically-incentivized workload routing. ## Changes ### Protocol Buffer Extensions - Added BeaconMetadata message (latency, jitter, packet loss, queue depth, price/GB, reputation) - Added BeaconAttestation, PolicyContext, WorkloadProof messages for future phases - Extended NodeStateResponse to include beacon metadata - Added new cluster events: BEACON_ATTEST, POLICY_QUERY, PROOF_VERIFY ### New Beacon Package (pkg/beacon/) - client.go: Beacon client with ed25519 cryptographic signing, metrics tracking - registry.go: Node registry with metric-based filtering and stale node cleanup - attestation.go: Cryptographic attestation generation and verification ### Cluster Integration - Integrated beacon client into Agent struct in pkg/cluster/serf.go - Added beacon metadata to NodeStateResponse broadcasts in monitorWorkloads() - Added IBRL Prometheus metric: hypercore_ibrl_beacon_connected ### CLI Enhancement - Added 'hypercore cluster metrics' command to display IBRL node metrics - Shows latency, price, reputation, queue depth per node ## Architecture - Zero-dependency standalone operation (no external beacon required) - Thread-safe metric updates with mutex protection - Ed25519 signatures for cryptographic attestations - Backwards compatible with existing clusters ## Build Status ✅ Build successful ✅ Proto generation complete ✅ No breaking changes Total: ~766 lines added across 3 new files + 5 modified files See IBRL_PHASE1_SUMMARY.md for complete details.
Added comprehensive documentation for IBRL integration planning and implementation: - ARCHITECTURE_ANALYSIS.md (626 lines): Complete Hypercore codebase analysis including cluster module architecture, Serf integration, workload scheduling, and existing monitoring infrastructure - IBRL_INTEGRATION_GUIDE.md (577 lines): Step-by-step integration guide with proto extensions, package structure, integration points, testing strategy, deployment checklist, and phased rollout plan - QUICK_REFERENCE.md (337 lines): Quick reference for file locations, port assignments, code entry points, and configuration defaults - EXPLORATION_SUMMARY.txt: Summary of exploration findings and key discoveries These documents provide essential context for IBRL development and serve as reference material for Phase 2 (Policy VM) and Phase 3 (Proof & Settlement).
Implemented intelligent, policy-based workload scheduling to replace first-fit
with multi-criteria node selection based on latency, price, reputation, and
queue depth metrics.
## Changes
### New Policy Engine Package (pkg/policy/)
- policy.go: JSON-based policy language with hard constraints and soft scoring
- engine.go: Policy evaluation engine with weighted node ranking
- Supports two modes: "enforce" (strict) and "permissive" (fallback)
- Thread-safe policy updates with mutex protection
### Policy Features
- Hard Constraints: max_latency_ms, max_price_per_gb, min_reputation_score,
max_queue_depth, max_packet_loss, max_jitter_ms
- Weighted Scoring: configurable weights for latency, price, reputation, queue
- Node Ranking: selects optimal node based on composite score
- Graceful Fallback: falls back to broadcast if policy selection fails
### Cluster Integration (pkg/cluster/serf.go)
- Added policyEngine to Agent struct
- Modified SpawnRequest() to use policy-based node selection
- SelectNodes() evaluates all cluster members against policy
- Tries nodes in priority order (highest score first)
- Logs policy decisions for observability
### CLI Enhancements
- Added --cluster-policy flag for cluster-wide default policy
- Added --policy flag for per-spawn policy override
- Modified config.go and flags.go to support policy file paths
### Example Policies (examples/policies/)
- low-latency.json: Prioritize low-latency nodes (60% latency weight)
- cost-optimized.json: Minimize cost (70% price weight)
- balanced.json: Equal weights across all metrics
- high-trust.json: Only use nodes with reputation >= 0.9
- permissive.json: Accept any available node (default)
### Documentation
- IBRL_E2E_TESTING_GUIDE.md (435 lines): Complete testing guide with
multi-node setup, policy testing scenarios, troubleshooting
- IBRL_PHASE2_SUMMARY.md: Architecture overview, scoring algorithm,
usage examples, performance characteristics
## Algorithm
For each node:
```
score = (latency_weight × latency_score) +
(price_weight × price_score) +
(reputation_weight × reputation_score) +
(queue_weight × queue_score)
```
Nodes ranked by score (descending), spawns attempted in priority order.
## Build Status
✅ Build successful
✅ No breaking changes
✅ Backward compatible (policies optional)
## Usage
Start cluster with policy:
```bash
sudo ./bin/hypercore cluster \
--cluster-policy examples/policies/balanced.json
```
Spawn with policy override:
```bash
./bin/hypercore cluster spawn \
--image-ref nginx:latest \
--policy examples/policies/low-latency.json
```
Total: ~774 lines added (2 new packages, 5 examples, 2 docs)
See IBRL_PHASE2_SUMMARY.md and IBRL_E2E_TESTING_GUIDE.md for details.
mmchougule
changed the title
…aluate all nodes and route intelligently. refactor: remove obsolete documentation files Deleted tmp .md file
- Added new configuration options for beacon endpoint, price, and reputation in Config struct. - Updated ClusterCommand to handle containerd availability on Mac, providing user guidance for metrics and list commands. - Modified Agent struct to include beacon parameters and updated NewAgent function to initialize the beacon client with these parameters. - Implemented HTTP health check for the beacon endpoint, falling back to TCP connection if necessary. - Enhanced workload monitoring to include latency and jitter metrics, improving decision-making for the policy engine. These changes improve the robustness of the beacon integration and provide better observability for cluster performance.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the first working components of the Internet's Bandwidth Resource Layer (IBRL) inside Hypercore.
This upgrade allows cluster nodes to broadcast live performance metrics and be evaluated by a policy engine before workload deployment.
Routing decisions are now deterministic and policy-driven rather than random broadcast.