diff --git a/.gitignore b/.gitignore index 8c172705..f846707b 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,6 @@ +# Demo presentation docs (local prep, not shipped documentation) +docs/DEMO_STRATEGY_WEBEX.md + # Pre-existing inputs (not products of the wiggum loop) REVERSE.md specs/ diff --git a/README.md b/README.md index 952cfa98..1b3fb4d7 100644 --- a/README.md +++ b/README.md @@ -29,16 +29,9 @@ feeding back into it. Everything below is self-contained: the instructions live here rather than in a separate specification. -> **Built with [Specstride](https://specstride.ai).** Not a badge — the method. -> Every spec, plan, task and verification gate in this lab was driven through -> Specstride's spec-driven pipeline: the autonomous tier, the controllers and -> the fabric weren't hand-waved into existence, they were *specified* into -> existence, then proven against the running system. A network that writes -> itself deserves a process that does too — **spec it, stride it, ship it.** - ## Demo -Full walkthrough (~6½ min, 1.7x) — the intent tier end to end: three services +Full walkthrough (~6½ min, 6x) — the intent tier end to end: three services provisioned from plain-language prompts typed into the operator console (a **vlan**, an **ip-vrf** with its prefix, and a **mac-vrf** stretched across both leaves), each one confirmed through the mapper and allocator agents, @@ -47,7 +40,7 @@ reported deployed, and then proven in the terminal with `kubectl` (the the SONiC leaf itself (CONFIG_DB, the bridge, the tenant VRF's L3VNI and Type-5 EVPN route, the VNI tunnel map on both leaves). -[![Three services provisioned from plain-language prompts and proven on the fabric](docs/media/agentic-netops-demo-thumb.jpg)](docs/media/agentic-netops-demo.mp4) +https://github.com/user-attachments/assets/09b5c615-8dc5-4e3e-8937-f6eb16ea770f Every "deployed" claim in the recording was re-verified from `kubectl` JSON; the collected evidence is in diff --git a/TUTORIAL.md b/TUTORIAL.md index 4eb5471e..68865495 100644 --- a/TUTORIAL.md +++ b/TUTORIAL.md @@ -13,6 +13,11 @@ the same intent is applied twice. Those are the sections that matter here. ## The loop +> Preparing to present this lab? See [docs/DEMO_STRATEGY_WEBEX.md](docs/DEMO_STRATEGY_WEBEX.md) +> (run of show, building-blocks narrative, lessons learned) and +> [docs/DEMO_TUTORIAL.md](docs/DEMO_TUTORIAL.md) (hands-on steps for SDC/Kubenet/KUID/SRv6, +> services, and UUID/identifier retrieval). + ``` plain language ┌──────────────┐ "connect tenant ──────►│ supervisor │ classify: provisionable? @@ -226,9 +231,36 @@ Real, reproduced, documented rather than hidden: - The former SONiC ASan/L2-VNI and EVPN Type-5 gaps (D-A2/D-A3) are resolved on the pinned `sonic-vs-gnmi:202505-v1` image and their waivers are retired. -- **SRv6 conformance needs the `sonic-vm` profile**, which requires an - operator-built vrnetlab image *and* a `lab/profiles/sonic-vm/bootstrap/` directory - that does not exist yet. +- **SRv6 conformance runs on the `sonic-vm` profile**, which is now fully + wired AND qualified: `lab/profiles/sonic-vm/bootstrap/` (VM-aware, + SSH-driven gNMI/TLS provisioning) and `lab/topology.sonic-vm.clab.yml` + exist, `scripts/build-sonic-vm.sh` builds the vrnetlab image, and the full + qualification gate (`scripts/lib/qualify.sh`: Capabilities/Get/Set/ + Subscribe, sonic-srv6, persistence, EVPN Type-2/3/5, SRv6 underlay/End/ + End.DT46/decap/counters, YANG paths) passes end-to-end against + `local/sonic-vm:202505`. It still requires the **operator** to build the + `local/sonic-vm` image + (`scripts/build-sonic-vm.sh --image-url --update-pin`), + because no public registry serves it, plus KVM/nested virtualization on the + host. Until that build exists on your host, `--profile sonic-vm` fails at + deploy with a missing-image error (the documented gate-fail path). + sonic-vm-profile notes (all verified on the pinned image): + - gNMI TLS is driven by the **`GNMI` CONFIG_DB table** (`GNMI|gnmi` + + `GNMI|certs`), not the legacy `TELEMETRY` table — `gnmi-native.sh` + renders `telemetry_vars.j2` against `GNMI`, and an undefined table makes + it fall back to `--noTLS`. The bootstrap applies both tables. + - The vrnetlab launcher resets the guest admin password to containerlab's + `PASSWORD` env (default `admin`); the guest is reachable over SSH on the + **launcher's own mgmt IP**, not 127.0.0.1 (loopback→slirp stalls). + - The image default config gives every node `Loopback0=10.1.0.1/32` and + bgpd AS 65100/route-id 10.1.0.1 — test topologies must not use 10.1.0.1 + as a peer address (martian-dropped) and must set distinct router-ids. + - The guest kernel (SONiC Debian 6.1) has **no kernel seg6 lwtunnel** + support: the kernel-side `encap seg6` dataplane witnesses are + auto-skipped on this profile; SRv6 control-plane state is still asserted + over gNMI. Kernel seg6 witnesses run on the `sonic-vs` profile. + - Persistence on `sonic-vm` is verified with a guest `config reload` + (re-reads `/etc/sonic/config_db.json`) rather than a container restart. - **Two pinned images have no local build step** (`grafana/flow-plugin`, `ghcr.io/agentic-netops/topology-generator`); `deployment/ui` ends in `ImagePullBackOff`. - **`docs/INTENT_TIER_OPS_READINESS.md` holds resource figures that were never diff --git a/docs/DEMO_TUTORIAL.md b/docs/DEMO_TUTORIAL.md new file mode 100644 index 00000000..dfc38498 --- /dev/null +++ b/docs/DEMO_TUTORIAL.md @@ -0,0 +1,280 @@ +# agentic-netops demo tutorial — building blocks, services, and identifiers + +A step-by-step companion to [TUTORIAL.md](../TUTORIAL.md) focused on the four +building blocks — **SDC (sdcio)**, **Kubenet**, **KUID**, **SRv6** — the services +you can provision live, and how to retrieve object UUIDs and allocated identifiers +from Kubernetes. For the call-planning view see +[DEMO_STRATEGY_WEBEX.md](DEMO_STRATEGY_WEBEX.md). + +> Naming note: "kuuid" in the project is **KUID** (`id.kuid.dev`), the +> index/claim allocator, deployed alongside **Kubenet** (`network.kubenet.dev`). + +--- + +## 0. Bring the lab up + +```bash +cd /root/agentic-netops +cp .env.example .env # set LLM_MODEL — without it the tier deploys but cannot reason +./scripts/provision.sh --profile sonic-vs --cluster-name agentic-netops --with-intent-tier +export CTX='--context kind-agentic-netops' +``` + +> **Paste-safety tip:** every command below is a single line on purpose. When +> pasting multi-line commands from notes or slides, a broken `\` continuation +> makes kubectl read the column spec as an object *name* — the tell-tale error is +> `Error from server (NotFound): ... "custom-columns=..." not found`. If you see +> it, re-run the line as one; also confirm `echo $CTX` prints +> `--context kind-agentic-netops`. + +Confirm the three stacks are installed: + +```bash +kubectl $CTX get crd | grep -E 'kubenet|kuid|sdcio' +# networks.network.kubenet.dev srv6services.agentic-netops.io +# ipindexes.id.kuid.dev asnindexes.id.kuid.dev vniindexes.id.kuid.dev claims.id.kuid.dev +# configs.sdc.sdcio.dev targets.sdc.sdcio.dev schemas.sdc.sdcio.dev ... +``` + +Fabric sanity: two spines, two leaves, four clients in containerlab +(`docker ps | grep clab-agentic-netops`), underlay established: + +```bash +docker exec clab-agentic-netops-fabric-leaf01 vtysh -c 'show bgp summary' +``` + +--- + +## 1. Building block: SDC / sdcio (model-driven config plane) + +SDC runs in-cluster (Schema server, Config server, Data server, Cache; manifests in +`deploy/sdc/`). It is the only path to the devices: controllers write Config +resources, SDC renders them against the SONiC YANG schema and applies over gNMI. + +```bash +# the fabric nodes as SDC targets, and their sync state +kubectl $CTX get targets.sdc.sdcio.dev -A + +# desired config per service, held declaratively +kubectl $CTX get configs.sdc.sdcio.dev -A + +# schemas loaded for SONiC +kubectl $CTX get schemas.sdc.sdcio.dev -A +``` + +What to say: agents never open a device session — they only write Kubernetes +resources. SDC is where "declarative" becomes bytes on the box, and because every +apply is a reconcile of desired state, re-applying unchanged intent produces **zero** +writes: + +```bash +./tests/integration/idempotence.sh # asserts no SDC spec writes, no gNMI Sets +``` + +--- + +## 2. Building block: Kubenet (service intent, `network.kubenet.dev`) + +Kubenet provides the topology and network-service model. This repo deploys a +`Topology` (the 2×2 Clos), and adds the `Network` CR (service types: vlan, +mac-vrf, ip-vrf, acl) plus the `SRv6Service` CR. + +```bash +kubectl $CTX -n kubenet-system get topology +kubectl $CTX -n agentic-netops-intent get networks.network.kubenet.dev + +# watch desired state and the controller's verdict, live +kubectl $CTX -n agentic-netops-intent get networks.network.kubenet.dev -o 'custom-columns=NAME:.metadata.name,TYPE:.metadata.annotations.agentic-netops\.io/service-type,READY:.status.conditions[?(@.type=="Ready")].status' -w +``` + +`Ready=True` is written only after every rendered operation applied **and** every +per-node verification passed — and it is re-verified every five minutes. + +--- + +## 3. Building block: KUID (identifier allocation, `id.kuid.dev`) + +KUID models scarce identifiers as indexes; consumers file `Claim`s. The lab ships +IP/ASN/VNI pools for the fabric plus a dedicated SRv6 chain: + +```bash +kubectl $CTX -n kuid-system get ipindexes,asnindexes,vniindexes +# srv6-locator-v6 2001:db8:feed::/48 <- locator pool +# srv6-sids-v6 2001:db8:feed:100::/64 <- SID pool under it +# srv6-service-ids VNI range 500000-500999 <- SRv6 service IDs +``` + +Provisioning a service makes a claim appear; withdrawing intent releases it — +shared fabric state and unrelated claims survive: + +```bash +kubectl $CTX -n kuid-system get claims -o wide +./tests/integration/update_delete_survivability.sh # proves claim release +``` + +--- + +## 4. Building block: SRv6 + +`SRv6Service` expresses a bidirectional IPv6 VPN: a KUID-allocated locator, two +attachments, a path with primary/alternate waypoints, and an immutable VRF. +Structural CEL rules on the CRD reject bad shapes (non-IPv6 prefixes, identical +endpoints, locator outside /48-/64) at admission time. + +Try it directly: + +```bash +kubectl $CTX -n agentic-netops-intent apply -f - <<'EOF' +apiVersion: agentic-netops.io/v1alpha1 +kind: SRv6Service +metadata: + name: demo-srv6 +spec: + topologyRef: agentic-netops-topology + vrf: demo-vrf + underlay: ipv6 + locatorPrefix: 2001:db8:feed::/48 + servicePrefix: 2001:db8:10::/64 + path: + primary: spine01 + attachments: + - node: leaf01 + interface: ethernet4 + - node: leaf02 + interface: ethernet4 + client: + name: srv6-client01 + ip: 2001:db8:10::11/64 + server: + name: srv6-client02 + ip: 2001:db8:10::22/64 +EOF +kubectl $CTX -n agentic-netops-intent get srv6services demo-srv6 -w +``` + +Verify on the fabric: + +```bash +docker exec clab-agentic-netops-fabric-leaf01 vtysh -c 'show segment-routing srv6 locator' +``` + +Honest limitations to state: full SRv6 conformance runs on the `sonic-vm` profile; +on the virtual-SONiC image the proven path is GCU-write → gNMI read-back witnesses +(locator, `SRV6_MY_SIDS`, counters) — details in +[SRV6_GNMI_CAPABILITY_FINDINGS.md](SRV6_GNMI_CAPABILITY_FINDINGS.md). + +--- + +## 5. Provision services from plain language + +Open the console at http://localhost:30000 (the scenario cards are exactly what the +supervisor advertises on `GET /suggested-prompts`), say the prompt, `confirm` twice. + +| Prompt | Construct | +|--------|-----------| +| "Provision a vlan 120 on leaf01 ethernet1 for tenant acme" | vlan | +| "Extend vlan 100 as a mac-vrf across leaf01 ethernet2 and leaf02 ethernet2 for tenant blue" | mac-vrf | +| "Give tenant initech an ip-vrf carrying 10.50.0.0/24 on leaf01 wan1" | ip-vrf | +| "Apply an acl on leaf01 ethernet1 and leaf02 ethernet1 for tenant acme: permit tcp 443 from 10.0.0.0/24, deny everything else" | acl | + +Watch the tier reason: + +```bash +kubectl $CTX -n agentic-netops-agents logs -f deploy/supervisor +``` + +Prove each one where it counts: + +```bash +# vlan: CONFIG_DB on the leaf +docker exec clab-agentic-netops-fabric-leaf01 docker exec sonic /usr/local/bin/sonic-cfggen -d --print-data | grep -A3 VLAN + +# mac-vrf: EVPN overlay on BOTH leaves — Remote VTEPs = 1 is the signal that matters +docker exec clab-agentic-netops-fabric-leaf01 vtysh -c 'show evpn vni' +docker exec clab-agentic-netops-fabric-leaf02 vtysh -c 'show evpn vni' + +# end-to-end data path +docker exec clab-agentic-netops-fabric-client01 ping -c3 192.0.2.21 +``` + +Also demo the refusal: name a port that does not exist ("leaf01 ethernet99"). The +translator refuses **before** submission and lists the site's real ports. And the +informational class: "What service types do you support?" answers, provisions +nothing. + +--- + +## 6. Retrieving UUIDs and identifiers from Kubernetes + +Three layers, each answering a different audit question. + +### 6.1 Kubernetes object UUIDs — *which object is this?* + +Every CR carries an immutable etcd-level `metadata.uid`: + +```bash +# all Networks with their UUIDs +kubectl $CTX -n agentic-netops-intent get networks.network.kubenet.dev -o 'custom-columns=NAME:.metadata.name,UID:.metadata.uid,CREATED:.metadata.creationTimestamp' + +# SRv6Services as nameuid +kubectl $CTX -n agentic-netops-intent get srv6services -o 'jsonpath={range .items[*]}{.metadata.name}{"\t"}{.metadata.uid}{"\n"}{end}' + +# one object, full metadata including uid +kubectl $CTX -n agentic-netops-intent get network -o jsonpath='{.metadata.uid}' +``` + +### 6.2 Correlation IDs — *which sentence created it?* + +The console stamps each request with a `correlation_id`, carried on the Network as +a label — the thread from chat to CR to events: + +```bash +kubectl $CTX -n agentic-netops-intent get networks -l agentic-netops.io/correlation-id= -o 'custom-columns=NAME:.metadata.name,TYPE:.metadata.annotations.agentic-netops\.io/service-type,READY:.status.conditions[?(@.type=="Ready")].status,REASON:.status.conditions[?(@.type=="Ready")].reason' + +# the controller's own verdict as events +kubectl $CTX -n agentic-netops-intent get events --field-selector reason=ApplySucceeded +``` + +A worked example of the whole chain (prompt → correlation id → Ready condition → +fabric evidence) is captured in +[media/agentic-netops-intent-tier-demo-evidence.json](media/agentic-netops-intent-tier-demo-evidence.json). + +### 6.3 KUID claims — *which fabric identifiers did it consume?* + +Not Kubernetes UUIDs — the actual network identity allocated to each service: + +```bash +kubectl $CTX -n kuid-system get claims -o 'custom-columns=NAME:.metadata.name,POOL:.spec.from.name,PURPOSE:.spec.purpose' +``` + +Together the three layers give a complete audit chain: `metadata.uid` proves which +object you are auditing, the correlation-id proves which intent created it, and the +KUID claim proves which VNI/locator/SID it consumed. + +--- + +## 7. Prove autonomy (optional, ~10 min) + +```bash +./tests/integration/drift_preservation.sh # managed paths restored, unmanaged preserved +./tests/integration/failure_recovery_invalid_yang.sh # target killed mid-write still converges +./tests/integration/idempotence.sh # no churn +./tests/integration/update_delete_survivability.sh # claims released, shared state intact +./tests/integration/fabric_verify.sh run # fails closed, never "green by assumption" +``` + +## 8. Tear down + +```bash +./scripts/off.sh --delete-kind true # idempotent; safe to re-run +``` + +## Troubleshooting quick hits + +| Symptom | Fix | +|---------|-----| +| `kubectl top` returns nothing | kind has no metrics-server — expected | +| gNMI `Unauthenticated` | credentials rotate per provision — re-read the `gnmi-lab-creds` secret (TUTORIAL.md appendix) | +| `namespaces "kubenet-system" not found` | wrong kubectl context — check `$CTX` | +| Console up but no reasoning | `LLM_MODEL` empty in `.env` | +| Provision dry-run hits the wrong cluster | `provision.sh` uses your *current* context — set it first | diff --git a/lab/profiles/sonic-vm/bootstrap/README.md b/lab/profiles/sonic-vm/bootstrap/README.md new file mode 100644 index 00000000..0230f2f7 --- /dev/null +++ b/lab/profiles/sonic-vm/bootstrap/README.md @@ -0,0 +1,31 @@ +# SONiC VM Profile — Limited Bootstrap + +Scope: identical to sonic-vs — management reachability, TLS materials, gNMI service +enablement, and persistence. No underlay/overlay control-plane or data-plane +configuration is applied by this profile; EVPN/SRv6 qualification drives +configuration over gNMI during qualification (scripts/lib/qualify.sh). + +VM-specific execution model (differs from sonic-vs): +- The containerlab `sonic-vm` kind runs a QEMU *launcher* container; SONiC runs + inside a VM. `clab::bootstrap` docker-exec's `init-sonic-bootstrap.sh` into the + launcher, which then provisions the guest over SSH (vrnetlab forwards guest + port 22 into the launcher namespace; `sshpass` ships in the vrnetlab sonic + image; factory credentials `admin`/`YourPaSsWoRd` are overridable via + `SONIC_VM_ADMIN_USER` / `SONIC_VM_ADMIN_PASS`). +- TLS key/cert are installed into the guest's /etc/sonic/telemetry/ (mandatory + .cer suffixes, same YANG validation as the VS profile). +- gNMI/telemetry is enabled via TELEMETRY|gnmi + TELEMETRY|certs in the guest + CONFIG_DB (`config load` + `config save` for persistence), TLS only, JSON_IETF + encoding. The gNMI account is created as a Linux user in the guest (PAM/sshd + password auth), with the lab-generated credentials from gnmi_creds.json. +- There is deliberately no configure-fabric-bgp.sh here: it drives FRR/redis via + docker exec, which cannot cross into the VM guest. Fabric configuration is + gNMI-driven by qualification. + +Artifacts: +- gnmi_config_db.json — minimal CONFIG_DB snippet enabling gNMI/telemetry with TLS paths. +- install-gnmi-certs.sh — guest-side idempotent TLS installer (invoked over SSH). +- init-sonic-bootstrap.sh — launcher-side orchestrator (SSH into guest, run 1-4). + +The image itself is built by scripts/build-sonic-vm.sh (vrnetlab, KVM required) +and pinned in versions.lock.yaml (sonic_images.sonic_vm + sonic_yang.compatibility). diff --git a/lab/profiles/sonic-vm/bootstrap/gnmi_config_db.json b/lab/profiles/sonic-vm/bootstrap/gnmi_config_db.json new file mode 100644 index 00000000..90b05912 --- /dev/null +++ b/lab/profiles/sonic-vm/bootstrap/gnmi_config_db.json @@ -0,0 +1,32 @@ +{ + "DEVICE_METADATA": { + "localhost": { + "switch_type": "npu" + } + }, + "GNMI": { + "gnmi": { + "port": "8080", + "client_auth": "true", + "log_level": "2" + }, + "certs": { + "server_key": "/etc/sonic/telemetry/gnmi.key", + "server_crt": "/etc/sonic/telemetry/gnmi.cer", + "ca_crt": "/etc/sonic/telemetry/ca.cer" + } + }, + "TELEMETRY": { + "gnmi": { + "port": "8080", + "client_auth": "true", + "user_auth": "password", + "log_level": "2" + }, + "certs": { + "server_key": "/etc/sonic/telemetry/gnmi.key", + "server_crt": "/etc/sonic/telemetry/gnmi.cer", + "ca_crt": "/etc/sonic/telemetry/ca.cer" + } + } +} diff --git a/lab/profiles/sonic-vm/bootstrap/init-sonic-bootstrap.sh b/lab/profiles/sonic-vm/bootstrap/init-sonic-bootstrap.sh new file mode 100755 index 00000000..cbcab47d --- /dev/null +++ b/lab/profiles/sonic-vm/bootstrap/init-sonic-bootstrap.sh @@ -0,0 +1,103 @@ +#!/usr/bin/env bash +# SONiC VM (vrnetlab / containerlab sonic-vm kind) bootstrap: TLS + gNMI only. +# +# CONTEXT (different from sonic-vs): clab::bootstrap docker-exec's THIS script +# into the node container, but for the sonic-vm kind that container is the +# vrnetlab QEMU *launcher* — SONiC itself runs inside a VM. docker exec never +# reaches the guest. vrnetlab forwards guest TCP ports into the launcher +# namespace, so guest SSH (22) is reachable from here via the launcher's own +# mgmt IP (NOT 127.0.0.1 — see the credentials note below), and sshpass is +# baked into the vrnetlab sonic image. We provision the guest over SSH. +# +# What this script does inside the GUEST (all over SSH): +# 1. install TLS key/cert (mandatory .cer suffixes) into /etc/sonic/telemetry +# 2. merge gnmi_config_db.json (TELEMETRY|gnmi + TELEMETRY|certs) via +# `config load` and persist it with `config save` +# 3. create the gNMI Linux user with the lab-generated password +# (sonic-gnmi user_auth=password dials PAM/sshd, so the user must exist +# in the guest with that password — same mechanism as the VS profile) +# 4. restart the gnmi service and wait for :8080 to listen +# +# Idempotence: clab::bootstrap gates on the /etc/agentic-netops/.bootstrapped +# marker in the launcher; every guest step here is also safe to re-run. +set -uo pipefail + +# Credentials: vrnetlab's launch.py logs into the factory guest and RESETS the +# admin password to $PASSWORD (containerlab default: admin) during its own +# bootstrap — the factory YourPaSsWoRd is then rejected. Follow the env when +# present. NOTE: 127.0.0.1:22 is deliberately avoided as the guest address — +# the in-container loopback path into qemu's slirp hostfwd stalls (TCP accepts +# but the guest banner never returns); the launcher's own mgmt IP works. +ADMIN_USER=${SONIC_VM_ADMIN_USER:-${USERNAME:-admin}} +ADMIN_PASS=${SONIC_VM_ADMIN_PASS:-${PASSWORD:-admin}} +GUEST=${SONIC_VM_GUEST_ADDR:-$(hostname -i | awk '{print $1}')} +SSH_OPTS="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 -o LogLevel=ERROR" + +log() { echo "[vm-bootstrap] $*"; } +die() { echo "[vm-bootstrap] ERROR: $*" >&2; exit 1; } + +command -v sshpass >/dev/null 2>&1 || die "sshpass not found in launcher (vrnetlab sonic image should provide it)" + +# shellcheck disable=SC2086 # SSH_OPTS is intentionally word-split (multiple options) +gssh() { sshpass -p "$ADMIN_PASS" ssh $SSH_OPTS "${ADMIN_USER}@${GUEST}" "$@"; } +gscp() { sshpass -p "$ADMIN_PASS" scp $SSH_OPTS "$@"; } + +# 0) Wait for the guest to finish booting (SONiC VM takes ~1-3 min; sshd is +# one of the last services). Bounded at ~10 minutes. +log "waiting for guest SSH on ${GUEST}:22 ..." +ready=0 +for i in $(seq 1 60); do + if gssh true 2>/dev/null; then ready=1; break; fi + sleep 10 +done +[[ "$ready" == 1 ]] || die "guest SSH never became reachable after 10 minutes" +log "guest SSH is up" + +# 1) Stage TLS material + config snippet into the guest +gscp -q /etc/agentic-netops/gnmi/ca.crt "${ADMIN_USER}@${GUEST}:/tmp/" || die "scp ca.crt failed" +gscp -q /etc/agentic-netops/gnmi/gnmi.crt "${ADMIN_USER}@${GUEST}:/tmp/" || die "scp gnmi.crt failed" +gscp -q /etc/agentic-netops/gnmi/gnmi.key "${ADMIN_USER}@${GUEST}:/tmp/" || die "scp gnmi.key failed" +gscp -q /etc/sonic/bootstrap/gnmi_config_db.json "${ADMIN_USER}@${GUEST}:/tmp/gnmi_config_db.json" || die "scp config snippet failed" +gscp -q /etc/sonic/bootstrap/install-gnmi-certs.sh "${ADMIN_USER}@${GUEST}:/tmp/install-gnmi-certs.sh" || die "scp cert installer failed" + +# 2) Install certs + merge TELEMETRY config + persist +gssh "GNMI_CERT_SRC_DIR=/tmp/agentic-netops-gnmi bash -c ' + mkdir -p /tmp/agentic-netops-gnmi && + install -m 0644 /tmp/ca.crt /tmp/agentic-netops-gnmi/ca.crt && + install -m 0644 /tmp/gnmi.crt /tmp/agentic-netops-gnmi/gnmi.crt && + install -m 0600 /tmp/gnmi.key /tmp/agentic-netops-gnmi/gnmi.key && + chmod +x /tmp/install-gnmi-certs.sh && /tmp/install-gnmi-certs.sh +'" || die "guest cert install failed" + +gssh "sudo config load /tmp/gnmi_config_db.json -y" || die "config load of TELEMETRY snippet failed" +gssh "sudo config save -y" || log "WARN: config save failed; gNMI settings will not persist a guest reimage" + +# 3) Create the gNMI Linux user inside the guest (PAM/sshd password auth) +# NOTE: parsed with sed, not jq — the vrnetlab launcher container has no jq. +creds_field() { sed -n 's/.*"'"$1"'"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' "$CREDS" 2>/dev/null | head -1; } +CREDS=/etc/sonic/bootstrap/gnmi_creds.json +if [[ -f "$CREDS" ]]; then + GNMI_USER=$(creds_field username) + GNMI_PASS=$(creds_field password) +fi +if [[ -n "${GNMI_USER:-}" && -n "${GNMI_PASS:-}" ]]; then + gssh "sudo useradd -m -s /bin/bash '$GNMI_USER' 2>/dev/null; echo '$GNMI_USER:$GNMI_PASS' | sudo chpasswd" \ + || die "gNMI user creation in guest failed" + log "created/updated gNMI user $GNMI_USER in guest" +else + log "WARN: no gnmi_creds.json staged; gNMI password auth will fail" +fi + +# 4) Restart gNMI and wait for the port (SONiC VM uses systemd) +gssh "sudo systemctl restart gnmi 2>/dev/null || sudo systemctl restart telemetry 2>/dev/null || true" +for i in $(seq 1 30); do + if gssh "(exec 3<>/dev/tcp/127.0.0.1/8080) 2>/dev/null"; then + log "gNMI port 8080 is listening in guest" + log "completed" + exit 0 + fi + sleep 4 +done + +log "WARN: gNMI port 8080 not confirmed listening in guest after 2 minutes" +exit 1 diff --git a/lab/profiles/sonic-vm/bootstrap/install-gnmi-certs.sh b/lab/profiles/sonic-vm/bootstrap/install-gnmi-certs.sh new file mode 100755 index 00000000..1d0c1f76 --- /dev/null +++ b/lab/profiles/sonic-vm/bootstrap/install-gnmi-certs.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# Idempotently install gNMI TLS certs into /etc/sonic/telemetry (VM variant). +# +# Unlike the sonic-vs variant (which runs INSIDE the SONiC container), this +# script runs inside the guest SONiC VM: the launcher-side init script +# (init-sonic-bootstrap.sh) copies the staged TLS material into the guest and +# invokes this over SSH. Paths and the mandatory .cer suffixes are identical +# to sonic-vs — sonic-telemetry YANG validates cert paths against +# "(/[a-zA-Z0-9_-]+)*/([a-zA-Z0-9_-]+).cer", and GCU validates the ENTIRE +# CONFIG_DB before any patch, so a ".crt" path in TELEMETRY|certs makes every +# GCU write fail with Data Loading Failed. +set -euo pipefail + +dst=/etc/sonic/telemetry +src=${GNMI_CERT_SRC_DIR:-/tmp/agentic-netops-gnmi} +sudo mkdir -p "$dst" +sudo install -m 0600 -T "$src/gnmi.key" "$dst/gnmi.key" +sudo install -m 0644 -T "$src/gnmi.crt" "$dst/gnmi.cer" +sudo install -m 0644 -T "$src/ca.crt" "$dst/ca.cer" + +echo "[install-gnmi-certs] installed certs to $dst" diff --git a/lab/profiles/sonic-vm/profile.yaml b/lab/profiles/sonic-vm/profile.yaml index 8a6ec2e5..a4a5df73 100644 --- a/lab/profiles/sonic-vm/profile.yaml +++ b/lab/profiles/sonic-vm/profile.yaml @@ -1,16 +1,24 @@ # Agentic NetOps SONiC lab profile: sonic-vm (conformance overlay) profile: sonic-vm -kind: linux -image: ghcr.io/sonic-net/sonic:202405@sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb +kind: sonic-vm +image: local/sonic-vm:202505 runtime: privileged: true devices: - /dev/kvm bootstrap: - path: lab/profiles/sonic-vs/bootstrap + path: lab/profiles/sonic-vm/bootstrap files: - - lab/profiles/sonic-vs/bootstrap/init-sonic-bootstrap.sh - - lab/profiles/sonic-vs/bootstrap/install-gnmi-certs.sh - - lab/profiles/sonic-vs/bootstrap/gnmi_config_db.json + - lab/profiles/sonic-vm/bootstrap/init-sonic-bootstrap.sh + - lab/profiles/sonic-vm/bootstrap/install-gnmi-certs.sh + - lab/profiles/sonic-vm/bootstrap/gnmi_config_db.json notes: | - Requires KVM/nested virtualization. Reuses sonic-vs minimal bootstrap to enable gNMI/TLS and persistence. + Requires KVM/nested virtualization. Deploys lab/topology.sonic-vm.clab.yml + (containerlab native sonic-vm kind, vrnetlab QEMU launcher). Bootstrap is + VM-aware: clab::bootstrap execs init-sonic-bootstrap.sh in the launcher, + which provisions the guest over SSH (TLS certs, TELEMETRY/gNMI enablement + with TLS, gNMI Linux user, persistence via `config save`). Limited to TLS + + gNMI enablement; the EVPN/SRv6 tests program configuration via gNMI during + qualification. Build the image with scripts/build-sonic-vm.sh and keep the + digest pinned in versions.lock.yaml (authoritative; this file is + informational and is not consumed by deploy tooling). diff --git a/lab/topology.sonic-vm.clab.yml b/lab/topology.sonic-vm.clab.yml new file mode 100644 index 00000000..ff13e736 --- /dev/null +++ b/lab/topology.sonic-vm.clab.yml @@ -0,0 +1,130 @@ +name: agentic-netops-fabric +mgmt: + network: agentic-netops-mgmt # external Agentic NetOps-owned Docker network (created by scripts/provision.sh) + ipv4-subnet: 172.31.0.0/16 # reused management subnet (no overlap with pod/service per preflight) + mtu: 9216 + +# Conformance-profile topology (AGENTIC_NETOPS_PROFILE=sonic-vm). Selected by +# scripts/lib/containerlab.sh when the profile is sonic-vm; the fabric nodes are +# full SONiC VMs (containerlab native `sonic-vm` kind, vrnetlab QEMU launcher) +# instead of the containerized sonic-vs image. Node names, mgmt IPs, links, +# clients and labels are identical to lab/topology.clab.yml so that the +# bootstrap, tests, and gNMI qualification see the same fabric. +# +# The sonic-vm kind runs nodes privileged with /dev/kvm (preflight enforces +# KVM for this profile) and boots a full SONiC VM per node (~1-3 min, 2 vCPU / +# 4 GB RAM each by default; tune via QEMU_SMP / QEMU_MEMORY env if needed). +# Persistence: the vrnetlab sonic image declares its own VOLUME /config where +# backup.sh stores the VM's saved config_db.json — it survives container +# restarts but not destroy/recreate. Do NOT add a named mount at /config here: +# docker rejects it with 'Duplicate mount point: /config' (image already owns it). +# +# The image is operator-built via scripts/build-sonic-vm.sh (vrnetlab) and +# pinned by digest in versions.lock.yaml; this file references the local tag +# the build script produces. + +topology: + defaults: + labels: + agentic-netops.owner: agentic-netops + agentic-netops.topology: sonic-evpn + + nodes: + spine01: + kind: sonic-vm + image: local/sonic-vm:202505 + mgmt-ipv4: 172.31.0.11 + labels: + role: spine + asn: "65000" + spine02: + kind: sonic-vm + image: local/sonic-vm:202505 + mgmt-ipv4: 172.31.0.12 + labels: + role: spine + asn: "65000" + leaf01: + kind: sonic-vm + image: local/sonic-vm:202505 + mgmt-ipv4: 172.31.0.21 + labels: + role: leaf + asn: "65101" + leaf02: + kind: sonic-vm + image: local/sonic-vm:202505 + mgmt-ipv4: 172.31.0.22 + labels: + role: leaf + asn: "65102" + client01: + kind: linux + image: localhost:5000/linux-net:202605@sha256:c1df8a372168f5b7d3c2c50265fa99d5eb5b6deacdbb31b327a52376a74c6d7f + mgmt-ipv4: 172.31.0.101 + labels: + role: evpn-client + attach: leaf01 + exec: + # both EVPN clients share the bridged Vlan100 subnet (192.0.2.0/24) so + # Type-2 MAC/IP reachability is exercisable; /31s would put them in + # different subnets and break bridged L2 reachability + - ip link set dev eth1 up + - ip addr add 192.0.2.11/24 dev eth1 + - ip -6 addr add 2001:db8:9::11/64 dev eth1 + client02: + kind: linux + image: localhost:5000/linux-net:202605@sha256:c1df8a372168f5b7d3c2c50265fa99d5eb5b6deacdbb31b327a52376a74c6d7f + mgmt-ipv4: 172.31.0.102 + labels: + role: evpn-client + attach: leaf02 + exec: + - ip link set dev eth1 up + - ip addr add 192.0.2.21/24 dev eth1 + - ip -6 addr add 2001:db8:9::21/64 dev eth1 + srv6-client01: + kind: linux + image: localhost:5000/linux-srv6:202605@sha256:c1df8a372168f5b7d3c2c50265fa99d5eb5b6deacdbb31b327a52376a74c6d7f + mgmt-ipv4: 172.31.0.111 + labels: + role: srv6-client + attach: leaf01 + exec: + - ip link set dev eth1 up + - ip addr add 192.0.2.31/31 dev eth1 + - ip -6 addr add 2001:db8:3::31/127 dev eth1 + srv6-client02: + kind: linux + image: localhost:5000/linux-srv6:202605@sha256:c1df8a372168f5b7d3c2c50265fa99d5eb5b6deacdbb31b327a52376a74c6d7f + mgmt-ipv4: 172.31.0.112 + labels: + role: srv6-client + attach: leaf02 + exec: + - ip link set dev eth1 up + - ip addr add 192.0.2.41/31 dev eth1 + - ip -6 addr add 2001:db8:4::41/127 dev eth1 + + links: + # Spine-leaf fabric (explicit interface mapping) — MTU applies per link + - endpoints: [spine01:eth1, leaf01:eth1] + mtu: 9216 + - endpoints: [spine01:eth2, leaf02:eth1] + mtu: 9216 + - endpoints: [spine02:eth1, leaf01:eth2] + mtu: 9216 + - endpoints: [spine02:eth2, leaf02:eth2] + mtu: 9216 + + # Leaf to EVPN Linux endpoints + - endpoints: [leaf01:eth3, client01:eth1] + mtu: 9216 + - endpoints: [leaf02:eth3, client02:eth1] + mtu: 9216 + + # Leaf to SRv6 Linux endpoints (one per leaf) + - endpoints: [leaf01:eth4, srv6-client01:eth1] + mtu: 9216 + - endpoints: [leaf02:eth4, srv6-client02:eth1] + mtu: 9216 diff --git a/scripts/build-sonic-vm.sh b/scripts/build-sonic-vm.sh new file mode 100755 index 00000000..134064c7 --- /dev/null +++ b/scripts/build-sonic-vm.sh @@ -0,0 +1,234 @@ +#!/usr/bin/env bash +# Build the operator-built sonic-vm conformance image via vrnetlab. +# +# Produces the image pinned in versions.lock.yaml (sonic_images.sonic_vm): +# local/sonic-vm: (and local/sonic-vm:latest) +# +# Grounded build path (srl-labs/vrnetlab, containerlab sonic-vm kind): +# - source artifact: target/sonic-vs.img.gz from a SONiC VS build +# (https://sonic-build.azurewebsites.net/ui/sonic/pipelines, +# https://sonic-net.github.io/SONiC/sonic_latest_images.html) +# - vrnetlab expects it decompressed and renamed sonic-vs-.qcow2 +# (its Makefile globs *.qcow2 and extracts the version from the filename) +# - `make docker-image` produces vrnetlab/sonic_sonic-vs: +# +# Requirements: docker, ~10 GB free disk (image ~3.2 GB after import), git, +# network access. KVM is only needed at RUN time (preflight enforces it). +# +# Usage: +# scripts/build-sonic-vm.sh --image-url \ +# [--version 202505] [--update-pin] [--keep-workdir] +# [--resume ] +# +# --update-pin rewrites the sonic_vm digest (and the matching +# sonic_yang.compatibility entry) in versions.lock.yaml to the freshly built +# image, then runs scripts/lib/verify_pins.sh to validate the lock. +# +# --resume reuses a previous --keep-workdir directory: the partial +# artifact download is continued with HTTP range requests instead of being +# re-downloaded from scratch. +set -euo pipefail + +ROOT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd) +VERSION="202505" +IMAGE_URL="${SONIC_VS_IMAGE_URL:-}" +UPDATE_PIN=0 +KEEP=0 +RESUME_DIR="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --image-url) IMAGE_URL="$2"; shift 2 ;; + --version) VERSION="$2"; shift 2 ;; + --update-pin) UPDATE_PIN=1; shift ;; + --keep-workdir) KEEP=1; shift ;; + --resume) RESUME_DIR="$2"; shift 2 ;; + -h|--help) grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "unknown arg: $1 (see --help)" >&2; exit 2 ;; + esac +done + +command -v docker >/dev/null 2>&1 || { echo "docker not found" >&2; exit 1; } +command -v git >/dev/null 2>&1 || { echo "git not found" >&2; exit 1; } +[[ -n "$IMAGE_URL" ]] || { + echo "ERROR: --image-url is required (or export SONIC_VS_IMAGE_URL)." >&2 + echo "Browse VS-build artifacts at:" >&2 + echo " https://sonic-build.azurewebsites.net/ui/sonic/pipelines" >&2 + echo " https://sonic-net.github.io/SONiC/sonic_latest_images.html" >&2 + exit 2 +} + +echo "[sonic-vm] workdir + vrnetlab checkout" +if [[ -n "$RESUME_DIR" ]]; then + [[ -d "$RESUME_DIR" ]] || { echo "ERROR: --resume dir not found: $RESUME_DIR" >&2; exit 1; } + WORK="$RESUME_DIR" + KEEP=1 # never delete a resumed workdir automatically + if [[ ! -d "$WORK/vrnetlab" ]]; then + git clone --depth 1 https://github.com/srl-labs/vrnetlab "$WORK/vrnetlab" + fi + echo "[sonic-vm] resuming in $WORK (artifact.bin at $(du -h "$WORK/artifact.bin" 2>/dev/null | cut -f1) so far)" +else + # Work on real disk, not /tmp: /tmp is often tmpfs (RAM-backed) and the Azure + # artifact is ~23 GB zipped / ~9 GB uncompressed — it would exhaust a RAM-backed + # tmpfs. Override the parent dir with SONIC_VM_WORKDIR if needed. + WORK_PARENT=${SONIC_VM_WORKDIR:-/var/tmp} + mkdir -p "$WORK_PARENT" + WORK=$(mktemp -d "${WORK_PARENT}/sonic-vm-build.XXXXXX") + git clone --depth 1 https://github.com/srl-labs/vrnetlab "$WORK/vrnetlab" +fi +trap '[[ "$KEEP" == 1 ]] || rm -rf "$WORK"' EXIT + +# Resumable download with size verification. Azure artifact connections are +# routinely dropped mid-transfer (observed: server closed at 12.1G of 23.4G and +# curl exited 0), so: loop with HTTP range resume (-C -), then verify the +# fetched size against the advertised Content-Length before trusting it. +artifact_download() { + local out="$1" url="$2" expected have rc attempts=0 resumed=0 + + # Local file support: --image-url may point at an already-downloaded + # artifact (curl cannot read filesystem paths as URLs, so short-circuit). + if [[ -f "$url" ]]; then + echo "[sonic-vm] using local artifact: $url ($(du -h "$url" | cut -f1))" + if [[ "$(head -c 2 "$url")" != "PK" ]]; then + # raw gzip (not a ZIP): validate now; ZIPs are validated at extract time + if ! gzip -t "$url" 2>/dev/null; then + echo "ERROR: local artifact is not a valid gzip file and not a ZIP: $url" >&2 + return 1 + fi + echo "[sonic-vm] local gzip integrity OK" + fi + cp -f "$url" "$out" + return 0 + fi + + # Content-Length must come from a 200/206 response — error pages (405/400) + # carry their own tiny Content-Length (observed: "93") which poisoned the + # size check. Key each length by the status code of its header block. + expected=$(curl -sIL --retry 3 "$url" | tr -d '\r' | awk ' + tolower($1) ~ /^http\// { code = $2 } + tolower($1) == "content-length:" { len[code] = $2 } + END { for (c in len) if (c == 200 || c == 206) { print len[c]; break } }') + have=0; [[ -f "$out" ]] && have=$(stat -c%s "$out") + if [[ -n "$expected" && "$have" -eq "$expected" ]]; then + echo "[sonic-vm] artifact already complete (${have} bytes); skipping download" + return 0 + fi + while :; do + attempts=$((attempts + 1)) + if [[ $attempts -ge 20 ]]; then + echo "ERROR: download failed after $attempts attempts" >&2 + return 1 + fi + have=0; [[ -f "$out" ]] && have=$(stat -c%s "$out") + + if [[ "$have" -gt 0 && "$resumed" -eq 0 ]]; then + # One resume attempt. If the server refuses ranges (curl 33) or rejects + # the ranged GET outright (curl 22 — this artifact host 400s them), + # drop the partial and restart clean; further resume attempts are + # pointless. + echo "[sonic-vm] download attempt $attempts: resuming from byte ${have} of ${expected:-unknown}" + set +e; curl -fL -C - --retry 5 --retry-delay 5 -o "$out" "$url"; rc=$?; set -e + if [[ $rc -eq 0 ]]; then + have=$(stat -c%s "$out" 2>/dev/null || echo 0) + if [[ -z "$expected" || "$have" -eq "$expected" ]]; then return 0; fi + echo "[sonic-vm] WARN: got ${have} bytes, expected ${expected}" >&2 + fi + echo "[sonic-vm] WARN: server does not support range resume (rc=$rc); restarting download from scratch" >&2 + rm -f "$out"; resumed=1 + continue + fi + + echo "[sonic-vm] download attempt $attempts (full transfer)" + set +e; curl -fL --retry 5 --retry-delay 5 -o "$out" "$url"; rc=$?; set -e + if [[ $rc -eq 0 ]]; then + have=$(stat -c%s "$out" 2>/dev/null || echo 0) + if [[ -z "$expected" || "$have" -eq "$expected" ]]; then + return 0 + fi + echo "[sonic-vm] WARN: connection closed early (${have}/${expected} bytes); retrying" >&2 + else + echo "[sonic-vm] WARN: curl failed (rc=$rc); retrying in 5s" >&2 + sleep 5 + fi + done +} + +echo "[sonic-vm] downloading SONiC VS image" +artifact_download "$WORK/artifact.bin" "$IMAGE_URL" + +echo "[sonic-vm] preparing sonic-vs-${VERSION}.qcow2 (vrnetlab naming requirement)" +SRC="$WORK/artifact.bin" +# Azure artifact URLs serve the ENTIRE target/ bundle as a ZIP, not the raw +# image. Detect the ZIP magic and extract only the sonic-vs.img.gz member. +if [[ "$(head -c 2 "$SRC")" == "PK" ]]; then + command -v unzip >/dev/null 2>&1 || { echo "unzip not found; required to open the artifact ZIP" >&2; exit 1; } + # Validate the ZIP central directory BEFORE trusting it — a truncated + # download produces exactly the "End-of-central-directory signature not + # found" failure. --resume re-completes the file instead of failing here. + if ! unzip -tqq "$SRC" >/dev/null 2>&1; then + echo "ERROR: artifact ZIP is corrupt/incomplete (download truncated?)." >&2 + echo "Re-run with: $0 --resume '$WORK' --image-url --version $VERSION --update-pin" >&2 + exit 1 + fi + echo "[sonic-vm] artifact is an Azure ZIP bundle; locating sonic-vs.img.gz member" + MEMBER=$(unzip -l "$SRC" | awk '{print $4}' | grep -E '(^|/)sonic-vs\.img\.gz$' | head -1) + [[ -n "$MEMBER" ]] || { echo "ERROR: no sonic-vs.img.gz inside the artifact ZIP — wrong platform artifact? (need the 'vs' pipeline)" >&2; exit 1; } + echo "[sonic-vm] extracting ${MEMBER}" + unzip -j -o "$SRC" "$MEMBER" -d "$WORK" >/dev/null + SRC="$WORK/sonic-vs.img.gz" +fi +gzip -dc "$SRC" > "$WORK/sonic-vs.img" +mv "$WORK/sonic-vs.img" "$WORK/vrnetlab/sonic/sonic-vs-${VERSION}.qcow2" + +echo "[sonic-vm] building vrnetlab image (this bakes the qcow2 into docker; may take several minutes)" +make -C "$WORK/vrnetlab/sonic" docker-image + +SRC_TAG="vrnetlab/sonic_sonic-vs:${VERSION}" +docker image inspect "$SRC_TAG" >/dev/null 2>&1 || { + # older vrnetlab layouts may name it differently; locate it + SRC_TAG=$(docker images --format '{{.Repository}}:{{.Tag}}' | grep -E '^vrnetlab/.*sonic' | head -1) + [[ -n "$SRC_TAG" ]] || { echo "ERROR: vrnetlab sonic image not found after build" >&2; exit 1; } +} +echo "[sonic-vm] retagging ${SRC_TAG} -> local/sonic-vm:${VERSION}" +docker tag "$SRC_TAG" "local/sonic-vm:${VERSION}" +docker tag "$SRC_TAG" "local/sonic-vm:latest" + +DIGEST=$(docker image inspect "local/sonic-vm:${VERSION}" --format '{{index .RepoDigests 0}}' 2>/dev/null | sed 's/.*@//' || true) +if [[ -z "$DIGEST" ]]; then + # locally built images have no repo digest until pushed; use the image ID + DIGEST="sha256:$(docker image inspect "local/sonic-vm:${VERSION}" --format '{{.Id}}' | sed 's/sha256://')" + echo "[sonic-vm] NOTE: image is local-only; using image ID as digest stand-in: $DIGEST" >&2 + echo "[sonic-vm] push to a registry (or 'docker save | docker load' on the target host) for a real digest." >&2 +fi +echo "[sonic-vm] digest: $DIGEST" +docker images --digests | grep -E 'sonic' || true + +if [[ "$UPDATE_PIN" == 1 ]]; then + echo "[sonic-vm] updating versions.lock.yaml sonic_vm pins" + LOCK="${ROOT_DIR}/versions.lock.yaml" + cp "$LOCK" "${LOCK}.bak" + sed -i "s|^\( image: local/sonic-vm\)$|\1|" "$LOCK" + # sonic_images.sonic_vm.digest + python3 - "$LOCK" "$DIGEST" <<'PY' +import re, sys +lock, digest = sys.argv[1], sys.argv[2] +text = open(lock).read() +old = re.search(r'sonic_vm:\n image: local/sonic-vm\n digest: (sha256:[0-9a-f]{64})', text) +if not old: + sys.exit("sonic_vm pin block not found in versions.lock.yaml") +old_digest = old.group(1) +text = text.replace(old_digest, digest) +open(lock, "w").write(text) +print(f"replaced {old_digest} -> {digest}") +PY + if [[ -x "${ROOT_DIR}/scripts/lib/verify_pins.sh" ]]; then + "${ROOT_DIR}/scripts/lib/verify_pins.sh" || { + echo "[sonic-vm] ERROR: verify_pins failed after pin update; restoring backup" >&2 + mv "${LOCK}.bak" "$LOCK" + exit 1 + } + fi + echo "[sonic-vm] pin updated and verified (backup at ${LOCK}.bak)" +fi + +echo "[sonic-vm] done. Next: scripts/provision.sh --profile sonic-vm" diff --git a/scripts/lib/containerlab.sh b/scripts/lib/containerlab.sh index 1b8adb61..d5dad142 100755 --- a/scripts/lib/containerlab.sh +++ b/scripts/lib/containerlab.sh @@ -3,14 +3,27 @@ set -euo pipefail ROOT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd) -TOPO_FILE="${ROOT_DIR}/lab/topology.clab.yml" MGMT_NET="agentic-netops-mgmt" LABEL_OWNER="agentic-netops" +# Topology file depends on the active profile: the sonic-vm conformance profile +# deploys full SONiC VMs (containerlab native sonic-vm kind) and must not reuse +# the sonic-vs linux-kind topology. AGENTIC_NETOPS_PROFILE is exported by +# scripts/provision.sh; anything else falls back to the fast-profile topology. +clab::topo_file() { + if [[ "${AGENTIC_NETOPS_PROFILE:-sonic-vs}" == "sonic-vm" ]]; then + echo "${ROOT_DIR}/lab/topology.sonic-vm.clab.yml" + else + echo "${ROOT_DIR}/lab/topology.clab.yml" + fi +} +TOPO_FILE=$(clab::topo_file) + clab::require() { command -v containerlab >/dev/null 2>&1 || { echo "missing containerlab" >&2; exit 1; }; } clab::deploy() { clab::require + TOPO_FILE=$(clab::topo_file) echo "[clab] ensuring external Docker network ${MGMT_NET} exists (subnet-correct)" # Delegate to the shared helper so the subnet is always user-configured # (172.31.0.0/16), which containerlab requires for explicit per-node mgmt IPs. @@ -33,6 +46,7 @@ clab::deploy() { clab::inspect() { clab::require + TOPO_FILE=$(clab::topo_file) # containerlab >=0.7x uses -f/--format (not -o) for output format selection containerlab inspect -t "${TOPO_FILE}" --format json } @@ -79,18 +93,32 @@ clab::bootstrap() { # authenticates via the local user + sshd (see init-sonic-bootstrap.sh 2b), # so the CONFIG_DB table was never needed. docker cp "$bdir/gnmi_config_db.json" "$c:/etc/sonic/bootstrap/gnmi_config_db.json" - docker exec "$c" bash -lc "jq -n --arg u '${GNMI_USER}' --arg p '${GNMI_PASS}' '{username:\$u,password:\$p}' > /etc/sonic/bootstrap/gnmi_creds.json" + # Inject runtime-generated credentials (not static; generated per lab). + # Generated on the HOST and docker-cp'd: the sonic-vs image ships jq but + # the vrnetlab sonic-vm launcher does not ("jq: command not found" made + # the whole bootstrap fail there). + local creds_tmp + creds_tmp=$(mktemp) + jq -n --arg u "${GNMI_USER}" --arg p "${GNMI_PASS}" '{username:$u,password:$p}' > "$creds_tmp" + docker cp "$creds_tmp" "$c:/etc/sonic/bootstrap/gnmi_creds.json" + rm -f "$creds_tmp" # start.sh generates /etc/sonic/config_db.json as the first step of its # boot chain; init-sonic-bootstrap merges into that file and would fail # (or merge into nothing) if it runs too early. Wait it out (observed # 2026-09-01 racing the boot by minutes). - local wi - for wi in $(seq 1 60); do - docker exec "$c" test -s /etc/sonic/config_db.json 2>/dev/null && break - sleep 5 - done - if ! docker exec "$c" test -s /etc/sonic/config_db.json; then - echo "[clab] bootstrap: ${node} config_db.json never appeared; continuing anyway" >&2 + # + # sonic-vm: this wait would burn 5 min per node — the docker-exec target is + # the vrnetlab QEMU launcher, not the SONiC guest, and never has + # /etc/sonic/config_db.json. The VM bootstrap waits for guest SSH itself. + if [[ "$profile" != "sonic-vm" ]]; then + local wi + for wi in $(seq 1 60); do + docker exec "$c" test -s /etc/sonic/config_db.json 2>/dev/null && break + sleep 5 + done + if ! docker exec "$c" test -s /etc/sonic/config_db.json; then + echo "[clab] bootstrap: ${node} config_db.json never appeared; continuing anyway" >&2 + fi fi docker exec "$c" bash -lc "chmod +x /etc/sonic/bootstrap/*.sh && /etc/sonic/bootstrap/init-sonic-bootstrap.sh" # VERIFY the two things gNMI password auth actually needs, and repair them if @@ -107,6 +135,10 @@ clab::bootstrap() { # :8080 -- and no node had any uid>=1000 user at all. gnmic could not subscribe, # Prometheus held only scrape metadata, and every Grafana panel read "No data". # The whole observability stack deployed green and showed nothing. + # For sonic-vm the gNMI user/sshd live in the VM guest (provisioned over + # SSH by init-sonic-bootstrap.sh); the launcher container has neither, so + # the repair below would be a no-op at best and misleading at worst. + if [[ "$profile" != "sonic-vm" ]]; then docker exec "$c" bash -lc ' CREDS=/etc/sonic/bootstrap/gnmi_creds.json U=$(jq -r ".username // empty" "$CREDS" 2>/dev/null) @@ -121,6 +153,7 @@ clab::bootstrap() { supervisorctl start sshd >/dev/null 2>&1 || /usr/sbin/sshd >/dev/null 2>&1 || true fi ' || true + fi docker exec "$c" touch /etc/agentic-netops/.bootstrapped echo "[clab] bootstrap: ${node} done" done @@ -147,13 +180,16 @@ clab::bootstrap() { clab::destroy() { clab::require + TOPO_FILE=$(clab::topo_file) echo "[clab] destroying ${TOPO_FILE}" containerlab destroy -t "${TOPO_FILE}" --cleanup || true - # Remove the per-node named volumes that persist SONiC /etc/sonic. Containerlab + # Remove the per-node named volumes that persist SONiC config. Containerlab # does not always reclaim topology-declared named volumes, so we drop them - # explicitly by their deterministic names (only the ones this lab owns). + # explicitly by their deterministic names (only the ones this lab owns): + # -etc-sonic for the sonic-vs profile, -vm-config for sonic-vm. local v - for v in agentic-netops-spine01-etc-sonic agentic-netops-spine02-etc-sonic agentic-netops-leaf01-etc-sonic agentic-netops-leaf02-etc-sonic; do + for v in agentic-netops-spine01-etc-sonic agentic-netops-spine02-etc-sonic agentic-netops-leaf01-etc-sonic agentic-netops-leaf02-etc-sonic \ + agentic-netops-spine01-vm-config agentic-netops-spine02-vm-config agentic-netops-leaf01-vm-config agentic-netops-leaf02-vm-config; do docker volume rm "$v" >/dev/null 2>&1 || true done # Verify teardown leaves no owned lab containers or volumes @@ -165,7 +201,7 @@ clab::destroy() { fi # Check for owned volumes (persistent /etc/sonic) local vol_left - vol_left=$(docker volume ls -q | grep -E '^agentic-netops-.*-etc-sonic$' || true) + vol_left=$(docker volume ls -q | grep -E '^agentic-netops-.*-(etc-sonic|vm-config)$' || true) if [[ -n "$vol_left" ]]; then echo "[clab] WARN: leftover Agentic NetOps volumes not removed:\n$vol_left" >&2 exit 1 diff --git a/scripts/lib/persistence.sh b/scripts/lib/persistence.sh index 18abef38..09ae0c40 100755 --- a/scripts/lib/persistence.sh +++ b/scripts/lib/persistence.sh @@ -6,7 +6,9 @@ # itself still goes through GCU apply/remove patches. srv6_force_clean() { local c=$1 - docker exec "$c" bash -c 'redis-cli -n 4 --scan --pattern "SRV6_MY_LOCATORS*" | xargs -r redis-cli -n 4 del >/dev/null; redis-cli -n 4 --scan --pattern "SRV6_MY_SIDS*" | xargs -r redis-cli -n 4 del >/dev/null; config save -y >/dev/null 2>&1' + # shellcheck source=../../tests/integration/lib/node_exec.sh + source "$(dirname -- "${BASH_SOURCE[0]}")/../../tests/integration/lib/node_exec.sh" + node_run "$c" 'redis-cli -n 4 --scan --pattern "SRV6_MY_LOCATORS*" | xargs -r redis-cli -n 4 del >/dev/null; redis-cli -n 4 --scan --pattern "SRV6_MY_SIDS*" | xargs -r redis-cli -n 4 del >/dev/null; sudo config save -y >/dev/null 2>&1' } # Persistence qualification helper: program a value, restart SONiC containers, @@ -57,7 +59,7 @@ node_for_ip() { gcu_apply() { local c=$1 patch=$2 - printf '%s' "$patch" | docker exec -i "$c" bash -c 'cat > /tmp/.persist_patch.json && config apply-patch -f CONFIGDB -p /tmp/.persist_patch.json 2>/dev/null >/dev/null' + printf '%s' "$patch" | node_run_i "$c" 'cat > /tmp/.persist_patch.json && sudo config apply-patch -f CONFIGDB -p /tmp/.persist_patch.json 2>/dev/null >/dev/null' } gcu_write() { @@ -75,7 +77,7 @@ PY # Persist to /etc/sonic/config_db.json: start.sh reloads CONFIG_DB from the file # on every boot (configdb-load.sh), so a redis-only write would be lost — the # persisted file is exactly what T014 is about ("Ensure /etc/sonic persists"). - docker exec "$node" bash -c 'config save -y >/dev/null 2>&1' || die "config save failed on $node" + node_run "$node" 'sudo config save -y >/dev/null 2>&1' || die "config save failed on $node" } gcu_delete() { @@ -86,7 +88,7 @@ gcu_delete() { {"op":"remove","path":"/SRV6_MY_LOCATORS"} ]' gcu_apply "$node" "$patch" || note "cleanup patch failed on $node (leftover ${LOC_NAME} tolerable)" - docker exec "$node" bash -c 'config save -y >/dev/null 2>&1' || true + node_run "$node" 'sudo config save -y >/dev/null 2>&1' || true } gnmi_get() { @@ -131,6 +133,18 @@ restart_sonic_containers() { local ids; ids=$(docker ps -q --filter "name=${CLAB_PREFIX}") || true if [[ -z "$ids" ]]; then echo "[persist] no Agentic NetOps fabric containers found" >&2; return 1; fi local c + # sonic-vm: the launcher is a QEMU wrapper — the persistence surface is the + # GUEST's /etc/sonic/config_db.json. `config reload` re-reads exactly that + # file into CONFIG_DB (redis is flushed and swss re-runs), exercising the + # same persisted-path guarantee as the supervisord restart below. + if node_is_vm "$(docker ps --format '{{.Names}}' | grep "^${CLAB_PREFIX}" | head -n1)"; then + for c in $(docker ps --format '{{.Names}}' | grep "^${CLAB_PREFIX}"); do + node_is_vm "$c" || continue + echo "[persist] config reload in guest of $c" + node_run "$c" 'sudo config reload -y' >/dev/null 2>&1 || echo "[persist] WARN: config reload failed on $c" >&2 + done + return 0 + fi # stop supervisord cleanly inside every node: all programs (redis, swss, # bgpd, telemetry) go down — this is the destructive part of the restart. # Wait for the old process to fully release its HTTP socket before any diff --git a/tests/integration/cycles_runner.sh b/tests/integration/cycles_runner.sh index 647d73e6..1494851b 100755 --- a/tests/integration/cycles_runner.sh +++ b/tests/integration/cycles_runner.sh @@ -134,9 +134,14 @@ echo "[cycles] ===== off-from-partial =====" | tee -a "$PROOFS/cycles.run.log" # T077 off-from-PARTIAL: the provision here must FAIL partway so off.sh cleans a # genuinely partial state. sonic-vs now passes the capability gate (re-pinned v2 # gNMI image), so the old sonic-vs call degraded into a full-provision test; the -# documented gate-fail path is the sonic-vm profile (capability gate fails → -# exit=1 → off.sh cleans the partial state) — the same shape off-conformance -# exercises below. (Fix 2026-09-01, operator reconciliation.) +# documented gate-fail path is the sonic-vm profile — provision fails whenever +# the operator-built vrnetlab image (local/sonic-vm, versions.lock.yaml +# sonic_images.sonic_vm) is absent on the host (exit=1 → off.sh cleans the +# partial state), the same shape off-conformance exercises below. On a host +# where that image HAS been built and passes the gate, this cycle degrades to +# an extra full-provision/off pass, which the runner tolerates (exit logged, +# not fatal). (Fix 2026-09-01, operator reconciliation; updated when the +# sonic-vm bootstrap/topology wiring landed.) run_provision "$PROOFS/partial-provision.log" sonic-vm run_off "$PROOFS/off-from-partial.log" run_off_noop "$PROOFS/off-from-partial-noop.log" diff --git a/tests/integration/evpn_srv6_suite.sh b/tests/integration/evpn_srv6_suite.sh index 8520e3f2..85b92d57 100755 --- a/tests/integration/evpn_srv6_suite.sh +++ b/tests/integration/evpn_srv6_suite.sh @@ -14,6 +14,11 @@ # Every test asserts reply/state *content*; none decides pass/fail on an empty result. set -euo pipefail +# node-side op dispatch (docker exec vs SSH-into-guest for sonic-vm) +# shellcheck source=lib/node_exec.sh +source "$(dirname -- "${BASH_SOURCE[0]}")/lib/node_exec.sh" + + GNMIC_BIN=${GNMIC_BIN:-gnmic} GNMI_USER=${GNMI_USER:-admin} GNMI_PASS=${GNMI_PASS:-admin} @@ -63,7 +68,7 @@ node_for_ip() { gcu_apply() { local c=$1 patch=$2 - printf '%s' "$patch" | docker exec -i "$c" bash -c 'cat > /tmp/.gate_patch.json && config apply-patch -f CONFIGDB -p /tmp/.gate_patch.json 2>/dev/null >/dev/null' + printf '%s' "$patch" | node_run_i "$c" 'cat > /tmp/.gate_patch.json && sudo config apply-patch -f CONFIGDB -p /tmp/.gate_patch.json 2>/dev/null >/dev/null' return $? } @@ -74,7 +79,7 @@ gcu_apply() { # itself still goes through GCU apply/remove patches. srv6_force_clean() { local c=$1 - docker exec "$c" bash -c 'redis-cli -n 4 --scan --pattern "SRV6_MY_LOCATORS*" | xargs -r redis-cli -n 4 del >/dev/null; redis-cli -n 4 --scan --pattern "SRV6_MY_SIDS*" | xargs -r redis-cli -n 4 del >/dev/null; config save -y >/dev/null 2>&1' + node_run "$c" 'redis-cli -n 4 --scan --pattern "SRV6_MY_LOCATORS*" | xargs -r redis-cli -n 4 del >/dev/null; redis-cli -n 4 --scan --pattern "SRV6_MY_SIDS*" | xargs -r redis-cli -n 4 del >/dev/null; sudo config save -y >/dev/null 2>&1' } @@ -202,14 +207,34 @@ seg6_witness() { local node probe out node=$(node_for_ip "${t%%:*}") [[ -n "$node" ]] || die "$label: cannot resolve containerlab node for $t" + # Kernel seg6 lwtunnel is required for this witness. The sonic-vs image + # runs on the container host kernel (supports seg6); the sonic-vm guest + # runs the SONiC Debian kernel where LWTUNNEL/seg6 is disabled — "ip + # route ... encap seg6" is silently accepted but the encap is dropped. + # Probe once and skip the kernel witness with an explicit note when the + # node kernel cannot host it (the gNMI CONFIG_DB SRv6 witnesses above + # still cover SRv6 control-plane state on both profiles). + if [[ -z "${SEG6_KERNEL_OK:-}" ]]; then + local probe_out + probe_out=$(node_run "$node" "sudo ip -6 route replace 2001:db8:7fff:ffff::/64 encap seg6 mode encap segs fc00::1 dev lo 2>/dev/null && sudo ip -6 route show 2001:db8:7fff:ffff::/64; sudo ip -6 route del 2001:db8:7fff:ffff::/64 >/dev/null 2>&1" 2>/dev/null || true) + if grep -q 'encap seg6' <<<"$probe_out"; then + SEG6_KERNEL_OK=1 + else + SEG6_KERNEL_OK=0 + note "$t: kernel seg6 lwtunnel unavailable — skipping $label kernel witness (control-plane SRv6 state asserted via gNMI)" + continue + fi + elif [[ "$SEG6_KERNEL_OK" == "0" ]]; then + continue + fi # WITNESS_TAG is a decimal epoch; fold it into a valid 16-bit hextet local tag_hex tag_hex=$(printf '%04x' $(( WITNESS_TAG % 65536 ))) probe="2001:db8:7f:${tag_hex}::/64" - docker exec "$node" ip -6 route replace "$probe" encap seg6 mode "$mode" segs "$segs" dev lo \ + node_run "$node" "sudo ip -6 route replace '$probe' encap seg6 mode '$mode' segs '$segs' dev lo" \ || die "$label $t: kernel rejected seg6 $mode encap route" - out=$(docker exec "$node" ip -6 route show "$probe") - docker exec "$node" ip -6 route del "$probe" >/dev/null 2>&1 || true + out=$(node_run "$node" "sudo ip -6 route show '$probe'") + node_run "$node" "sudo ip -6 route del '$probe'" >/dev/null 2>&1 || true grep -qF "$expect" <<<"$out" || die "$label $t: seg6 state does not show '$expect' (got: $out)" note "$t: $label asserted (seg6 $mode programmed and read back)" done @@ -256,6 +281,10 @@ evpn_session_up() { local na nb na=$(node_for_ip "$a"); nb=$(node_for_ip "$b") [[ -n "$na" && -n "$nb" ]] || die "EVPN: cannot resolve containerlab nodes for $a/$b" + if node_is_vm "$na"; then + evpn_session_up_vm "$na" "$nb" + return $? + fi # The peering runs over the management addresses; SONiC renders every interface # from CONFIG_DB and leaves eth0 admin-down once intent-config is loaded, so the # witness owns its interface state (idempotent bring-up with address intact). @@ -267,9 +296,21 @@ evpn_session_up() { for n in "$na" "$nb"; do docker exec "$n" supervisorctl start bgpd >/dev/null 2>&1 || true done - # wait for the bgpd processes to exist (spawning can lag on a freshly booted, - # CPU-saturated node — racing it leaves the session stuck in Active) - local i up=0 + evpn_wait_bgpd "$na" "$nb" + docker exec "$na" vtysh -c 'configure terminal' -c 'router bgp 65101' \ + -c 'no bgp ebgp-requires-policy' -c "neighbor $b remote-as 65102" \ + -c 'address-family l2vpn evpn' -c "neighbor $b activate" >/dev/null 2>&1 \ + || die "EVPN: bgpd on $na rejected EVPN AF configuration" + docker exec "$nb" vtysh -c 'configure terminal' -c 'router bgp 65102' \ + -c 'no bgp ebgp-requires-policy' -c "neighbor $a remote-as 65101" \ + -c 'address-family l2vpn evpn' -c "neighbor $a activate" >/dev/null 2>&1 \ + || die "EVPN: bgpd on $nb rejected EVPN AF configuration" + evpn_wait_established "$na" "$nb" +} + +# evpn_wait_bgpd — wait until bgpd exists on both sonic-vs nodes +evpn_wait_bgpd() { + local na=$1 nb=$2 n i up=0 for i in $(seq 1 30); do up=1 for n in "$na" "$nb"; do @@ -279,15 +320,11 @@ evpn_session_up() { sleep 2 done [[ $up -eq 1 ]] || die "EVPN: bgpd did not come up on $na/$nb" - docker exec "$na" vtysh -c 'configure terminal' -c 'router bgp 65101' \ - -c 'no bgp ebgp-requires-policy' -c "neighbor $b remote-as 65102" \ - -c 'address-family l2vpn evpn' -c "neighbor $b activate" >/dev/null 2>&1 \ - || die "EVPN: bgpd on $na rejected EVPN AF configuration" - docker exec "$nb" vtysh -c 'configure terminal' -c 'router bgp 65102' \ - -c 'no bgp ebgp-requires-policy' -c "neighbor $a remote-as 65101" \ - -c 'address-family l2vpn evpn' -c "neighbor $a activate" >/dev/null 2>&1 \ - || die "EVPN: bgpd on $nb rejected EVPN AF configuration" - # poll for Established; nudge the session past connect backoff halfway through +} + +# evpn_wait_established — poll for Established; nudge past backoff +evpn_wait_established() { + local na=$1 nb=$2 i for i in $(seq 1 20); do if docker exec "$na" vtysh -c 'show bgp l2vpn evpn summary json' 2>/dev/null | grep -q '"state":"Established"'; then return 0 @@ -300,11 +337,66 @@ evpn_session_up() { done } +# evpn_session_up_vm — sonic-vm flavour. +# The guests have no shared management network (each sits behind its own +# qemu-slirp), so the eBGP EVPN session is brought up across the real fabric: +# leaf01 eth1 — spine01 (L3 forward) — leaf02 eth1. The spine needs no BGP — +# kernel forwarding between the two /30s carries the peering TCP. +# Peer addresses: na=10.2.0.1 (bgp 65101), nb=10.2.0.6 (bgp 65102). +# NOTE: NOT 10.1.0.x — every sonic-vm guest boots with Loopback0=10.2.0.1/32 +# (image default config), which makes 10.2.0.1 a local address on every node: +# the kernel martian-drops packets carrying it as source. +evpn_session_up_vm() { + local na=$1 nb=$2 ns n + ns=$(docker ps --format '{{.Names}}' | grep "^${CLAB_PREFIX}" | grep spine | head -n1) + [[ -n "$ns" ]] || die "EVPN-VM: cannot resolve spine node" + node_run "$ns" "sudo ip addr replace 10.2.0.2/30 dev eth1 2>/dev/null || true; sudo ip addr replace 10.2.0.5/30 dev eth2 2>/dev/null || true; sudo ip link set eth1 up; sudo ip link set eth2 up" || true + node_run "$na" "sudo ip addr replace 10.2.0.1/30 dev eth1; sudo ip link set eth1 up" || true + node_run "$nb" "sudo ip addr replace 10.2.0.6/30 dev eth1; sudo ip link set eth1 up" || true + node_run "$na" 'ping -q -c1 -W2 10.2.0.2 >/dev/null' || die "EVPN-VM: leaf01 cannot reach spine over eth1" + node_run "$nb" 'ping -q -c1 -W2 10.2.0.5 >/dev/null' || die "EVPN-VM: leaf02 cannot reach spine over eth1" + # static routes so the leaves can reach each other's peering subnet via the spine + node_run "$na" 'sudo ip route replace 10.2.0.4/30 via 10.2.0.2 dev eth1' || true + node_run "$nb" 'sudo ip route replace 10.2.0.0/30 via 10.2.0.5 dev eth1' || true + node_run "$nb" 'ping -q -c1 -W3 10.2.0.1 >/dev/null' || die "EVPN-VM: leaf02 cannot route to leaf01 via spine" + for n in "$na" "$nb"; do + node_run "$n" 'sudo systemctl start bgp >/dev/null 2>&1' || true + done + # The image boots bgpd with AS 65100 and router-id 10.1.0.1 (Loopback0) on EVERY + # node — peering must reuse the running AS (iBGP) and set distinct router-ids or + # the OPEN clash fails with Bad BGP Identifier. + # wait for the bgpd to exist inside each guest's bgp container + local i up=0 + for i in $(seq 1 30); do + up=1 + for n in "$na" "$nb"; do + node_run "$n" 'pgrep -x bgpd >/dev/null' || up=0 + done + [[ $up -eq 1 ]] && break + sleep 2 + done + [[ $up -eq 1 ]] || die "EVPN-VM: bgpd did not come up on $na/$nb" + node_run "$na" "sudo vtysh -c 'configure terminal' -c 'router bgp 65100' -c 'bgp router-id 10.2.0.1' -c 'neighbor 10.2.0.6 remote-as 65100' -c 'address-family l2vpn evpn' -c 'neighbor 10.2.0.6 activate'" >/dev/null 2>&1 \ + || die "EVPN-VM: bgpd on $na rejected EVPN AF configuration" + node_run "$nb" "sudo vtysh -c 'configure terminal' -c 'router bgp 65100' -c 'bgp router-id 10.2.0.6' -c 'neighbor 10.2.0.1 remote-as 65100' -c 'address-family l2vpn evpn' -c 'neighbor 10.2.0.1 activate'" >/dev/null 2>&1 \ + || die "EVPN-VM: bgpd on $nb rejected EVPN AF configuration" + for i in $(seq 1 20); do + if node_run "$na" 'vtysh -c "show bgp l2vpn evpn summary json"' 2>/dev/null | grep -q '"state":"Established"'; then + return 0 + fi + (( i == 10 )) && { + node_run "$na" 'vtysh -c "clear bgp *"' >/dev/null 2>&1 || true + node_run "$nb" 'vtysh -c "clear bgp *"' >/dev/null 2>&1 || true + } + sleep 4 + done +} + # assert EVPN session Established with real content from `show bgp l2vpn evpn summary json` assert_evpn_established() { local label=$1 node=$2 peer=$3 local out - out=$(docker exec "$node" vtysh -c 'show bgp l2vpn evpn summary json' 2>&1) + out=$(node_run "$node" 'vtysh -c "show bgp l2vpn evpn summary json"' 2>&1) grep -q '"state":"Established"' <<<"$out" \ || die "$label: L2VPN EVPN session to $peer not Established (output: $out)" grep -q '"afiSafiName":"l2vpn-evpn"' <<<"$out" || true # json key varies; Established is the assert @@ -331,7 +423,7 @@ EVPN_Type5() { # in the table dump the way SONiC's bgpd exposes it) local node out node=$(node_for_ip "${tgts[0]%%:*}") - out=$(docker exec "$node" vtysh -c 'show bgp l2vpn evpn json' 2>&1) + out=$(node_run "$node" 'vtysh -c "show bgp l2vpn evpn json"' 2>&1) jq -e . >/dev/null 2>&1 <<<"$out" || die "EVPN-Type5: EVPN RIB not parsable as JSON: $out" assert_evpn_established "EVPN-Type5 (IP prefix AF)" "$node" "${tgts[1]%%:*}" } diff --git a/tests/integration/lib/node_exec.sh b/tests/integration/lib/node_exec.sh new file mode 100644 index 00000000..48b27564 --- /dev/null +++ b/tests/integration/lib/node_exec.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# node_exec.sh — dispatch a command into the SONiC node from a suite. +# +# sonic-vs profile: the container IS the SONiC switch; docker exec works. +# sonic-vm profile: the container is the vrnetlab QEMU *launcher* and the +# SONiC OS runs inside the guest VM — config/redis-cli/ip exist only in the +# guest, so node-side operations route over SSH (vrnetlab forwards guest :22 +# to the launcher's mgmt address; credentials come from the container env +# USERNAME/PASSWORD which vrnetlab's bootstrap set on the guest). + +# node_is_vm — true when the container is a sonic-vm launcher +node_is_vm() { + [[ "$(docker inspect "$1" --format '{{index .Config.Labels "clab-node-kind"}}' 2>/dev/null)" == "sonic-vm" ]] +} + +# node_guest_addr — the launcher's mgmt IP (guest SSH entry point) +node_guest_addr() { + docker inspect "$1" --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' +} + +# node_run — run in the SONiC node (no stdin) +node_run() { + local c=$1; shift + if node_is_vm "$c"; then + local ip + ip=$(node_guest_addr "$c") + sshpass -p "${GUEST_PASS:-admin}" ssh -o StrictHostKeyChecking=no \ + -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR \ + "${GUEST_USER:-admin}@${ip}" "$*" + else + docker exec "$c" bash -c "$*" + fi +} + +# node_run_i — run with stdin piped through +node_run_i() { + local c=$1; shift + if node_is_vm "$c"; then + local ip + ip=$(node_guest_addr "$c") + sshpass -p "${GUEST_PASS:-admin}" ssh -o StrictHostKeyChecking=no \ + -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR \ + "${GUEST_USER:-admin}@${ip}" "$*" + else + docker exec -i "$c" bash -c "$*" + fi +} diff --git a/tests/integration/sonic_gnmi_suite.sh b/tests/integration/sonic_gnmi_suite.sh index 78759752..95597f95 100755 --- a/tests/integration/sonic_gnmi_suite.sh +++ b/tests/integration/sonic_gnmi_suite.sh @@ -11,6 +11,10 @@ # documented limitation, not asserted here. set -euo pipefail +# node-side op dispatch (docker exec vs SSH-into-guest for sonic-vm) +# shellcheck source=lib/node_exec.sh +source "$(dirname -- "${BASH_SOURCE[0]}")/lib/node_exec.sh" + GNMIC_BIN=${GNMIC_BIN:-gnmic} GNMI_USER=${GNMI_USER:-admin} GNMI_PASS=${GNMI_PASS:-admin} @@ -69,7 +73,7 @@ node_for_ip() { # gcu_apply — apply a JSON patch to CONFIG_DB through GCU gcu_apply() { local c=$1 patch=$2 - printf '%s' "$patch" | docker exec -i "$c" bash -c 'cat > /tmp/.gate_patch.json && config apply-patch -f CONFIGDB -p /tmp/.gate_patch.json 2>/dev/null >/dev/null' + printf '%s' "$patch" | node_run_i "$c" 'cat > /tmp/.gate_patch.json && sudo config apply-patch -f CONFIGDB -p /tmp/.gate_patch.json 2>/dev/null >/dev/null' return $? } @@ -80,7 +84,7 @@ gcu_apply() { # itself still goes through GCU apply/remove patches. srv6_force_clean() { local c=$1 - docker exec "$c" bash -c 'redis-cli -n 4 --scan --pattern "SRV6_MY_LOCATORS*" | xargs -r redis-cli -n 4 del >/dev/null; redis-cli -n 4 --scan --pattern "SRV6_MY_SIDS*" | xargs -r redis-cli -n 4 del >/dev/null; config save -y >/dev/null 2>&1' + node_run "$c" 'redis-cli -n 4 --scan --pattern "SRV6_MY_LOCATORS*" | xargs -r redis-cli -n 4 del >/dev/null; redis-cli -n 4 --scan --pattern "SRV6_MY_SIDS*" | xargs -r redis-cli -n 4 del >/dev/null; sudo config save -y >/dev/null 2>&1' } diff --git a/versions.lock.yaml b/versions.lock.yaml index c2ab580e..04573ea8 100644 --- a/versions.lock.yaml +++ b/versions.lock.yaml @@ -97,7 +97,7 @@ sonic_images: artifact ships no gNMI telemetry server. sonic_vm: image: local/sonic-vm - digest: sha256:b2c77f0a1426e7a93cad0b191f0c04dcf98b3f4b7467348ed550f4b1a706e3d5 + digest: sha256:5f3b6c503d67d0b166beba1aedc9e2ccd46b5723f4f932b9a64d8accdb76edd2 notes: Conformance profile; build via vrnetlab; requires KVM; digest is operator-built # --------------------------------------------------------------------------- @@ -159,7 +159,7 @@ sonic_yang: - image: localhost:5000/sonic-vs-gnmi:202505-v1@sha256:d1043aed28c98071c997a46d7e9e47823abacb06c31c068183541f8b5b5529e8 oc_version: openconfig@f3443414 native_version: sonic_yang@cfc766e1 - - image: local/sonic-vm@sha256:b2c77f0a1426e7a93cad0b191f0c04dcf98b3f4b7467348ed550f4b1a706e3d5 + - image: local/sonic-vm@sha256:5f3b6c503d67d0b166beba1aedc9e2ccd46b5723f4f932b9a64d8accdb76edd2 oc_version: openconfig@f3443414 native_version: sonic_yang@cfc766e1 notes: >