diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/Readme.md b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/Readme.md new file mode 100644 index 000000000..bf7b630fd --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/Readme.md @@ -0,0 +1,204 @@ +# **Air-Gapped Agentic AI: Orchestrating Local SLMs and IoT on Edge Kubernetes** + +- [**MicroHack Introduction**](#microhack-introduction) +- [**What You'll Build**](#what-youll-build) +- [**MicroHack Context & Scenario**](#microhack-context--scenario) +- [**Architecture**](#architecture) +- [**Objectives**](#objectives) +- [**General Prerequisites & Facilitator Guidance**](#general-prerequisites--facilitator-guidance) +- [**MicroHack Challenges**](#microhack-challenges) +- [**Contributors**](#contributors) + +--- + +## MicroHack Introduction + +Industrial, manufacturing, defense, and critical-infrastructure organizations increasingly require **sovereign edge intelligence**: the ability to ingest high-frequency OT sensor streams, reason over local Standard Operating Procedures (SOPs), and execute low-latency autonomous corrective actions **even when severed from the public cloud**. + +This **Ops & Architecture-focused MicroHack** walks you through building and operating a disconnected, end-to-end **Agentic AI + Industrial IoT workflow** running entirely on **Azure Arc-enabled Kubernetes**. You will combine **Azure IoT Operations** (as the unified edge MQTT and dataflow plane) with **Foundry Local** (`Microsoft.Foundry` Arc extension serving quantized CPU Small Language Models via Kubernetes Gateway API and Istio) and a prebuilt **Autonomous Edge Triage Agent** that authenticates locally via Kubernetes **Service Account Tokens (SAT)**. + +Finally, you will **"Cut the Cord"** using strict Kubernetes network isolation policies to prove that your edge AI agent continues to detect anomalies, reason against local runbooks, and actuate PLC safety interlocks with **zero outbound cloud traffic**. + +> [!TIP] +> **No application coding required.** All simulator and agent workloads are provided as ready-to-deploy Kubernetes manifests. You will spend your time on the **sovereign edge platform, dataflow routing, local model serving, workload identity, and air-gap enforcement layers**. + +--- + +## What You'll Build + +Across five progressive challenges, you will assemble a complete closed-loop industrial edge AI stack on your own isolated Kubernetes cluster: + +| # | Layer / Challenge | Core Technology | What It Produces | +| :--- | :--- | :--- | :--- | +| **01** | **Sovereign Edge Foundation** | Ubuntu 24.04 + K3s + **Azure Arc (`connectedk8s`)** | An Arc-connected Kubernetes cluster with OIDC, Workload Identity, Custom Locations, and kernel `inotify` tuning | +| **02** | **Unified Edge IoT Data Plane** | **Azure IoT Operations (`Microsoft.IoTOperations`)** | Native MQTT Broker + CNC Telemetry Simulator + an AIO Dataflow that filters anomalies to `factory/line1/anomalies` | +| **03** | **Local CPU SLM Serving** | **Foundry Local (`Microsoft.Foundry`)** + Istio Gateway API | A locally cached **`Phi-4-mini-instruct-generic-cpu:5`** ONNX INT4 model served at an internal `/v1/chat/completions` gateway | +| **04** | **Zero-Cloud SAT Auth & Edge Agent** | Kubernetes **Service Account Tokens (SAT)** + Local SOP `ConfigMap` | An autonomous agent pod that authenticates locally (`audience: foundry-local`), reasons over **SOP-CNC-042**, and throttles PLC RPM | +| **05** | **"Cut the Cord" Air-Gap Mode** | Kubernetes `NetworkPolicy` + Critical Fault Injector | Verified sub-8-second emergency safety interlock (`-60%` RPM, `SAFE_HOLD`) while public internet/Azure egress is blocked | + +![Air-Gapped Agentic AI Architecture — Sense, Filter, Reason, and Actuate across Azure Arc-enabled Kubernetes, Azure IoT Operations, and Foundry Local](img/architecture.png) + +--- + +## MicroHack Context & Scenario + +### Sovereign Manufacturing Line Scenario + +Contoso Precision Manufacturing operates a high-speed CNC & Extrusion line in a regulated facility where intermittent WAN links and strict data-sovereignty rules prohibit streaming raw high-frequency machine telemetry to public cloud LLM endpoints for real-time control. + +```mermaid +sequenceDiagram + autonumber + participant PLC as Simulated CNC / Extruder Sensors + participant MQ as Azure IoT Operations (MQTT Broker) + participant DF as AIO Dataflow (Anomaly Filter) + participant AG as Autonomous Edge Agent Pod + participant GW as Istio Internal Gateway (Foundry Local) + participant SLM as Foundry Local ModelDeployment (ONNX GenAI CPU) + + PLC->>MQ: Publish telemetry (factory/line1/telemetry) + MQ->>DF: Stream sensor window (Temp, Vibration, Pressure) + DF->>MQ: Route threshold breach (factory/line1/anomalies) + MQ->>AG: Deliver anomaly event + Note over AG: Read local SOP ConfigMap & Projected SAT Token + AG->>GW: POST /phi-4-mini/v1/chat/completions (Bearer SAT) + GW->>SLM: Route inference + tool definitions + SLM-->>AG: Tool call: trigger_plc_interlock(spindle_rpm_delta=-40%, coolant_boost=true) + AG->>MQ: Publish actuation command (factory/line1/commands) + MQ->>PLC: Execute local closed-loop stabilization (0 ms cloud dependency) +``` + +### Required & Optional Reading + +- [Deployment overview for Foundry Local on Azure Local](https://learn.microsoft.com/azure/azure-sovereign-clouds/private/foundry-local/deploy-overview) +- [Deploy Foundry Local as an Azure Arc extension](https://learn.microsoft.com/azure/azure-sovereign-clouds/private/foundry-local/deploy-foundry-local-arc-extension) +- [Deploy a catalog model on Foundry Local (`ModelDeployment` CRD)](https://learn.microsoft.com/azure/azure-sovereign-clouds/private/foundry-local/how-to-deploy-model) +- [Configure Kubernetes Service Account Token (SAT) authentication for Foundry Local](https://learn.microsoft.com/azure/azure-sovereign-clouds/private/foundry-local/how-to-configure-service-account-token-authentication) +- [Azure IoT Operations overview](https://learn.microsoft.com/azure/iot-operations/overview-iot-operations) +- [Prepare an Azure Arc-enabled Kubernetes cluster for Azure IoT Operations](https://learn.microsoft.com/azure/iot-operations/deploy-iot-ops/howto-prepare-cluster) + +--- + +## Architecture + +Both **Azure IoT Operations (`Microsoft.IoTOperations`)** and **Foundry Local (`Microsoft.Foundry`)** deploy onto **Azure Arc-enabled Kubernetes (`Microsoft.Kubernetes/connectedClusters`, K8s 1.29+)**. + +```mermaid +flowchart TB + subgraph AzureCloud["Azure Control Plane (Connected Bootstrap Phase Only)"] + ArcCP["Azure Arc Control Plane
(connectedClusters & k8s-extensions)"] + ADR["Azure Device Registry &
Schema Registry Storage"] + end + + subgraph EdgeHost["Cloud-Simulated Factory Edge Host (Ubuntu 24.04 + K3s on Standard_D16s_v5)"] + subgraph K8sCluster["Azure Arc-Enabled Kubernetes Cluster"] + subgraph NS_AIO["Namespace: azure-iot-operations"] + Broker["AIO Native MQTT Broker
(aio-broker)"] + Dataflow["AIO Dataflow Engine
(Telemetry -> Anomalies)"] + end + + subgraph NS_Istio["Namespace: istio-system & cert-manager"] + Gateway["Kubernetes Gateway API
(Istio inference-internal-gateway)"] + CertMgr["azure-cert-manager
(cert-manager + trust-manager)"] + end + + subgraph NS_Foundry["Namespace: foundry-local-operator"] + FLOp["Foundry Local Inference Operator
(Microsoft.Foundry Extension)"] + ModelPVC[("Local Model Cache PVC
(Quantized ONNX INT4 Weights)")] + SLMPod["ModelDeployment: phi-4-mini
(compute: cpu, runtime: onnx-genai)"] + FLOp --> SLMPod + ModelPVC --> SLMPod + end + + subgraph NS_Agent["Namespace: edge-agents"] + SimPod["Manufacturing Sensor Simulator
(CNC / Extruder MQTT Publisher)"] + SOP["Local SOP Runbook
(ConfigMap)"] + SAT["Projected Kubernetes SAT
(audience: foundry-local)"] + AgentPod["Autonomous Edge Agent Pod
(Prebuilt Container)"] + NetPol{{"Challenge 5 NetworkPolicy:
Block All External Egress"}} + end + end + end + + ArcCP -. "1. Bootstrap Extensions
(Severed in Challenge 5)" .-> K8sCluster + ADR -. "Schema Sync" .-> NS_AIO + SimPod -- "2. MQTT Telemetry" --> Broker + Broker --> Dataflow + Dataflow -- "3. Filtered Anomalies" --> Broker + Broker -- "4. Subscribe Anomalies" --> AgentPod + SOP --> AgentPod + SAT --> AgentPod + AgentPod -- "5. OpenAI REST + SAT Token" --> Gateway + Gateway --> SLMPod + AgentPod -- "6. Autonomous PLC Command" --> Broker +``` + +### Edge Simulation Environment (`fast-k3s`) + +To make this MicroHack fast, reliable, and cost-effective for workshops, [`resources/infra/main.bicep`](resources/infra/main.bicep) provisions a single **Ubuntu 24.04 Azure VM (`Standard_D16s_v5` — 16 vCPUs, 64 GiB RAM)** with Premium SSDs and a single-node **K3s** cluster (`v1.33+`, the Microsoft GA validated baseline for Azure IoT Operations) in **~10 minutes**. + +> **Note on Production Topologies:** Single-node and multi-node K3s on Ubuntu are officially validated by Microsoft for Azure IoT Operations. Because all workloads in this MicroHack deploy via **Azure Arc-enabled Kubernetes (`connectedClusters`)** extensions and standard Kubernetes CRDs, the exact same workload manifests and extensions also run unchanged on physical **Azure Local (`aksarc`)** clusters in factory datacenters. + +--- + +## Objectives + +After completing this MicroHack, you will be able to: + +- Prepare and onboard an edge Kubernetes cluster to **Azure Arc (`connectedk8s`)** with OIDC issuer, Workload Identity, and kernel parameters tuned for industrial edge workloads. +- Deploy **Azure IoT Operations** (`Microsoft.IoTOperations`) and configure its native MQTT Broker and Dataflows to filter high-frequency manufacturing telemetry at the edge. +- Deploy **Foundry Local** (`Microsoft.Foundry` Arc extension) with Kubernetes Gateway API, Istio, and `azure-cert-manager`, and provision a CPU-optimized quantized Small Language Model (`compute: cpu`, `runtime: onnx-genai`) via a declarative `ModelDeployment` custom resource. +- Configure **Kubernetes Service Account Token (SAT)** authentication (`audience: foundry-local`, role `GeneralInferenceUser`) so in-cluster AI agents can invoke Foundry Local endpoints without cloud identity dependencies. +- Prove **air-gapped operational sovereignty** by blocking external cluster egress with Kubernetes `NetworkPolicy` and verifying sub-3-second autonomous fault mitigation. + +--- + +## General Prerequisites & Facilitator Guidance + +Before starting Challenge 1, ensure you have: + +- A modern browser (Edge, Chrome, or Firefox) to connect to your **Edge Host VM** via **Azure Bastion** in the Azure Portal. +- **Azure Subscription** with **Owner** (or **Contributor** + **User Access Administrator** / **RBAC Administrator**) permissions on your lab Resource Group. +- **Registered Resource Providers** (handled automatically by [`labautomation/shared-deploy-lab.ps1`](labautomation/shared-deploy-lab.ps1) or [`resources/infra/register-providers.sh`](resources/infra/register-providers.sh)): + - `Microsoft.Kubernetes`, `Microsoft.KubernetesConfiguration`, `Microsoft.ExtendedLocation` + - `Microsoft.IoTOperations`, `Microsoft.DeviceRegistry`, `Microsoft.SecretSyncController` +- **CLI Tools** (pre-installed automatically on the Edge Host VM via `cloud-init`): + - Azure CLI (`az`) >= 2.67 with extensions: `connectedk8s`, `k8s-extension`, `customlocation`, `azure-iot-ops` + - `kubectl` >= 1.33, `helm` >= 3.14, `jq`, and `mosquitto-clients` + +> [!TIP] +> **You're ready to start when:** +> 1. You can open the **`AzurePortalEdgeVmUrl`** from your lab dashboard (Credentials tab) and connect to your `edgeai-host-...` VM via **Azure Bastion** using `EdgeHostAdminUsername` (`edgeadmin`) and `EdgeHostAdminPassword`. +> 2. Running `cat /var/log/edge-bootstrap-complete.log` inside the VM prints `Edge host bootstrap completed successfully.` and `kubectl get nodes` shows your K3s node as `Ready`. + +> [!IMPORTANT] +> **Facilitator Guidance (from Live Azure Run Validation):** +> - **Isolated Per-Attendee Stack:** [`labautomation/deploy-lab.ps1`](labautomation/deploy-lab.ps1) provisions a dedicated `Standard_D16s_v5` (16 vCPUs, 64 GiB RAM) Ubuntu 24.04 K3s host, NAT Gateway, Azure Bastion, Key Vault, and HNS-enabled Storage Account per attendee. There is **no shared cluster contention** between attendees. +> - **Quota & Subscription Packing:** `lab-defaults.json` packs **4 attendees per subscription** (64 DSv5 vCPUs total, well within the standard 100-vCPU regional quota). To cut compute cost in half for smaller budgets, you can override `vmSize = 'Standard_D8s_v5'` (8 vCPUs, 32 GiB RAM), which still comfortably fits both Azure IoT Operations (`--broker-mem-profile Low`) and Foundry Local's `Phi-4-mini` CPU deployment. +> - **Regional Split (`swedencentral` VM + `westeurope` Arc/AIO Control Plane):** `lab-defaults.json` defaults to `swedencentral` for VM deployment (where `Standard_D16s_v5` capacity is abundant), while Challenges 01–02 register the **Azure Arc `connectedCluster`**, **Schema Registry**, **Device Registry Namespace**, and **Azure IoT Operations** resources in **`westeurope`** (an officially supported AIO control-plane region). All four control-plane resources must share the same `ARC_LOCATION` (`westeurope`) to avoid `HostResourceRegionMismatch`. +> - **Expected Timing:** Total machine/CLI wait time across all 5 challenges is **~40–45 minutes** (Challenge 01: ~5m, Challenge 02: ~14m, Challenge 03: ~11m including the 5.38 GB `Phi-4-mini` ONNX download, Challenge 04: ~2m, Challenge 05: ~2m), making this ideal for a **3- to 5-hour MicroHack** with architectural discussion and reflection. + +--- + +## MicroHack Challenges + +* [**Challenge 01 — Provision & Verify the Sovereign Edge Kubernetes Foundation**](challenges/challenge-01.md) **<- Start here** +* [**Challenge 02 — Deploy Azure IoT Operations & Simulated Manufacturing Telemetry**](challenges/challenge-02.md) +* [**Challenge 03 — Deploy Foundry Local Operator & Serve a Quantized CPU SLM**](challenges/challenge-03.md) +* [**Challenge 04 — Wire the Autonomous Edge Agent with Zero-Cloud SAT Authentication**](challenges/challenge-04.md) +* [**Challenge 05 — "Cut the Cord": Air-Gapped Fault Simulation & Verification**](challenges/challenge-05.md) +* [**Finish & Takeaways**](challenges/finish.md) + +## Solutions — Spoiler Warning + +* [**Solution 01 — Edge Kubernetes & Azure Arc Onboarding**](walkthrough/challenge-01/solution-01.md) +* [**Solution 02 — Azure IoT Operations & Telemetry Pipeline**](walkthrough/challenge-02/solution-02.md) +* [**Solution 03 — Foundry Local Extension & CPU ModelDeployment**](walkthrough/challenge-03/solution-03.md) +* [**Solution 04 — Autonomous Edge Agent & SAT Authentication**](walkthrough/challenge-04/solution-04.md) +* [**Solution 05 — Air-Gapped NetworkPolicy & Autonomous Actuation**](walkthrough/challenge-05/solution-05.md) + +--- + +## Contributors + +* Digvijay Chauhan diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-01.md b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-01.md new file mode 100644 index 000000000..fc0d626de --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-01.md @@ -0,0 +1,81 @@ +# Challenge 01 — Provision & Verify the Sovereign Edge Kubernetes Foundation + +**[Home](../Readme.md)** - [Next Challenge →](challenge-02.md) + +## 🎯 Objective + +Validate your cloud-simulated Edge Host environment, inspect the Linux kernel and Kubernetes prerequisites required for industrial edge workloads, and onboard the Kubernetes cluster to **Azure Arc (`Microsoft.Kubernetes/connectedClusters`)** with OIDC Issuer, Workload Identity, and Custom Locations enabled. + +## 🧭 Context + +Before deploying industrial telemetry brokers or local language models at a manufacturing site, platform engineers must establish a resilient, Arc-connected Kubernetes foundation. In production factories, this foundation runs on **AKS enabled by Azure Arc (`aksarc`) on Azure Local** hardware racks or **Ubuntu + K3s** edge nodes. + +In this challenge, you will connect to your isolated **Edge Host VM** (`Standard_D16s_v5` running Ubuntu 24.04 + single-node K3s), verify that kernel `inotify` and file-descriptor limits are tuned for high-throughput MQTT and AI model serving, and connect your Kubernetes cluster to Azure Arc. + +> [!IMPORTANT] +> **Every subsequent challenge builds on this cluster registration.** Take time to verify that both `oidcIssuerProfile.enabled` and `custom-locations` are active before moving to Challenge 02. + +## ✅ Tasks + +1. **Connect to your Edge Host VM via Azure Bastion.** *(~2 min)* + Open the **`AzurePortalEdgeVmUrl`** link from your lab dashboard (or navigate to your `EdgeHostVmName` in the Azure Portal under `ResourceGroupName`), select **Connect → Bastion**, and sign in with `EdgeHostAdminUsername` (`edgeadmin`) and `EdgeHostAdminPassword`. + +2. **Verify Edge Host Bootstrap, Kernel Limits & K3s Readiness.** *(~3 min)* + - Confirm that `cloud-init` has finished (`/var/log/edge-bootstrap-complete.log`). + - Check that the Linux kernel parameters (`fs.inotify.max_user_instances`, `fs.inotify.max_user_watches`, and `fs.file-max`) meet Azure IoT Operations requirements. + - Verify that your single-node **K3s** cluster (`v1.30+`), `kubectl`, `helm`, and the Azure CLI extensions (`connectedk8s`, `k8s-extension`, `customlocation`, `azure-iot-ops`) are installed and `Ready`. + - Clone the MicroHack repository (`https://github.com/microsoft/MicroHack.git`) to `~/MicroHack` and change into `03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI`. + +3. **Sign In to Azure CLI & Configure Your Environment File (`~/edge-env.sh`).** *(~3 min)* + - Sign in with `az login --use-device-code` (or use the VM's pre-authorized User-Assigned Managed Identity). + - Create `~/edge-env.sh` using the VM's Azure Instance Metadata Service (`169.254.169.254`) to automatically set `SUBSCRIPTION_ID`, `RESOURCE_GROUP`, `VM_LOCATION`, `ARC_LOCATION="westeurope"`, `CLUSTER_NAME`, and `SCHEMA_STORAGE_ACCOUNT`. + - Verify that all required Azure Resource Providers (`Microsoft.Kubernetes`, `Microsoft.KubernetesConfiguration`, `Microsoft.ExtendedLocation`, `Microsoft.IoTOperations`, `Microsoft.DeviceRegistry`, `Microsoft.SecretSyncController`) are in the `Registered` state. + +4. **Onboard the K3s Cluster to Azure Arc & Enable Custom Locations.** *(~5 min)* + - Connect your cluster to Azure Arc in `$ARC_LOCATION` (`westeurope`) using `az connectedk8s connect` with `--enable-oidc-issuer`, `--enable-workload-identity`, and `--disable-auto-upgrade`. + - Enable `cluster-connect` and `custom-locations` on your connected cluster using `az connectedk8s enable-features` and the `Microsoft.ExtendedLocation` Service Principal Object ID (`CustomLocationsRpObjectId` from your dashboard, or resolved via `az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b`). + +> [!TIP] +> **Why do we pass `--disable-auto-upgrade` when onboarding an industrial edge cluster?** +> Azure IoT Operations depends on specific validated versions of system Arc extensions. Disabling automatic agent upgrades prevents unplanned control-plane restarts during active manufacturing production shifts. + +## 📊 Key Parameters & Technical Reference + +| Parameter / Concept | Value / Hint | Why It Matters | +| :--- | :--- | :--- | +| **Control-Plane Region (`ARC_LOCATION`)** | `westeurope` | While your Edge Host VM runs in `swedencentral` (for VM SKU capacity), **Azure Device Registry** and **Azure IoT Operations** require an AIO control-plane region (`westeurope`, `northeurope`, `eastus2`, `westus2`, `westus3`, `germanywestcentral`, `uksouth`). Your `connectedClusters` resource, `CustomLocation`, `SchemaRegistry`, and `IoTOperations` instance must all share this region. | +| **OIDC & Workload Identity** | `--enable-oidc-issuer --enable-workload-identity` | Required by Azure IoT Operations (`adr-schema-registry` and secret sync controller). | +| **Custom Locations App ID** | `bc313c14-388c-4e7d-a58e-70017303ee3b` | Application ID of the first-party `Microsoft.ExtendedLocation` resource provider (`az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv`, or use `CustomLocationsRpObjectId` from your lab dashboard). | + +## 🤔 Grounding & Architecture Questions + +Before moving to Challenge 02, consider these architectural questions (suggested answers are in the Solution walkthrough): + +1. **Why must `fs.inotify.max_user_watches` and `fs.inotify.max_user_instances` be raised on the Linux host before deploying Azure IoT Operations and Foundry Local?** +2. **What is the role of an Azure Arc Custom Location (`Microsoft.ExtendedLocation/customLocations`), and how does it allow Azure Resource Manager (ARM) to deploy resources onto an on-premises Kubernetes cluster?** +3. **Why can the Edge Host VM reside in `swedencentral` while the Azure Arc `connectedCluster` resource is registered in `westeurope`?** + +## ✅ Success Criteria + +* `kubectl get nodes` shows your edge node in the `Ready` state on Kubernetes v1.33+. +* `sysctl fs.inotify.max_user_instances fs.inotify.max_user_watches fs.file-max` confirms the tuned kernel values (`8192`, `524288`, `100000`). +* `az connectedk8s show` reports `connectivityStatus: Connected` and `oidcIssuerProfile.enabled: true` in `westeurope`. +* All 9 pods in the `azure-arc` namespace are in the `Running` state. + +![Challenge 01 — Azure Portal Resource Group Overview](../img/portal-resource-group-overview.png) + +![Challenge 01 — Azure Portal Arc-enabled Kubernetes Cluster Connected (K3s v1.33.13+k3s2)](../img/portal-arc-cluster-connected.png) + +![Challenge 01 — CLI & Arc-enabled Kubernetes Verification](../img/challenge-01-portal-and-arc.png) + +## 📚 Learning Resources + +* [Prepare an Azure Arc-enabled Kubernetes cluster for Azure IoT Operations](https://learn.microsoft.com/azure/iot-operations/deploy-iot-ops/howto-prepare-cluster) +* [Connect an existing Kubernetes cluster to Azure Arc](https://learn.microsoft.com/azure/azure-arc/kubernetes/quickstart-connect-cluster) +* [Create and manage Custom Locations on Azure Arc-enabled Kubernetes](https://learn.microsoft.com/azure/azure-arc/kubernetes/custom-locations) + +## 🔓 Solution — Spoiler Warning + +[Solution Steps](../walkthrough/challenge-01/solution-01.md) + +[Next Challenge →](challenge-02.md) | [Back to Overview](../Readme.md) diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-02.md b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-02.md new file mode 100644 index 000000000..12122cce4 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-02.md @@ -0,0 +1,77 @@ +# Challenge 02 — Deploy Azure IoT Operations & Simulated Manufacturing Telemetry + +**[Home](../Readme.md)** - [← Previous Challenge](challenge-01.md) - [Next Challenge →](challenge-03.md) + +## 🎯 Objective + +Deploy **Azure IoT Operations (`Microsoft.IoTOperations`)** onto your Azure Arc-enabled Kubernetes cluster, expose an internal cluster MQTT listener, deploy the **Manufacturing Telemetry Simulator**, and configure an **AIO Dataflow** to filter and route anomalous machine readings to a dedicated anomaly topic. + +## 🧭 Context + +Contoso Precision Manufacturing's CNC-Extruder line (`CNC-EXTRUDER-01`) emits telemetry every 5 seconds (spindle RPM, bearing temperature in °C, vibration RMS in mm/s, and coolant flow in LPM). Streaming every normal reading to a CPU-based Small Language Model would saturate edge compute. + +Instead, you will use **Azure IoT Operations (AIO)** as the unified edge data plane: +1. All raw sensor readings arrive on the AIO MQTT Broker topic `factory/line1/telemetry`. +2. An **AIO Dataflow (`cnc-anomaly-router`)** evaluates readings in real time at the broker level and forwards **only anomalous windows** (`bearingTempC > 85.0` or `vibrationMmPerSec > 7.5`) to `factory/line1/anomalies`. + +> [!WARNING] +> **Watch out for AIO Dataflow filter semantics!** In Azure IoT Operations Dataflows, `builtInTransformationSettings.filter` is a **drop filter**: when the boolean expression evaluates to `true`, the message is **dropped**, and when it evaluates to `false`, the message passes through. To keep only anomalies (`temp > 85.0 || vib > 7.5`), your filter expression must drop normal readings (`$1 <= 85.0 && $2 <= 7.5`). + +## ✅ Tasks + +1. **Create the Schema Registry, Device Registry Namespace & Initialize AIO.** *(~7 min)* + - Create an **Azure IoT Operations Schema Registry** (`az iot ops schema registry create`) in `$ARC_LOCATION` (`westeurope`) backed by your HNS-enabled Storage Account (`$SCHEMA_STORAGE_ACCOUNT`). + - Create an **Azure Device Registry Namespace** (`az iot ops ns create`) in `$ARC_LOCATION` (`westeurope`). + - Run `az iot ops init` on your cluster to install the foundational `cert-manager` (`microsoft.certmanagement`) and `azure-secret-store` Arc extensions. + +2. **Deploy the Azure IoT Operations Instance.** *(~8 min)* + - Run `az iot ops create` in `$ARC_LOCATION` (`westeurope`) referencing both `--sr-resource-id` and `--ns-resource-id`, using the `Low` memory profile (`1` frontend replica, `1` backend partition) so plenty of CPU/RAM remains available for Foundry Local in Challenge 03. + - Confirm that all pods in the `azure-iot-operations` namespace (`aio-broker-*`, `aio-dataflow-*`, `adr-schema-registry-*`) reach `Running`. + +3. **Configure the Internal MQTT `BrokerListener` & Anomaly `Dataflow`.** *(~2 min)* + - Inspect and apply [`resources/manifests/01-iot-sim/aio-anomaly-dataflow.yaml`](../resources/manifests/01-iot-sim/aio-anomaly-dataflow.yaml). + - Verify that the `aio-broker-internal` Kubernetes Service (`port: 1883`, `nodePort: 31883`) and the `cnc-anomaly-router` `Dataflow` are created in `azure-iot-operations`. + +4. **Deploy the Manufacturing Telemetry Simulator & Verify Anomaly Routing.** *(~3 min)* + - Apply [`resources/manifests/01-iot-sim/telemetry-simulator.yaml`](../resources/manifests/01-iot-sim/telemetry-simulator.yaml) to start publishing CNC telemetry every 5 seconds to `factory/line1/telemetry` (with a thermal/vibration anomaly injected every 6th tick). + - Use `mosquitto_sub -h 127.0.0.1 -p 31883 -t "factory/line1/anomalies" -C 1 -v` to confirm that `NOMINAL` ticks are dropped and only `WARNING_THERMAL_VIBRATION_DRIFT` events arrive on `factory/line1/anomalies`. + +## 📊 Key Parameters & Technical Reference + +| Parameter / Resource | Value / Hint | Why It Matters | +| :--- | :--- | :--- | +| **Schema Registry & Device Registry Region** | `--location westeurope` (`$ARC_LOCATION`) | Must match the `location` of your Azure Arc `connectedClusters` resource from Challenge 01 to avoid `HostResourceRegionMismatch`. | +| **Device Registry Namespace (`--ns-resource-id`)** | `az iot ops ns create` | Required alongside `--sr-resource-id` by `az iot ops create` (`v1.4+`). | +| **MQTT Broker Memory Profile** | `--broker-mem-profile Low` (1 frontend, 1 backend partition) | Leaves ample CPU and memory headroom on the 16-vCPU/64-GiB node for Foundry Local and `Phi-4-mini` in Challenge 03. | +| **AIO `BrokerListener` Service Type** | `serviceType: NodePort` (`port: 1883`, `nodePort: 31883`) | The `default` listener already claims `serviceType: ClusterIp`. Using `NodePort` creates `aio-broker-internal.azure-iot-operations.svc.cluster.local:1883` internally AND exposes `127.0.0.1:31883` on the host for quick `mosquitto_sub` verification. | +| **AIO Dataflow Filter Semantics** | `expression: "$1 <= 85.0 && $2 <= 7.5"` | In Azure IoT Operations Dataflows, `filter` is a **drop filter**: messages where `expression` evaluates to `true` are **dropped**, and `false` passes through. | + +## 🤔 Grounding & Architecture Questions + +1. **Why do we filter normal sensor ticks inside an Azure IoT Operations Dataflow rather than forwarding every 5-second MQTT message to the Edge AI Agent?** +2. **Why did `aio-broker-webhook` reject creating a second `BrokerListener` with `serviceType: ClusterIp`, and how does a `NodePort` service still provide an internal cluster DNS name (`aio-broker-internal.azure-iot-operations.svc.cluster.local:1883`)?** + +## ✅ Success Criteria + +* `kubectl get pods -n azure-iot-operations` shows all broker, dataflow, and schema-registry pods `Running`. +* `kubectl get brokerlistener,dataflow -n azure-iot-operations` lists `aio-broker-internal` and `cnc-anomaly-router`. +* `kubectl logs -n edge-agents deploy/manufacturing-telemetry-simulator` shows telemetry published every 5 seconds. +* Subscribing to `factory/line1/anomalies` prints only readings with `bearingTempC > 85.0` or `vibrationMmPerSec > 7.5`. + +![Challenge 02 — Azure Portal Azure IoT Operations Instance Overview](../img/portal-aio-instance-overview.png) + +![Challenge 02 — Live Azure Bastion Terminal: Broker, Dataflow & Anomaly Stream](../img/bastion-challenge-02-aio-anomaly.png) + +![Challenge 02 — Azure IoT Operations & Anomaly Dataflow Filtering](../img/challenge-02-aio-and-dataflow.png) + +## 📚 Learning Resources + +* [Deploy Azure IoT Operations to an Arc-enabled Kubernetes cluster](https://learn.microsoft.com/azure/iot-operations/deploy-iot-ops/howto-deploy-iot-operations) +* [Configure MQTT Broker listeners in Azure IoT Operations](https://learn.microsoft.com/azure/iot-operations/manage-mqtt-broker/howto-configure-brokerlistener) +* [Filter data in an Azure IoT Operations Dataflow](https://learn.microsoft.com/azure/iot-operations/connect-to-cloud/howto-dataflow-filter) + +## 🔓 Solution — Spoiler Warning + +[Solution Steps](../walkthrough/challenge-02/solution-02.md) + +[← Previous Challenge](challenge-01.md) | [Next Challenge →](challenge-03.md) | [Back to Overview](../Readme.md) diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-03.md b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-03.md new file mode 100644 index 000000000..0aa83e68d --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-03.md @@ -0,0 +1,76 @@ +# Challenge 03 — Deploy Foundry Local Operator & Serve a Quantized CPU SLM + +**[Home](../Readme.md)** - [← Previous Challenge](challenge-02.md) - [Next Challenge →](challenge-04.md) + +## 🎯 Objective + +Deploy **Foundry Local on Azure Arc-enabled Kubernetes** using its production operator architecture (**Kubernetes Gateway API**, **Istio**, **`cert-manager`**, and the **`Microsoft.Foundry` Arc extension**), then deploy a quantized CPU Small Language Model (`Phi-4-mini-instruct-generic-cpu:5` via `onnx-genai`) using a declarative `ModelDeployment` Custom Resource. + +## 🧭 Context + +In sovereign edge environments, local AI inference must be managed with the same Kubernetes-native rigor as the rest of the platform. Rather than running unmanaged ad-hoc containers, **Foundry Local** (`Microsoft.Foundry` Arc extension) installs the `foundry-local-operator` control plane on your cluster: +- It synchronizes a local model catalog (`foundry-local-catalog` ConfigMap with 177 models across CPU and GPU). +- It caches model weights onto an on-cluster OCI registry PVC (`StoreModel` job + `inference-operator-model-store`) so inference survives network outages and pod restarts. +- It routes OpenAI-compatible inference traffic (`/v1/chat/completions`) through the **Kubernetes Gateway API** (`inference-internal-gateway-istio`) backed by **Istio**. +- It supports CPU-backed quantized inference (`compute: cpu`, `runtime: onnx-genai`) when edge hosts do not have dedicated GPUs. + +> [!IMPORTANT] +> **Critical Coexistence Step (`cert-manager` + `Microsoft.Foundry`):** When you ran `az iot ops init` in Challenge 02, Azure IoT Operations already installed the `cert-manager` (`microsoft.certmanagement`) extension, but set `trust-manager.secretTargets.enabled=false`. Before installing `Microsoft.Foundry`, you **must** run `az k8s-extension update --name cert-manager` to enable `trust-manager.secretTargets.enabled=true` and `config.enableGatewayAPI=true`—otherwise Foundry Local pods will fail to mount `secret "foundry-local-operator-ca-bundle"`. + +## ✅ Tasks + +1. **Install Kubernetes Gateway API CRDs & Istio (`istio-base` + `istiod`).** *(~2 min)* + - Apply the Kubernetes Gateway API CRDs (`v1.4.0` standard channel) and Gateway API Inference Extension CRDs (`v1.5.0`) using `kubectl apply --server-side`. + - Install `istio-base` and `istiod` in `istio-system` via Helm with `pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true`. + - Confirm that `kubectl get gatewayclass istio` shows `ACCEPTED = True`. + +2. **Update the Existing `cert-manager` Extension for Foundry Local.** *(~1 min)* + - Run `az k8s-extension update` on `--name "cert-manager"` to enable `config.enableGatewayAPI=true`, `trust-manager.secretTargets.enabled=true`, and `trust-manager.secretTargets.authorizedSecretsAll=true`. + +3. **Install the Foundry Local Extension (`Microsoft.Foundry`) & Inspect the Catalog.** *(~1 min)* + - Run `az k8s-extension create` with `--name "inference-operator"`, `--extension-type Microsoft.Foundry`, `--release-namespace "foundry-local-operator"`, `--config entraAuth.enabled=false`, `--config satAuth.enabled=true`, and `--config satAuth.audience="foundry-local"`. + - Query the `foundry-local-catalog` ConfigMap in `foundry-local-operator` using `jq` to list all models that support `"cpu"` compute. + +4. **Deploy the `phi-4-mini` CPU `ModelDeployment` & Cache Weights Locally.** *(~8–9 min)* + - Apply [`resources/manifests/02-foundry-local/caller-service-account.yaml`](../resources/manifests/02-foundry-local/caller-service-account.yaml) and [`resources/manifests/02-foundry-local/model-deployment-cpu.yaml`](../resources/manifests/02-foundry-local/model-deployment-cpu.yaml). + - Watch the `StoreModel` resource (`foundry-local-phi-4-mini-cpu-onnx-5`) download the 5.38 GB ONNX INT4 weights and push them into the local cluster model store (`PHASE: Available`). + - Confirm that `kubectl get modeldeployment phi-4-mini -n foundry-local-operator` reaches `STATE: Running` and `READY: true`. + +## 📊 Key Parameters & Technical Reference + +| Component | Value / Hint | Why It Matters | +| :--- | :--- | :--- | +| **Gateway API Installation Order** | Install Gateway API CRDs (`v1.4.0`) + Inference Extension CRDs (`v1.5.0`) **before** `istiod` | `istiod` discovers the Gateway API & `InferencePool` CRDs at startup; installing CRDs after `istiod` requires an `istiod` restart. | +| **Updating `cert-manager` First** | `az k8s-extension update --name cert-manager ... --config trust-manager.secretTargets.enabled=true --config trust-manager.secretTargets.authorizedSecretsAll=true` | `az iot ops init` in Challenge 02 already installed `cert-manager` (`microsoft.certmanagement`), but with `trust-manager.secretTargets.enabled=false`. Updating it in place allows Foundry Local pods to mount `foundry-local-operator-ca-bundle`. | +| **Catalog Model Alias** | `spec.model.catalog.name: phi-4-mini` | In `foundry-local-catalog`, the alias `phi-4-mini` with `compute: cpu` and `runtime: onnx-genai` resolves to `Phi-4-mini-instruct-generic-cpu:5` (5.15 GB ONNX INT4 weights). | +| **Internal Endpoint URL** | `https://inference-internal-gateway-istio.foundry-local-operator.svc.cluster.local:443/phi-4-mini` | Populated in `.status.endpoints.internal` once the `StoreModel` cache job (~7 min) and pod startup (~2 min) complete. | + +## 🤔 Grounding & Architecture Questions + +1. **How does Foundry Local's `StoreModel` + `inference-operator-model-store` architecture ensure that model pods can restart or scale when the cluster is air-gapped?** +2. **Why did we set `entraAuth.enabled=false` and `satAuth.enabled=true` on the `inference-operator` extension for this sovereign edge scenario?** + +## ✅ Success Criteria + +* `kubectl get gatewayclass istio` shows `ACCEPTED = True`. +* `az k8s-extension list` shows both `cert-manager` (`microsoft.certmanagement`) and `inference-operator` (`microsoft.foundry`) with `provisioningState: Succeeded`. +* `kubectl get storemodel -n foundry-local-operator` shows `foundry-local-phi-4-mini-cpu-onnx-5` in `PHASE = Available`. +* `kubectl get modeldeployment phi-4-mini -n foundry-local-operator` reaches `STATE = Running` (`READY = true`) and populates `.status.endpoints.internal`. + +![Challenge 03 — Azure Portal Arc-enabled Kubernetes Extensions](../img/portal-arc-extensions.png) + +![Challenge 03 — Live Azure Bastion Terminal: StoreModel Caching & Operator Pods](../img/bastion-challenge-03-storing.png) + +![Challenge 03 — Foundry Local Extension, StoreModel Cache & Phi-4-mini CPU ModelDeployment](../img/challenge-03-foundry-local-operator.png) + +## 📚 Learning Resources + +* [Deployment overview for Foundry Local on Azure Local](https://learn.microsoft.com/azure/azure-sovereign-clouds/private/foundry-local/deploy-overview) +* [Deploy Foundry Local as an Azure Arc extension](https://learn.microsoft.com/azure/azure-sovereign-clouds/private/foundry-local/deploy-foundry-local-arc-extension) +* [Deploy a catalog model on Foundry Local (`ModelDeployment` CRD)](https://learn.microsoft.com/azure/azure-sovereign-clouds/private/foundry-local/how-to-deploy-model) + +## 🔓 Solution — Spoiler Warning + +[Solution Steps](../walkthrough/challenge-03/solution-03.md) + +[← Previous Challenge](challenge-02.md) | [Next Challenge →](challenge-04.md) | [Back to Overview](../Readme.md) diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-04.md b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-04.md new file mode 100644 index 000000000..5ef1acf29 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-04.md @@ -0,0 +1,71 @@ +# Challenge 04 — Wire the Autonomous Edge Agent with Zero-Cloud SAT Authentication + +**[Home](../Readme.md)** - [← Previous Challenge](challenge-03.md) - [Next Challenge →](challenge-05.md) + +## 🎯 Objective + +Configure **Kubernetes Service Account Token (SAT)** authentication between the `edge-agents` namespace and Foundry Local, deploy the local **Manufacturing SOP Runbook (`SOP-CNC-042`)**, and deploy the prebuilt **Autonomous Edge Triage & Actuation Agent** to close the loop between Azure IoT Operations and Foundry Local. + +## 🧭 Context + +When an industrial facility loses internet connectivity, workloads inside the Kubernetes cluster cannot reach Microsoft Entra ID in the public cloud to mint or refresh OAuth tokens. + +To ensure uninterrupted authentication in both connected and disconnected modes, Foundry Local supports **Kubernetes Service Account Token (SAT)** authentication (`method: serviceAccountToken`): +- The Kubernetes API server projects a short-lived, audience-bound JWT (`audience: foundry-local`) into the `autonomous-edge-agent` pod at `/var/run/secrets/foundry-local/token`. +- The `phi-4-mini` `ModelDeployment` authorizes `serviceAccountName: inference-client` in `serviceAccountNamespace: edge-agents` with role `GeneralInferenceUser`. +- When an anomaly arrives on `factory/line1/anomalies`, the Edge Agent reads the local SOP ConfigMap (`/etc/edge-sop/sop.md`), calls Foundry Local's `/v1/chat/completions` endpoint with its SAT token, and publishes the resulting PLC stabilization command back to `factory/line1/commands`. + +> [!TIP] +> **Always re-read projected Service Account Tokens from disk.** Kubernetes automatically rotates projected tokens before `expirationSeconds` (`3600s`). In `edge-agent-deployment.yaml`, the agent re-reads `/var/run/secrets/foundry-local/token` on every inference call so token rotation never causes a `401 Unauthorized` failure during long-running shifts. + +## ✅ Tasks + +1. **Test SAT Authentication Against the Internal Foundry Local Endpoint.** *(~3 min)* + - Mint a temporary 10-minute token using `kubectl create token inference-client -n edge-agents --audience foundry-local --duration 10m`. + - Call `https://inference-internal-gateway-istio.foundry-local-operator.svc.cluster.local:443/phi-4-mini/v1/chat/completions` from inside the cluster with `"model": "Phi-4-mini-instruct-generic-cpu:5"` and verify you receive a `200 OK` completion (`"content": "READY"`). + +2. **Deploy the Local Manufacturing SOP Runbook (`SOP-CNC-042`).** *(~1 min)* + - Inspect and apply [`resources/manifests/03-edge-agent/local-sop-configmap.yaml`](../resources/manifests/03-edge-agent/local-sop-configmap.yaml) into the `edge-agents` namespace. + +3. **Deploy the Prebuilt Autonomous Edge Agent.** *(~2 min)* + - Inspect [`resources/manifests/03-edge-agent/edge-agent-deployment.yaml`](../resources/manifests/03-edge-agent/edge-agent-deployment.yaml) to see how `serviceAccountName: inference-client`, the projected `foundry-local-sat` volume, and the `local-sop-runbook` ConfigMap are mounted. + - Apply the deployment and verify that `/var/run/secrets/foundry-local/token` is mounted inside the running pod. + +4. **Verify Closed-Loop Autonomous PLC Stabilization.** *(~3 min)* + - Tail the logs of `deploy/autonomous-edge-agent` and `deploy/manufacturing-telemetry-simulator` in `edge-agents` to watch live warning anomalies (`85°C–104.9°C`) trigger `adjust_plc_setpoint` (`spindle_rpm_delta_pct: -25`, `enable_coolant_boost: true`), reducing the simulated PLC spindle speed from `4200 RPM` to `3150 RPM`. + +## 📊 Key Parameters & Technical Reference + +| Component / Setting | Value / Hint | Why It Matters | +| :--- | :--- | :--- | +| **Caller ServiceAccount** | `name: inference-client`, `namespace: edge-agents`, `automountServiceAccountToken: false` | Prevents mounting the default Kubernetes API token; the pod explicitly mounts an audience-bound projected token instead. | +| **Projected Token Volume** | `audience: foundry-local`, `mountPath: /var/run/secrets/foundry-local` | Foundry Local's SAT sidecar validates that the incoming `Bearer` JWT has `aud: foundry-local` and matches `authorizedSubjects` (`GeneralInferenceUser`). | +| **Served Model Identifier** | `Phi-4-mini-instruct-generic-cpu:5` | The exact model ID exposed at `/v1/models` and `/v1/chat/completions` by the `phi-4-mini` `ModelDeployment`. | +| **Internal Inference URL** | `https://inference-internal-gateway-istio.foundry-local-operator.svc.cluster.local:443/phi-4-mini/v1/chat/completions` | Routes in-cluster HTTPS requests through the Istio Gateway to the `phi-4-mini` pod with zero external network traffic. | + +## 🤔 Grounding & Architecture Questions + +1. **Why does `caller-service-account.yaml` set `automountServiceAccountToken: false` on the `inference-client` ServiceAccount when the pod still needs a Service Account Token?** +2. **How does the Foundry Local auth sidecar verify a Kubernetes Service Account Token without calling out to Microsoft Entra ID?** + +## ✅ Success Criteria + +* A manual request using a token minted for `inference-client` (`--audience foundry-local`) returns a `200 OK` chat completion from `Phi-4-mini-instruct-generic-cpu:5`. +* `kubectl get pods -n edge-agents` shows both `manufacturing-telemetry-simulator` and `autonomous-edge-agent` in `Running` status. +* `kubectl logs -n edge-agents deploy/autonomous-edge-agent` shows incoming anomalies from `factory/line1/anomalies` and outgoing autonomous PLC commands (`adjust_plc_setpoint`) to `factory/line1/commands`. +* `kubectl logs -n edge-agents deploy/manufacturing-telemetry-simulator` prints `=== PLC ACTUATED: spindleRpm=3150, coolantBoost=True ===`. + +![Challenge 04 — Live Azure Bastion Terminal: Projected Kubernetes SAT JWT & Live Phi-4-mini Completion](../img/bastion-challenge-04-sat-and-inference.png) + +![Challenges 04 & 05 — Kubernetes SAT Auth & Closed-Loop PLC Actuation](../img/challenge-05-airgap-interlock.png) + +## 📚 Learning Resources + +* [Configure Kubernetes Service Account Token (SAT) authentication for Foundry Local](https://learn.microsoft.com/azure/azure-sovereign-clouds/private/foundry-local/how-to-configure-service-account-token-authentication) +* [Kubernetes Projected Service Account Token Volumes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection) + +## 🔓 Solution — Spoiler Warning + +[Solution Steps](../walkthrough/challenge-04/solution-04.md) + +[← Previous Challenge](challenge-03.md) | [Next Challenge →](challenge-05.md) | [Back to Overview](../Readme.md) diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-05.md b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-05.md new file mode 100644 index 000000000..c07f54b0f --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/challenge-05.md @@ -0,0 +1,69 @@ +# Challenge 05 — "Cut the Cord": Air-Gapped Fault Simulation & Verification + +**[Home](../Readme.md)** - [← Previous Challenge](challenge-04.md) - [Finish →](finish.md) + +## 🎯 Objective + +Enforce strict air-gapped network isolation on the edge workload namespace using a Kubernetes `NetworkPolicy`, prove that public internet and Azure cloud endpoints are unreachable, inject a critical bearing thermal-runaway fault, and verify that the local AI agent autonomously triggers an emergency PLC safety interlock with **zero cloud dependency**. + +## 🧭 Context + +A fiber cut outside the manufacturing plant severs all WAN connectivity to the internet and public Azure regions. At the exact same moment, Coolant Pump B fails on `CNC-EXTRUDER-01`, causing bearing temperature to spike to **114.6°C** and vibration to reach **15.2 mm/s** (Critical Seizure risk under **SOP-CNC-042**). + +Because your architecture uses: +1. **Azure IoT Operations** running locally on the cluster (`aio-broker-internal`), +2. **Foundry Local** with pre-cached ONNX INT4 weights and an internal Istio Gateway (`inference-internal-gateway-istio`), +3. **Kubernetes SAT authentication** minted by the local K3s API server, and +4. **Local SOP grounding** via an in-cluster `ConfigMap`, + +the entire **sense → filter → reason → actuate** loop continues to operate locally without a single packet leaving the cluster. + +## ✅ Tasks + +1. **Apply the Sovereign Air-Gap `NetworkPolicy` (`enforce-airgap-sovereign-mode`).** *(~1 min)* + - Inspect and apply [`resources/manifests/04-airgap-policy/deny-external-egress.yaml`](../resources/manifests/04-airgap-policy/deny-external-egress.yaml) to lock down egress from the `edge-agents` namespace. + +2. **Prove That Public Cloud / Internet Egress Is Blocked.** *(~2 min)* + - Exec into `deploy/autonomous-edge-agent` and test an outbound HTTPS connection to `https://management.azure.com` (confirming it times out / is blocked by the `NetworkPolicy`). + - Confirm that internal cluster DNS resolution to `aio-broker-internal.azure-iot-operations.svc.cluster.local` still succeeds. + +3. **Inject a Critical Bearing Thermal-Runaway Fault While Air-Gapped.** *(~2 min)* + - Apply [`resources/manifests/04-airgap-policy/inject-critical-fault.yaml`](../resources/manifests/04-airgap-policy/inject-critical-fault.yaml) to launch the `inject-critical-bearing-fault` Kubernetes Job, publishing a `CRITICAL_THERMAL_RUNAWAY_BEARING_SEIZURE` event (`bearingTempC: 114.6`, `vibrationMmPerSec: 15.2`, `coolantFlowLpm: 1.4`) to `factory/line1/telemetry`. + +4. **Verify Autonomous Emergency PLC Interlock Execution.** *(~2 min)* + - Inspect the logs of `autonomous-edge-agent` and `manufacturing-telemetry-simulator` to confirm that `Phi-4-mini` evaluates the fault against **SOP-CNC-042** and issues `trigger_safety_interlock` (`spindle_rpm_delta_pct: -60`, `interlock_mode: "SAFE_HOLD"`), reducing the PLC spindle speed by 60% (`4200 RPM` → `1680 RPM`). + +## 📊 Key Parameters & Technical Reference + +| Policy Rule / Component | Value / Hint | Why It Matters | +| :--- | :--- | :--- | +| **`NetworkPolicy` Scope** | `namespace: edge-agents`, `policyTypes: [Egress]` | Default-denies all external internet and public Azure cloud egress (`0.0.0.0/0`), allowing only explicit in-cluster namespaces. | +| **Allowed In-Cluster Egress** | `kube-system` (port `53`), `azure-iot-operations` (ports `1883`, `18883`), `foundry-local-operator` & `istio-system` (ports `80`, `443`, `8080`) | Ensures CoreDNS, local AIO MQTT pub/sub, and local Foundry Local inference continue working while external WAN access is severed. | +| **Critical Fault Thresholds (`SOP-CNC-042`)** | `bearingTempC >= 105.0°C` OR `vibrationMmPerSec >= 13.0 mm/s` OR `coolantFlowLpm < 2.0 LPM` | The fault injector publishes `114.6°C`, `15.2 mm/s`, and `1.4 LPM`, prompting `Phi-4-mini` to select `trigger_safety_interlock` (`-60%` RPM → `1680 RPM`). | + +## 🤔 Grounding & Architecture Questions + +1. **Why must the `NetworkPolicy` explicitly permit UDP/TCP port `53` to `kube-system` even when all services are inside the same Kubernetes cluster?** +2. **In `agent.py`, why do we include a deterministic SOP safety guard (`local-sop-deterministic-guard`) alongside the SLM call for critical industrial equipment?** + +## ✅ Success Criteria + +* `kubectl get networkpolicy -n edge-agents` shows `enforce-airgap-sovereign-mode` active. +* Outbound connectivity checks from inside the `autonomous-edge-agent` pod to `https://management.azure.com` fail/time out, proving air-gap isolation. +* After running `inject-critical-bearing-fault`, `kubectl logs -n edge-agents deploy/autonomous-edge-agent` shows `tool=trigger_safety_interlock` and `spindle_rpm_delta_pct=-60`. +* `kubectl logs -n edge-agents deploy/manufacturing-telemetry-simulator` confirms `spindleRpm=1680` (`-60%` from `4200 RPM`) and `coolantBoost=True`. + +![Challenge 05 — Live Azure Bastion Terminal: "Cut the Cord" NetworkPolicy & Autonomous PLC Interlock](../img/bastion-challenge-05-cut-the-cord.png) + +![Challenge 05 — Air-Gapped NetworkPolicy & Emergency PLC Interlock Execution](../img/challenge-05-airgap-interlock.png) + +## 📚 Learning Resources + +* [Foundry Local on Azure Local in disconnected environments overview](https://learn.microsoft.com/azure/azure-sovereign-clouds/private/foundry-local/disconnected-operations/concept-overview) +* [Kubernetes Network Policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) + +## 🔓 Solution — Spoiler Warning + +[Solution Steps](../walkthrough/challenge-05/solution-05.md) + +[← Previous Challenge](challenge-04.md) | [Finish →](finish.md) | [Back to Overview](../Readme.md) diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/finish.md b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/finish.md new file mode 100644 index 000000000..12b34ac52 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/challenges/finish.md @@ -0,0 +1,22 @@ +# Congratulations — You Completed the Air-Gapped Agentic Edge AI MicroHack! + +You have built and validated an end-to-end **Sovereign Edge AI and Industrial IoT architecture** running entirely on **Azure Arc-enabled Kubernetes**. + +## What You Accomplished + +1. **Sovereign Edge Kubernetes Foundation (Challenge 01):** Prepared an Ubuntu + K3s (or nested Azure Local `aksarc`) edge host with kernel `inotify` tuning and onboarded it to **Azure Arc (`connectedk8s`)** with OIDC Issuer, Workload Identity, and Custom Locations. +2. **Unified Edge IoT Data Plane (Challenge 02):** Deployed **Azure IoT Operations (`Microsoft.IoTOperations`)**, configured an internal cluster MQTT `BrokerListener`, streamed simulated CNC/Extruder sensor data, and created a declarative **AIO Dataflow** to filter anomalies at the edge broker before invoking AI compute. +3. **Kubernetes-Native Local LLM Serving (Challenge 03):** Deployed **Foundry Local (`Microsoft.Foundry` Arc extension)** with **Kubernetes Gateway API**, **Istio**, and **`azure-cert-manager`**, and provisioned a quantized CPU Small Language Model (`Phi-4-mini-instruct`, `compute: cpu`, `runtime: onnx-genai`) via a `ModelDeployment` Custom Resource with locally cached weights. +4. **Zero-Cloud SAT Authentication & Edge Agent (Challenge 04):** Configured **Kubernetes Service Account Token (SAT)** authentication (`audience: foundry-local`, role `GeneralInferenceUser`) and deployed an **Autonomous Edge Agent** grounded on a local SOP `ConfigMap` (`SOP-CNC-042`). +5. **"Cut the Cord" Air-Gapped Validation (Challenge 05):** Applied a strict egress-blocking `NetworkPolicy`, injected a critical thermal-runaway fault (`114.6°C`), and verified autonomous closed-loop PLC actuation (`trigger_safety_interlock`) with **0 ms cloud dependency**. + +--- + +## Moving from Cloud Simulation to Physical Edge Production + +When transitioning this architecture from your MicroHack lab to physical factory floors: + +- **Compute & Hypervisor:** Replace the cloud-simulated edge VM with physical **Azure Local** OEM hardware nodes (or ruggedized Ubuntu K3s edge appliances), optionally adding NVIDIA L4/A2 GPUs (`compute: gpu`, `runtime: vllm` in your `ModelDeployment` spec) for higher concurrency. +- **Full Disconnected Appliance Operations:** For facilities that are permanently air-gapped from day zero, use **Azure Local Disconnected Operations** together with the **Foundry Local Expansion Pack**, which bundles Istio, Gateway API CRDs, `cert-manager`, and model artifacts into the local `edgeartifacts` container registry. + +[Back to MicroHack Overview](../Readme.md) diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/architecture.png b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/architecture.png new file mode 100644 index 000000000..f415f104f Binary files /dev/null and b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/architecture.png differ diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-challenge-02-aio-anomaly.png b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-challenge-02-aio-anomaly.png new file mode 100644 index 000000000..aa8c54027 Binary files /dev/null and b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-challenge-02-aio-anomaly.png differ diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-challenge-03-storing.png b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-challenge-03-storing.png new file mode 100644 index 000000000..db896bdbe Binary files /dev/null and b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-challenge-03-storing.png differ diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-challenge-04-sat-and-inference.png b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-challenge-04-sat-and-inference.png new file mode 100644 index 000000000..7a290f2a8 Binary files /dev/null and b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-challenge-04-sat-and-inference.png differ diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-challenge-05-cut-the-cord.png b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-challenge-05-cut-the-cord.png new file mode 100644 index 000000000..d27ed7dc8 Binary files /dev/null and b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-challenge-05-cut-the-cord.png differ diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-edge-host-login.png b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-edge-host-login.png new file mode 100644 index 000000000..8b09ad950 Binary files /dev/null and b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/bastion-edge-host-login.png differ diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/challenge-01-portal-and-arc.png b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/challenge-01-portal-and-arc.png new file mode 100644 index 000000000..4ebcc2f91 Binary files /dev/null and b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/challenge-01-portal-and-arc.png differ diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/challenge-02-aio-and-dataflow.png b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/challenge-02-aio-and-dataflow.png new file mode 100644 index 000000000..1c1545a42 Binary files /dev/null and b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/challenge-02-aio-and-dataflow.png differ diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/challenge-03-foundry-local-operator.png b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/challenge-03-foundry-local-operator.png new file mode 100644 index 000000000..b92f10497 Binary files /dev/null and b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/challenge-03-foundry-local-operator.png differ diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/challenge-05-airgap-interlock.png b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/challenge-05-airgap-interlock.png new file mode 100644 index 000000000..e73ec9e12 Binary files /dev/null and b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/challenge-05-airgap-interlock.png differ diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/portal-aio-instance-overview.png b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/portal-aio-instance-overview.png new file mode 100644 index 000000000..f7dd99e80 Binary files /dev/null and b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/portal-aio-instance-overview.png differ diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/portal-arc-cluster-connected.png b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/portal-arc-cluster-connected.png new file mode 100644 index 000000000..ce8ebbfb1 Binary files /dev/null and b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/portal-arc-cluster-connected.png differ diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/portal-arc-extensions.png b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/portal-arc-extensions.png new file mode 100644 index 000000000..8504a3ea4 Binary files /dev/null and b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/portal-arc-extensions.png differ diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/portal-resource-group-overview.png b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/portal-resource-group-overview.png new file mode 100644 index 000000000..5f691abb4 Binary files /dev/null and b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/img/portal-resource-group-overview.png differ diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/labautomation/README.md b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/labautomation/README.md new file mode 100644 index 000000000..58dc76c22 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/labautomation/README.md @@ -0,0 +1,9 @@ +# Lab Automation — Air-Gapped Agentic AI: Orchestrating Local SLMs and IoT on Edge Kubernetes + +This folder integrates the MicroHack with the automated lab provisioning platform. + +## Files + +- [`lab-defaults.json`](lab-defaults.json): Declares `resourcegroup-with-subscriptionowner` scope (required so participants can onboard Azure Arc connected clusters and create custom locations/role assignments), packing 4 labs per subscription (64 vCPUs total using `Standard_D16s_v5`). +- [`shared-deploy-lab.ps1`](shared-deploy-lab.ps1): Runs once per subscription prior to participant fan-out to register `Microsoft.Kubernetes`, `Microsoft.IoTOperations`, `Microsoft.Foundry`, `Microsoft.CertManagement`, and related resource providers. +- [`deploy-lab.ps1`](deploy-lab.ps1): Runs once per participant to deploy [`../resources/infra/main.bicep`](../resources/infra/main.bicep) (`Standard_D16s_v5` Ubuntu 24.04 VM with K3s, Helm, `kubectl`, and Azure CLI extensions pre-installed via `cloud-init-k3s.yaml`). diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/labautomation/deploy-lab.ps1 b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/labautomation/deploy-lab.ps1 new file mode 100644 index 000000000..a0db1067b --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/labautomation/deploy-lab.ps1 @@ -0,0 +1,92 @@ +<# +.SYNOPSIS +Deploys the Air-Gapped Agentic Edge AI lab resources into a participant's resource group. +.DESCRIPTION +Provisions the Edge Host VM (Standard_D16s_v5 Ubuntu 24.04 with K3s), +Azure Bastion, Schema Registry Storage Account, and Key Vault, then emits +HackboxCredential entries to the participant dashboard. +.PARAMETER DeploymentType +Defines the deployment scope; allowed values are subscription, resourcegroup, or resourcegroup-with-subscriptionowner. +.PARAMETER SubscriptionId +Specifies the Azure subscription that contains the lab resources. +.PARAMETER ResourceGroupName +In case of resourcegroup deployment, specifies the target resource group name. +.PARAMETER PreferredLocation +Specifies the preferred Azure regions (ordered by preference) for resource deployment. +.PARAMETER AllowedEntraUserIds +Optional list of Entra user object IDs permitted to access the lab resources. +#> +param( + [Parameter(Mandatory = $true)] + [ValidateSet('subscription', 'resourcegroup', 'resourcegroup-with-subscriptionowner')] + [string]$DeploymentType, + + [Parameter(Mandatory = $true)] + [string]$SubscriptionId, + + [string]$ResourceGroupName = '', + + [string[]]$PreferredLocation = @(), + + [string[]]$AllowedEntraUserIds = @() +) + +$ErrorActionPreference = 'Stop' +$scriptPath = Split-Path -Parent $MyInvocation.MyCommand.Definition +$templatePath = Join-Path (Split-Path -Parent $scriptPath) 'resources\infra\main.bicep' + +if ($DeploymentType -ne 'subscription' -and [string]::IsNullOrEmpty($ResourceGroupName)) { + throw "ResourceGroupName must be provided when DeploymentType is '$DeploymentType'." +} + +$effectiveLocation = if ($PreferredLocation.Count -gt 0 -and -not [string]::IsNullOrWhiteSpace($PreferredLocation[0])) { + $PreferredLocation[0] +} +else { + 'swedencentral' +} + +if ($DeploymentType -eq 'subscription') { + $stableHash = Get-MhhStableHash $AllowedEntraUserIds -Length 12 + $effectiveResourceGroup = "rg-edgeai-$stableHash" + if (-not (Get-AzResourceGroup -Name $effectiveResourceGroup -ErrorAction SilentlyContinue)) { + New-AzResourceGroup -Name $effectiveResourceGroup -Location $effectiveLocation | Out-Null + } +} +else { + $effectiveResourceGroup = $ResourceGroupName +} + +$adminUsername = 'edgeadmin' +$adminPassword = New-MhhStablePassword -Seed "edgeai-$effectiveResourceGroup" +$participantObjectId = if ($AllowedEntraUserIds.Count -gt 0) { $AllowedEntraUserIds[0] } else { '' } + +$deploymentResult = Invoke-MhhDeploymentWithRegionFallback ` + -PreferredLocations $PreferredLocation ` + -ResourceGroupName $effectiveResourceGroup ` + -RgOwnerEntraObjectIds $AllowedEntraUserIds ` + -TemplateFile $templatePath ` + -TemplateParameterObject @{ + edgeProfile = 'fast-k3s' + adminUsername = $adminUsername + adminPasswordOrKey = $adminPassword + authType = 'password' + deployBastion = $true + participantObjectId = $participantObjectId + } + +$outputs = $deploymentResult.Outputs +$tenantId = (Get-AzContext).Tenant.Id +$rgPortalUrl = "https://portal.azure.com/#@$tenantId/resource/subscriptions/$SubscriptionId/resourceGroups/$effectiveResourceGroup/overview" +$vmPortalUrl = "https://portal.azure.com/#@$tenantId/resource/subscriptions/$SubscriptionId/resourceGroups/$effectiveResourceGroup/providers/Microsoft.Compute/virtualMachines/$($outputs.edgeVmName.Value)/connect" + +@{ 'HackboxCredential' = @{ name = 'ResourceGroupName'; value = $effectiveResourceGroup; note = 'Lab Resource Group containing the Edge Host VM and supporting resources' } } +@{ 'HackboxCredential' = @{ name = 'AzurePortalResourceGroupUrl'; value = $rgPortalUrl; note = 'Direct link to your lab Resource Group in the Azure Portal' } } +@{ 'HackboxCredential' = @{ name = 'EdgeHostVmName'; value = $outputs.edgeVmName.Value; note = 'Ubuntu 24.04 K3s Edge Host VM (connect via Azure Bastion)' } } +@{ 'HackboxCredential' = @{ name = 'AzurePortalEdgeVmUrl'; value = $vmPortalUrl; note = 'Direct link to connect to your Edge Host VM via Azure Bastion' } } +@{ 'HackboxCredential' = @{ name = 'EdgeHostAdminUsername'; value = $adminUsername; note = 'SSH / Bastion username for the Edge Host VM' } } +@{ 'HackboxCredential' = @{ name = 'EdgeHostAdminPassword'; value = $adminPassword; note = 'SSH / Bastion password for the Edge Host VM' } } +@{ 'HackboxCredential' = @{ name = 'ArcClusterName'; value = $outputs.arcClusterSuggestedName.Value; note = 'Suggested name when onboarding K3s to Azure Arc in Challenge 1' } } +@{ 'HackboxCredential' = @{ name = 'ArcControlPlaneLocation'; value = 'westeurope'; note = 'Supported region for Azure Arc, Device Registry, and Azure IoT Operations control-plane resources' } } +@{ 'HackboxCredential' = @{ name = 'SchemaStorageAccount'; value = $outputs.schemaStorageAccountName.Value; note = 'HNS-enabled Storage Account for Azure IoT Operations Schema Registry' } } +@{ 'HackboxCredential' = @{ name = 'EdgeManagedIdentityClientId'; value = $outputs.userAssignedIdentityClientId.Value; note = 'Client ID of the User-Assigned Managed Identity attached to the Edge Host VM' } } diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/labautomation/lab-defaults.json b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/labautomation/lab-defaults.json new file mode 100644 index 000000000..4f9759eb5 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/labautomation/lab-defaults.json @@ -0,0 +1,9 @@ +{ + "$schema": "https://raw.githubusercontent.com/microsoft/MicroHack/refs/heads/main/lab-defaults-schema.json", + "groups": [], + "deploymentType": "resourcegroup-with-subscriptionowner", + "labsPerSubscription": 4, + "preferredLocation": "swedencentral, westus3, westeurope", + "estimatedDailyCostsUsd": 22.0, + "estimatedSharedDeploymentDailyCostsUsd": 0.0 +} diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/labautomation/shared-deploy-lab.ps1 b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/labautomation/shared-deploy-lab.ps1 new file mode 100644 index 000000000..c4a8aed38 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/labautomation/shared-deploy-lab.ps1 @@ -0,0 +1,61 @@ +<# +.SYNOPSIS +Hook that runs once per subscription before any deploy-lab.ps1 run starts. +.DESCRIPTION +Registers all Azure Resource Providers required for Azure Arc-enabled Kubernetes, +Azure IoT Operations, and Foundry Local so per-participant deployments do not race. +.PARAMETER SubscriptionId +Specifies the Azure subscription that contains the lab resources. +.PARAMETER PreferredLocation +Specifies the preferred Azure regions (ordered by preference) for resource deployment. +.PARAMETER AllowedEntraUserIds +Entra user object IDs of every participant holding a lab in this subscription. +#> +param( + [Parameter(Mandatory = $true)] + [string]$SubscriptionId, + + [Parameter(Mandatory = $true)] + [string[]]$PreferredLocation = @(), + + [Parameter(Mandatory = $false)] + [string[]]$AllowedEntraUserIds = @() +) + +$ErrorActionPreference = 'Stop' + +$requiredProviders = @( + 'Microsoft.Compute', + 'Microsoft.Network', + 'Microsoft.Storage', + 'Microsoft.KeyVault', + 'Microsoft.ExtendedLocation', + 'Microsoft.Kubernetes', + 'Microsoft.KubernetesConfiguration', + 'Microsoft.IoTOperations', + 'Microsoft.DeviceRegistry', + 'Microsoft.SecretSyncController' +) + +foreach ($provider in $requiredProviders) { + $state = (Get-AzResourceProvider -ProviderNamespace $provider -ErrorAction SilentlyContinue | Select-Object -First 1).RegistrationState + if ($state -ne 'Registered') { + Write-Host "[$SubscriptionId] Registering resource provider: $provider" + Register-AzResourceProvider -ProviderNamespace $provider -ErrorAction Stop | Out-Null + } + else { + Write-Host "[$SubscriptionId] Resource provider $provider is already registered." + } +} + +# Resolve the tenant-specific Object ID of the Microsoft.ExtendedLocation resource provider +# (AppId: bc313c14-388c-4e7d-a58e-70017303ee3b) so participants who lack Microsoft Graph +# directory-read permissions can still pass --custom-locations-oid in Challenge 01. +$customLocationsSp = Get-AzADServicePrincipal -ApplicationId 'bc313c14-388c-4e7d-a58e-70017303ee3b' -ErrorAction SilentlyContinue +if ($customLocationsSp -and $customLocationsSp.Id) { + @{ 'HackboxCredential' = @{ + name = 'CustomLocationsRpObjectId' + value = $customLocationsSp.Id + note = 'Microsoft.ExtendedLocation Service Principal Object ID for az connectedk8s enable-features' + } } +} diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/infra/cloud-init-k3s.yaml b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/infra/cloud-init-k3s.yaml new file mode 100644 index 000000000..20a62aa8e --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/infra/cloud-init-k3s.yaml @@ -0,0 +1,76 @@ +#cloud-config +package_update: true +package_upgrade: true +packages: + - curl + - jq + - git + - ca-certificates + - apt-transport-https + - gnupg + - lsb-release + - python3-pip + - python3-venv + - mosquitto-clients + +write_files: + - path: /etc/sysctl.d/99-azure-iot-operations.conf + permissions: '0644' + content: | + fs.inotify.max_user_instances = 8192 + fs.inotify.max_user_watches = 524288 + fs.file-max = 100000 + + - path: /usr/local/bin/bootstrap-edge-host.sh + permissions: '0755' + content: | + #!/usr/bin/env bash + set -euo pipefail + + echo "[1/5] Applying kernel sysctl limits for Azure IoT Operations & Foundry Local..." + sysctl --system + + echo "[2/5] Installing single-node K3s (Kubernetes 1.33 - Microsoft GA validated baseline for Azure IoT Operations)..." + curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.33.13+k3s2" sh -s - --write-kubeconfig-mode 644 + + # Configure kubeconfig for edgeadmin + ADMIN_USER="edgeadmin" + if id "$ADMIN_USER" &>/dev/null; then + USER_HOME=$(eval echo "~$ADMIN_USER") + mkdir -p "$USER_HOME/.kube" + cp /etc/rancher/k3s/k3s.yaml "$USER_HOME/.kube/config" + chown -R "$ADMIN_USER:$ADMIN_USER" "$USER_HOME/.kube" + chmod 600 "$USER_HOME/.kube/config" + echo 'export KUBECONFIG=~/.kube/config' >> "$USER_HOME/.bashrc" + fi + + echo "[3/5] Installing Helm 3..." + curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash + + echo "[4/5] Installing Azure CLI..." + curl -sL https://aka.ms/InstallAzureCLIDeb | bash + + echo "[5/6] Pre-installing Azure CLI extensions for Arc, IoT Operations, and Foundry Local..." + export AZURE_EXTENSION_DIR=/usr/local/lib/azure-cli-extensions + mkdir -p "$AZURE_EXTENSION_DIR" + chmod 755 "$AZURE_EXTENSION_DIR" + echo 'export AZURE_EXTENSION_DIR=/usr/local/lib/azure-cli-extensions' >> /etc/profile.d/azure-cli-extensions.sh + az extension add --upgrade --name connectedk8s --yes || true + az extension add --upgrade --name k8s-extension --yes || true + az extension add --upgrade --name customlocation --yes || true + az extension add --upgrade --name azure-iot-ops --yes || true + su - "$ADMIN_USER" -c "az extension add --upgrade --name connectedk8s --yes" || true + su - "$ADMIN_USER" -c "az extension add --upgrade --name k8s-extension --yes" || true + su - "$ADMIN_USER" -c "az extension add --upgrade --name customlocation --yes" || true + su - "$ADMIN_USER" -c "az extension add --upgrade --name azure-iot-ops --yes" || true + + echo "[6/6] Pre-caching container images & Python wheels for disconnected air-gap resilience..." + mkdir -p /opt/edge-wheels + pip3 download --dest /opt/edge-wheels paho-mqtt==2.1.0 || true + k3s ctr images pull docker.io/library/python:3.12-slim || true + k3s ctr images pull docker.io/library/eclipse-mosquitto:2.0 || true + + echo "Edge host bootstrap completed successfully." > /var/log/edge-bootstrap-complete.log + +runcmd: + - [ bash, -c, "/usr/local/bin/bootstrap-edge-host.sh 2>&1 | tee -a /var/log/edge-bootstrap.log" ] diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/infra/main.bicep b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/infra/main.bicep new file mode 100644 index 000000000..2a188228b --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/infra/main.bicep @@ -0,0 +1,426 @@ +// ============================================================================ +// Air-Gapped Agentic AI: Orchestrating Local SLMs and IoT on Edge Kubernetes +// Provisions the Fast K3s Edge Host Environment: +// - Standard_D16s_v5 Ubuntu 24.04 VM with single-node K3s (~10 min) +// - Virtual Network, Subnet, NSG, and optional Azure Bastion +// - Storage Account (HNS enabled) for Azure IoT Operations Schema Registry +// - Key Vault for Edge Secrets & Certificates +// - User-Assigned Managed Identity for Arc & AIO onboarding +// ============================================================================ + +targetScope = 'resourceGroup' + +@description('Azure region for all resources.') +param location string = resourceGroup().location + +@description('Short unique prefix/suffix for resource names.') +@minLength(3) +@maxLength(12) +param environmentPrefix string = 'edgeai' + +@description('Edge simulation topology profile.') +@allowed([ + 'fast-k3s' +]) +param edgeProfile string = 'fast-k3s' + +@description('VM size for the Ubuntu K3s edge host.') +param vmSize string = 'Standard_D16s_v5' + +@description('Administrator username for the Edge Host VM.') +param adminUsername string = 'edgeadmin' + +@description('Administrator password or SSH public key for the Edge Host VM.') +@secure() +param adminPasswordOrKey string + +@description('Authentication type for the Linux fast-k3s VM.') +@allowed([ + 'password' + 'sshPublicKey' +]) +param authType string = 'password' + +@description('Whether to deploy Azure Bastion for browser-based SSH/RDP access.') +param deployBastion bool = true + +@description('Optional Entra Object ID of the lab participant to grant RBAC roles.') +param participantObjectId string = '' + +var uniqueSuffix = substring(uniqueString(resourceGroup().id, environmentPrefix), 0, 6) +var vmName = '${environmentPrefix}-host-${uniqueSuffix}' +var arcClusterName = '${environmentPrefix}-arck8s-${uniqueSuffix}' +var vnetName = '${environmentPrefix}-vnet-${uniqueSuffix}' +var nsgName = '${environmentPrefix}-nsg-${uniqueSuffix}' +var natGwName = '${environmentPrefix}-natgw-${uniqueSuffix}' +var natPipName = '${environmentPrefix}-natpip-${uniqueSuffix}' +var bastionName = '${environmentPrefix}-bastion-${uniqueSuffix}' +var bastionPipName = '${environmentPrefix}-bastion-pip-${uniqueSuffix}' +var storageAccountName = take('st${replace(environmentPrefix, '-', '')}${uniqueSuffix}', 24) +var keyVaultName = take('${environmentPrefix}-kv-${uniqueSuffix}', 24) +var identityName = '${environmentPrefix}-id-${uniqueSuffix}' + +var linuxConfiguration = { + disablePasswordAuthentication: true + ssh: { + publicKeys: [ + { + path: '/home/${adminUsername}/.ssh/authorized_keys' + keyData: adminPasswordOrKey + } + ] + } +} + +// Cloud-init script for the fast-k3s Ubuntu 24.04 edge host +var cloudInitScript = loadFileAsBase64('cloud-init-k3s.yaml') + +// ---------------------------------------------------------------------------- +// 1. User-Assigned Managed Identity for Edge Onboarding +// ---------------------------------------------------------------------------- +resource edgeIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = { + name: identityName + location: location +} + +// ---------------------------------------------------------------------------- +// 2. Networking (VNet, Subnets, NSG, Optional Azure Bastion) +// ---------------------------------------------------------------------------- +resource nsg 'Microsoft.Network/networkSecurityGroups@2023-11-01' = { + name: nsgName + location: location + properties: { + securityRules: [ + { + name: 'Allow-Internal-VNet' + properties: { + priority: 100 + direction: 'Inbound' + access: 'Allow' + protocol: '*' + sourceAddressPrefix: 'VirtualNetwork' + sourcePortRange: '*' + destinationAddressPrefix: 'VirtualNetwork' + destinationPortRange: '*' + } + } + ] + } +} + +resource natPip 'Microsoft.Network/publicIPAddresses@2023-11-01' = { + name: natPipName + location: location + sku: { + name: 'Standard' + } + properties: { + publicIPAllocationMethod: 'Static' + } +} + +resource natGateway 'Microsoft.Network/natGateways@2023-11-01' = { + name: natGwName + location: location + sku: { + name: 'Standard' + } + properties: { + idleTimeoutInMinutes: 10 + publicIpAddresses: [ + { + id: natPip.id + } + ] + } +} + +resource vnet 'Microsoft.Network/virtualNetworks@2023-11-01' = { + name: vnetName + location: location + properties: { + addressSpace: { + addressPrefixes: [ + '10.42.0.0/16' + ] + } + subnets: [ + { + name: 'snet-edge-host' + properties: { + addressPrefix: '10.42.1.0/24' + networkSecurityGroup: { + id: nsg.id + } + natGateway: { + id: natGateway.id + } + } + } + { + name: 'AzureBastionSubnet' + properties: { + addressPrefix: '10.42.2.0/26' + } + } + ] + } +} + +resource bastionPip 'Microsoft.Network/publicIPAddresses@2023-11-01' = if (deployBastion) { + name: bastionPipName + location: location + sku: { + name: 'Standard' + } + properties: { + publicIPAllocationMethod: 'Static' + } +} + +resource bastion 'Microsoft.Network/bastionHosts@2023-11-01' = if (deployBastion) { + name: bastionName + location: location + sku: { + name: 'Standard' + } + properties: { + enableTunneling: true + ipConfigurations: [ + { + name: 'IpConf' + properties: { + subnet: { + id: vnet.properties.subnets[1].id + } + publicIPAddress: { + id: bastionPip.id + } + } + } + ] + } +} + +// ---------------------------------------------------------------------------- +// 3. Storage Account (Hierarchical Namespace Enabled for AIO Schema Registry) +// ---------------------------------------------------------------------------- +resource schemaStorage 'Microsoft.Storage/storageAccounts@2023-05-01' = { + name: storageAccountName + location: location + sku: { + name: 'Standard_LRS' + } + kind: 'StorageV2' + properties: { + isHnsEnabled: true + minimumTlsVersion: 'TLS1_2' + allowBlobPublicAccess: false + supportsHttpsTrafficOnly: true + } +} + +resource blobServices 'Microsoft.Storage/storageAccounts/blobServices@2023-05-01' = { + parent: schemaStorage + name: 'default' +} + +resource schemaContainer 'Microsoft.Storage/storageAccounts/blobServices/containers@2023-05-01' = { + parent: blobServices + name: 'schemas' +} + +// ---------------------------------------------------------------------------- +// 4. Key Vault for Edge Certificates & Secrets +// ---------------------------------------------------------------------------- +resource keyVault 'Microsoft.KeyVault/vaults@2023-07-01' = { + name: keyVaultName + location: location + properties: { + sku: { + family: 'A' + name: 'standard' + } + tenantId: subscription().tenantId + enableRbacAuthorization: true + enableSoftDelete: true + softDeleteRetentionInDays: 7 + } +} + +resource kvEdgeAdminUsernameSecret 'Microsoft.KeyVault/vaults/secrets@2023-07-01' = { + parent: keyVault + name: 'EdgeHostAdminUsername' + properties: { + value: adminUsername + contentType: 'text/plain' + } +} + +resource kvEdgeAdminPasswordSecret 'Microsoft.KeyVault/vaults/secrets@2023-07-01' = { + parent: keyVault + name: 'EdgeHostAdminPassword' + properties: { + value: adminPasswordOrKey + contentType: 'password' + } +} + +// ---------------------------------------------------------------------------- +// 5. Edge Host NIC & Virtual Machine +// ---------------------------------------------------------------------------- +resource nic 'Microsoft.Network/networkInterfaces@2023-11-01' = { + name: '${vmName}-nic' + location: location + properties: { + ipConfigurations: [ + { + name: 'ipconfig1' + properties: { + privateIPAllocationMethod: 'Dynamic' + subnet: { + id: vnet.properties.subnets[0].id + } + } + } + ] + } +} + +resource edgeVm 'Microsoft.Compute/virtualMachines@2024-03-01' = { + name: vmName + location: location + identity: { + type: 'SystemAssigned, UserAssigned' + userAssignedIdentities: { + '${edgeIdentity.id}': {} + } + } + properties: { + hardwareProfile: { + vmSize: vmSize + } + osProfile: { + computerName: take(vmName, 15) + adminUsername: adminUsername + adminPassword: adminPasswordOrKey + customData: edgeProfile == 'fast-k3s' ? cloudInitScript : null + linuxConfiguration: (edgeProfile == 'fast-k3s' && authType == 'sshPublicKey') ? linuxConfiguration : null + } + storageProfile: { + imageReference: edgeProfile == 'fast-k3s' + ? { + publisher: 'Canonical' + offer: 'ubuntu-24_04-lts' + sku: 'server' + version: 'latest' + } + : { + publisher: 'MicrosoftWindowsServer' + offer: 'WindowsServer' + sku: '2025-datacenter-azure-edition' + version: 'latest' + } + osDisk: { + name: '${vmName}-osdisk' + createOption: 'FromImage' + diskSizeGB: 256 + managedDisk: { + storageAccountType: 'Premium_LRS' + } + } + dataDisks: [ + { + lun: 0 + name: '${vmName}-modelcache-disk01' + createOption: 'Empty' + diskSizeGB: 256 + caching: 'ReadOnly' + managedDisk: { + storageAccountType: 'Premium_LRS' + } + } + { + lun: 1 + name: '${vmName}-iotdata-disk02' + createOption: 'Empty' + diskSizeGB: 256 + caching: 'ReadOnly' + managedDisk: { + storageAccountType: 'Premium_LRS' + } + } + ] + } + networkProfile: { + networkInterfaces: [ + { + id: nic.id + } + ] + } + } +} + +// ---------------------------------------------------------------------------- +// 6. RBAC Role Assignments for Edge Managed Identity & Optional Participant +// ---------------------------------------------------------------------------- +// Storage Blob Data Contributor on Schema Registry Storage +var storageBlobDataContributorRoleId = subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'ba92f5b4-2d11-453d-a403-e96b0029c9fe') +// Contributor on Resource Group (allows Edge Identity to onboard Arc, AIO, and Extensions) +var contributorRoleId = subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'b24988ac-6180-42a0-ab88-20f7382dd24c') +// Role Based Access Control Administrator on Resource Group (required by az iot ops create) +var rbacAdminRoleId = subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'f58310d9-a9f6-439a-9e8d-f62e7b41a168') + +resource identityStorageRole 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: guid(schemaStorage.id, edgeIdentity.id, storageBlobDataContributorRoleId) + scope: schemaStorage + properties: { + principalId: edgeIdentity.properties.principalId + principalType: 'ServicePrincipal' + roleDefinitionId: storageBlobDataContributorRoleId + } +} + +resource identityContributorRole 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: guid(resourceGroup().id, edgeIdentity.id, contributorRoleId) + scope: resourceGroup() + properties: { + principalId: edgeIdentity.properties.principalId + principalType: 'ServicePrincipal' + roleDefinitionId: contributorRoleId + } +} + +resource identityRbacAdminRole 'Microsoft.Authorization/roleAssignments@2022-04-01' = { + name: guid(resourceGroup().id, edgeIdentity.id, rbacAdminRoleId) + scope: resourceGroup() + properties: { + principalId: edgeIdentity.properties.principalId + principalType: 'ServicePrincipal' + roleDefinitionId: rbacAdminRoleId + } +} + +resource participantStorageRole 'Microsoft.Authorization/roleAssignments@2022-04-01' = if (!empty(participantObjectId)) { + name: guid(schemaStorage.id, participantObjectId, storageBlobDataContributorRoleId) + scope: schemaStorage + properties: { + principalId: participantObjectId + principalType: 'User' + roleDefinitionId: storageBlobDataContributorRoleId + } +} + +// ---------------------------------------------------------------------------- +// Outputs +// ---------------------------------------------------------------------------- +output edgeVmName string = edgeVm.name +output edgeProfileUsed string = edgeProfile +output vmSizeUsed string = vmSize +output arcClusterSuggestedName string = arcClusterName +output schemaStorageAccountName string = schemaStorage.name +output schemaStorageAccountId string = schemaStorage.id +output keyVaultName string = keyVault.name +output userAssignedIdentityId string = edgeIdentity.id +output userAssignedIdentityClientId string = edgeIdentity.properties.clientId diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/infra/register-providers.sh b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/infra/register-providers.sh new file mode 100644 index 000000000..fbc1aa858 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/infra/register-providers.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# ============================================================================ +# Registers all Azure Resource Providers required for: +# - Azure Arc-enabled Kubernetes (connectedClusters & extensions) +# - Azure IoT Operations (AIO, Device Registry, SecretSync) +# - Foundry Local on Azure Arc-enabled Kubernetes (Microsoft.Foundry & CertManagement) +# ============================================================================ +set -euo pipefail + +PROVIDERS=( + "Microsoft.Compute" + "Microsoft.Network" + "Microsoft.Storage" + "Microsoft.KeyVault" + "Microsoft.ExtendedLocation" + "Microsoft.Kubernetes" + "Microsoft.KubernetesConfiguration" + "Microsoft.IoTOperations" + "Microsoft.DeviceRegistry" + "Microsoft.SecretSyncController" +) + +echo "Registering required Azure Resource Providers..." +for rp in "${PROVIDERS[@]}"; do + echo " -> Registering $rp" + az provider register --namespace "$rp" --wait +done + +echo "All required resource providers are registered:" +az provider list --query "[?contains(['Microsoft.Kubernetes','Microsoft.KubernetesConfiguration','Microsoft.ExtendedLocation','Microsoft.IoTOperations','Microsoft.DeviceRegistry','Microsoft.SecretSyncController'], namespace)].{Provider:namespace, State:registrationState}" --output table diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/01-iot-sim/aio-anomaly-dataflow.yaml b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/01-iot-sim/aio-anomaly-dataflow.yaml new file mode 100644 index 000000000..21283a9b0 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/01-iot-sim/aio-anomaly-dataflow.yaml @@ -0,0 +1,55 @@ +# ============================================================================ +# Azure IoT Operations — Internal In-Cluster MQTT BrokerListener & Anomaly Dataflow +# Exposes a cluster-internal MQTT listener for edge workloads and routes +# high-temperature / high-vibration events from factory/line1/telemetry +# to factory/line1/anomalies. +# ============================================================================ +apiVersion: mqttbroker.iotoperations.azure.com/v1 +kind: BrokerListener +metadata: + name: aio-broker-internal + namespace: azure-iot-operations +spec: + brokerRef: default + serviceName: aio-broker-internal + serviceType: NodePort + ports: + - port: 1883 + nodePort: 31883 + protocol: Mqtt +--- +apiVersion: connectivity.iotoperations.azure.com/v1 +kind: Dataflow +metadata: + name: cnc-anomaly-router + namespace: azure-iot-operations +spec: + profileRef: default + mode: Enabled + operations: + - operationType: Source + name: source-telemetry + sourceSettings: + endpointRef: default + dataSources: + - factory/line1/telemetry + - operationType: BuiltInTransformation + name: filter-anomalies + builtInTransformationSettings: + # Note: In Azure IoT Operations Dataflows, when a filter expression evaluates to true, + # the message is DROPPED. Dropping messages where (bearingTempC <= 85.0 && vibrationMmPerSec <= 7.5) + # ensures ONLY anomalous readings pass through to factory/line1/anomalies. + filter: + - inputs: + - bearingTempC + - vibrationMmPerSec + expression: "$1 <= 85.0 && $2 <= 7.5" + map: + - inputs: + - "*" + output: "*" + - operationType: Destination + name: destination-anomalies + destinationSettings: + endpointRef: default + dataDestination: factory/line1/anomalies diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/01-iot-sim/telemetry-simulator.yaml b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/01-iot-sim/telemetry-simulator.yaml new file mode 100644 index 000000000..c5c45531c --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/01-iot-sim/telemetry-simulator.yaml @@ -0,0 +1,157 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: edge-agents + labels: + app.kubernetes.io/part-of: sovereign-edge-ai-microhack +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: telemetry-simulator-script + namespace: edge-agents +data: + simulator.py: | + import json + import os + import random + import time + from datetime import datetime, timezone + import paho.mqtt.client as mqtt + + MQTT_HOST = os.getenv("MQTT_HOST", "aio-broker-internal.azure-iot-operations.svc.cluster.local") + MQTT_PORT = int(os.getenv("MQTT_PORT", "1883")) + TELEMETRY_TOPIC = os.getenv("TELEMETRY_TOPIC", "factory/line1/telemetry") + COMMANDS_TOPIC = os.getenv("COMMANDS_TOPIC", "factory/line1/commands") + MACHINE_ID = os.getenv("MACHINE_ID", "CNC-EXTRUDER-01") + ANOMALY_EVERY_N = int(os.getenv("ANOMALY_EVERY_N", "6")) + + state = { + "spindleRpm": 4200, + "coolantBoost": False, + "lastAction": "NONE", + } + + def on_connect(client, userdata, flags, reason_code, properties=None): + print(f"[SIMULATOR] Connected to AIO MQTT Broker at {MQTT_HOST}:{MQTT_PORT} (rc={reason_code})", flush=True) + client.subscribe(COMMANDS_TOPIC, qos=1) + print(f"[SIMULATOR] Subscribed to PLC actuation topic: {COMMANDS_TOPIC}", flush=True) + + def on_message(client, userdata, msg): + try: + payload = json.loads(msg.payload.decode("utf-8")) + action = payload.get("decision", {}).get("tool_name", "UNKNOWN") + args = payload.get("decision", {}).get("arguments", {}) + print(f"[SIMULATOR] <<< RECEIVED AUTONOMOUS PLC COMMAND on {msg.topic}: action={action} args={json.dumps(args)}", flush=True) + if action in ("adjust_plc_setpoint", "trigger_safety_interlock"): + rpm_delta_pct = args.get("spindle_rpm_delta_pct", -30) + state["spindleRpm"] = max(1200, int(4200 * (100 + rpm_delta_pct) / 100)) + state["coolantBoost"] = bool(args.get("enable_coolant_boost", True)) + state["lastAction"] = action + print(f"[SIMULATOR] === PLC ACTUATED: spindleRpm={state['spindleRpm']}, coolantBoost={state['coolantBoost']} ===", flush=True) + except Exception as exc: + print(f"[SIMULATOR] Error processing command message: {exc}", flush=True) + + client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=f"sim-{MACHINE_ID}") + client.on_connect = on_connect + client.on_message = on_message + + while True: + try: + client.connect(MQTT_HOST, MQTT_PORT, keepalive=60) + break + except Exception as exc: + print(f"[SIMULATOR] Waiting for AIO MQTT broker ({MQTT_HOST}:{MQTT_PORT}): {exc}", flush=True) + time.sleep(5) + + client.loop_start() + tick = 0 + while True: + tick += 1 + is_anomaly = (tick % ANOMALY_EVERY_N == 0) + if is_anomaly: + bearing_temp = round(random.uniform(91.5, 104.2), 1) + vibration = round(random.uniform(8.4, 12.6), 2) + coolant_flow = round(random.uniform(2.1, 3.4), 1) + status = "WARNING_THERMAL_VIBRATION_DRIFT" + else: + bearing_temp = round(random.uniform(62.0, 74.5), 1) + vibration = round(random.uniform(1.8, 3.9), 2) + coolant_flow = round(random.uniform(7.5, 9.2), 1) if not state["coolantBoost"] else round(random.uniform(11.0, 12.5), 1) + status = "NOMINAL" + + event = { + "machineId": MACHINE_ID, + "timestamp": datetime.now(timezone.utc).isoformat(), + "tick": tick, + "spindleRpm": state["spindleRpm"], + "bearingTempC": bearing_temp, + "vibrationMmPerSec": vibration, + "coolantFlowLpm": coolant_flow, + "coolantBoostActive": state["coolantBoost"], + "lastAutonomousAction": state["lastAction"], + "status": status, + } + client.publish(TELEMETRY_TOPIC, json.dumps(event), qos=1) + print(f"[SIMULATOR] >>> Published {TELEMETRY_TOPIC} (tick={tick}, status={status}, temp={bearing_temp}C, vib={vibration}mm/s)", flush=True) + time.sleep(5) +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: manufacturing-telemetry-simulator + namespace: edge-agents + labels: + app: manufacturing-telemetry-simulator +spec: + replicas: 1 + selector: + matchLabels: + app: manufacturing-telemetry-simulator + template: + metadata: + labels: + app: manufacturing-telemetry-simulator + spec: + containers: + - name: simulator + image: python:3.12-slim + imagePullPolicy: IfNotPresent + command: + - /bin/sh + - -c + - | + (pip install --no-index --find-links=/opt/edge-wheels paho-mqtt==2.1.0 || pip install --no-cache-dir paho-mqtt==2.1.0) && python /app/simulator.py + env: + - name: MQTT_HOST + value: "aio-broker-internal.azure-iot-operations.svc.cluster.local" + - name: MQTT_PORT + value: "1883" + - name: TELEMETRY_TOPIC + value: "factory/line1/telemetry" + - name: COMMANDS_TOPIC + value: "factory/line1/commands" + - name: ANOMALY_EVERY_N + value: "6" + volumeMounts: + - name: script-vol + mountPath: /app + readOnly: true + - name: wheel-cache-vol + mountPath: /opt/edge-wheels + readOnly: true + resources: + requests: + cpu: "100m" + memory: "128Mi" + limits: + cpu: "250m" + memory: "256Mi" + volumes: + - name: script-vol + configMap: + name: telemetry-simulator-script + - name: wheel-cache-vol + hostPath: + path: /opt/edge-wheels + type: DirectoryOrCreate diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/02-foundry-local/caller-service-account.yaml b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/02-foundry-local/caller-service-account.yaml new file mode 100644 index 000000000..ede1ca8a2 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/02-foundry-local/caller-service-account.yaml @@ -0,0 +1,17 @@ +# ============================================================================ +# Dedicated Kubernetes ServiceAccount for Foundry Local SAT Authentication +# Reference: https://learn.microsoft.com/azure/azure-sovereign-clouds/private/foundry-local/how-to-configure-service-account-token-authentication +# ============================================================================ +apiVersion: v1 +kind: Namespace +metadata: + name: edge-agents + labels: + app.kubernetes.io/part-of: sovereign-edge-ai-microhack +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: inference-client + namespace: edge-agents +automountServiceAccountToken: false diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/02-foundry-local/model-deployment-cpu.yaml b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/02-foundry-local/model-deployment-cpu.yaml new file mode 100644 index 000000000..d87c0a2e0 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/02-foundry-local/model-deployment-cpu.yaml @@ -0,0 +1,38 @@ +# ============================================================================ +# Foundry Local ModelDeployment Custom Resource (CPU / ONNX Runtime GenAI) +# Deploys Phi-4-mini-instruct from the on-cluster Foundry Local catalog, +# exposes it internally via the Istio Gateway API (inference-internal-gateway), +# and authorizes the 'inference-client' ServiceAccount in 'edge-agents' +# via Kubernetes Service Account Token (SAT) authentication. +# ============================================================================ +apiVersion: foundrylocal.azure.com/v1 +kind: ModelDeployment +metadata: + name: phi-4-mini + namespace: foundry-local-operator +spec: + model: + catalog: + name: phi-4-mini + version: "latest" + workloadType: generative + compute: cpu + runtime: onnx-genai + replicas: 1 + endpoint: + exposure: internal + authenticationMethods: + - method: serviceAccountToken + enabled: true + roles: + - role: GeneralInferenceUser + authorizedSubjects: + - serviceAccountName: inference-client + serviceAccountNamespace: edge-agents + resources: + requests: + cpu: "4" + memory: "8Gi" + limits: + cpu: "8" + memory: "16Gi" diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/03-edge-agent/edge-agent-deployment.yaml b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/03-edge-agent/edge-agent-deployment.yaml new file mode 100644 index 000000000..87e8015d2 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/03-edge-agent/edge-agent-deployment.yaml @@ -0,0 +1,251 @@ +# ============================================================================ +# Autonomous Edge Triage & Actuation Agent Deployment +# - Runs under the 'inference-client' ServiceAccount +# - Mounts a projected Kubernetes Service Account Token (audience: foundry-local) +# at /var/run/secrets/foundry-local/token +# - Subscribes to 'factory/line1/anomalies' on the AIO MQTT Broker +# - Invokes Foundry Local (/v1/chat/completions) over the internal Istio Gateway +# - Publishes autonomous PLC commands to 'factory/line1/commands' +# ============================================================================ +apiVersion: v1 +kind: ConfigMap +metadata: + name: edge-agent-code + namespace: edge-agents +data: + agent.py: | + import json + import os + import ssl + import time + import urllib.request + from datetime import datetime, timezone + import paho.mqtt.client as mqtt + + MQTT_HOST = os.getenv("MQTT_HOST", "aio-broker-internal.azure-iot-operations.svc.cluster.local") + MQTT_PORT = int(os.getenv("MQTT_PORT", "1883")) + ANOMALIES_TOPIC = os.getenv("ANOMALIES_TOPIC", "factory/line1/anomalies") + COMMANDS_TOPIC = os.getenv("COMMANDS_TOPIC", "factory/line1/commands") + FOUNDRY_ENDPOINT = os.getenv( + "FOUNDRY_ENDPOINT", + "https://inference-internal-gateway-istio.foundry-local-operator.svc.cluster.local:443/phi-4-mini/v1/chat/completions", + ) + MODEL_NAME = os.getenv("MODEL_NAME", "Phi-4-mini-instruct-generic-cpu:5") + SAT_TOKEN_PATH = os.getenv("SAT_TOKEN_PATH", "/var/run/secrets/foundry-local/token") + SOP_PATH = os.getenv("SOP_PATH", "/etc/edge-sop/sop.md") + + def read_local_sop() -> str: + try: + with open(SOP_PATH, "r", encoding="utf-8") as f: + return f.read() + except Exception as exc: + return f"Fallback SOP: If temp >= 105C or vibration >= 13.0, call trigger_safety_interlock (-60% RPM). Otherwise call adjust_plc_setpoint (-25% RPM). ({exc})" + + def read_sat_token() -> str: + """Always re-read the projected Kubernetes Service Account Token so rotated tokens are picked up.""" + with open(SAT_TOKEN_PATH, "r", encoding="utf-8") as f: + return f.read().strip() + + def invoke_foundry_local(telemetry: dict, sop_text: str) -> dict: + token = read_sat_token() + system_prompt = ( + "You are an autonomous industrial edge AI agent. Evaluate the sensor anomaly against the SOP and reply ONLY with one compact JSON object on a single line (no markdown, no explanation outside JSON):\n" + '{"tool_name":"adjust_plc_setpoint"|"trigger_safety_interlock","arguments":{"spindle_rpm_delta_pct":int,"enable_coolant_boost":true,"interlock_mode":str,"priority":str},"reasoning":str}\n' + f"{sop_text}" + ) + request_body = { + "model": MODEL_NAME, + "messages": [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": json.dumps(telemetry)}, + ], + "temperature": 0.0, + "max_tokens": 140, + } + + ctx = ssl.create_default_context() + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + + req = urllib.request.Request( + FOUNDRY_ENDPOINT, + data=json.dumps(request_body).encode("utf-8"), + headers={ + "Content-Type": "application/json", + "Authorization": f"Bearer {token}", + }, + method="POST", + ) + start = time.perf_counter() + with urllib.request.urlopen(req, context=ctx, timeout=60) as resp: + raw = json.loads(resp.read().decode("utf-8")) + latency_ms = round((time.perf_counter() - start) * 1000, 1) + + content = raw["choices"][0]["message"]["content"].strip() + if "{" in content and "}" in content: + content = content[content.find("{"):content.rfind("}") + 1] + decision = json.loads(content) + decision["inferenceLatencyMs"] = latency_ms + return decision + + def evaluate_anomaly(client: mqtt.Client, telemetry: dict): + sop_text = read_local_sop() + start_ts = time.perf_counter() + try: + decision = invoke_foundry_local(telemetry, sop_text) + source = "foundry-local-onnx-genai" + except Exception as exc: + print(f"[EDGE-AGENT] Foundry Local call notice ({exc}); applying deterministic SOP fallback.", flush=True) + is_critical = ( + telemetry.get("bearingTempC", 0) >= 105.0 + or telemetry.get("vibrationMmPerSec", 0) >= 13.0 + or telemetry.get("coolantFlowLpm", 10) < 2.0 + ) + decision = { + "tool_name": "trigger_safety_interlock" if is_critical else "adjust_plc_setpoint", + "arguments": { + "spindle_rpm_delta_pct": -60 if is_critical else -25, + "enable_coolant_boost": True, + "interlock_mode": "SAFE_HOLD" if is_critical else "CLOSED_LOOP_THROTTLE", + "priority": "P1-CRITICAL-INTERLOCK" if is_critical else "P2-WARNING", + }, + "reasoning": f"SOP-CNC-042 rule evaluation for temp={telemetry.get('bearingTempC')}C, vib={telemetry.get('vibrationMmPerSec')}mm/s", + "inferenceLatencyMs": round((time.perf_counter() - start_ts) * 1000, 1), + } + source = "local-sop-deterministic-guard" + + command_event = { + "commandId": f"cmd-{int(time.time() * 1000)}", + "machineId": telemetry.get("machineId", "CNC-EXTRUDER-01"), + "timestamp": datetime.now(timezone.utc).isoformat(), + "triggerTelemetry": { + "bearingTempC": telemetry.get("bearingTempC"), + "vibrationMmPerSec": telemetry.get("vibrationMmPerSec"), + "coolantFlowLpm": telemetry.get("coolantFlowLpm"), + }, + "authMethod": "Kubernetes-SAT (audience: foundry-local)", + "inferenceEngine": source, + "decision": decision, + } + client.publish(COMMANDS_TOPIC, json.dumps(command_event), qos=1) + print( + f"[EDGE-AGENT] >>> Published autonomous actuation to {COMMANDS_TOPIC}: " + f"tool={decision.get('tool_name')} latency={decision.get('inferenceLatencyMs')}ms " + f"args={json.dumps(decision.get('arguments'))}", + flush=True, + ) + + def on_connect(client, userdata, flags, reason_code, properties=None): + print(f"[EDGE-AGENT] Connected to AIO MQTT Broker at {MQTT_HOST}:{MQTT_PORT} (rc={reason_code})", flush=True) + client.subscribe(ANOMALIES_TOPIC, qos=1) + print(f"[EDGE-AGENT] Subscribed to anomaly topic: {ANOMALIES_TOPIC}", flush=True) + + def on_message(client, userdata, msg): + try: + telemetry = json.loads(msg.payload.decode("utf-8")) + print( + f"[EDGE-AGENT] <<< Anomaly received on {msg.topic}: " + f"machine={telemetry.get('machineId')} temp={telemetry.get('bearingTempC')}C " + f"vib={telemetry.get('vibrationMmPerSec')}mm/s", + flush=True, + ) + evaluate_anomaly(client, telemetry) + except Exception as exc: + print(f"[EDGE-AGENT] Error handling message: {exc}", flush=True) + + if __name__ == "__main__": + client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id="autonomous-edge-agent-01") + client.on_connect = on_connect + client.on_message = on_message + while True: + try: + client.connect(MQTT_HOST, MQTT_PORT, keepalive=60) + break + except Exception as exc: + print(f"[EDGE-AGENT] Waiting for AIO MQTT Broker ({MQTT_HOST}:{MQTT_PORT}): {exc}", flush=True) + time.sleep(5) + client.loop_forever() +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: autonomous-edge-agent + namespace: edge-agents + labels: + app: autonomous-edge-agent +spec: + replicas: 1 + selector: + matchLabels: + app: autonomous-edge-agent + template: + metadata: + labels: + app: autonomous-edge-agent + spec: + serviceAccountName: inference-client + automountServiceAccountToken: false + containers: + - name: agent + image: python:3.12-slim + imagePullPolicy: IfNotPresent + command: + - /bin/sh + - -c + - | + (pip install --no-index --find-links=/opt/edge-wheels paho-mqtt==2.1.0 || pip install --no-cache-dir paho-mqtt==2.1.0) && python /app/agent.py + env: + - name: MQTT_HOST + value: "aio-broker-internal.azure-iot-operations.svc.cluster.local" + - name: MQTT_PORT + value: "1883" + - name: ANOMALIES_TOPIC + value: "factory/line1/anomalies" + - name: COMMANDS_TOPIC + value: "factory/line1/commands" + - name: FOUNDRY_ENDPOINT + value: "https://inference-internal-gateway-istio.foundry-local-operator.svc.cluster.local:443/phi-4-mini/v1/chat/completions" + - name: MODEL_NAME + value: "Phi-4-mini-instruct-generic-cpu:5" + - name: SAT_TOKEN_PATH + value: "/var/run/secrets/foundry-local/token" + - name: SOP_PATH + value: "/etc/edge-sop/sop.md" + volumeMounts: + - name: agent-code-vol + mountPath: /app + readOnly: true + - name: local-sop-vol + mountPath: /etc/edge-sop + readOnly: true + - name: foundry-local-sat + mountPath: /var/run/secrets/foundry-local + readOnly: true + - name: wheel-cache-vol + mountPath: /opt/edge-wheels + readOnly: true + resources: + requests: + cpu: "200m" + memory: "256Mi" + limits: + cpu: "500m" + memory: "512Mi" + volumes: + - name: agent-code-vol + configMap: + name: edge-agent-code + - name: local-sop-vol + configMap: + name: local-sop-runbook + - name: foundry-local-sat + projected: + sources: + - serviceAccountToken: + path: token + audience: foundry-local + expirationSeconds: 3600 + - name: wheel-cache-vol + hostPath: + path: /opt/edge-wheels + type: DirectoryOrCreate diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/03-edge-agent/local-sop-configmap.yaml b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/03-edge-agent/local-sop-configmap.yaml new file mode 100644 index 000000000..badc8d3e3 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/03-edge-agent/local-sop-configmap.yaml @@ -0,0 +1,30 @@ +# ============================================================================ +# Local Edge SOP Runbook ConfigMap (SOP-CNC-042) +# Stored on-cluster so the Autonomous Edge Agent grounds its decisions +# without any cloud vector database or external network dependency. +# ============================================================================ +apiVersion: v1 +kind: ConfigMap +metadata: + name: local-sop-runbook + namespace: edge-agents +data: + sop.md: | + # Contoso Sovereign Manufacturing — Edge Standard Operating Procedure (SOP-CNC-042) + + ## Machine Class: CNC-EXTRUDER Series (Line 1) + - Nominal Bearing Temperature: 60.0°C – 78.0°C + - Warning Bearing Temperature: 85.0°C – 104.9°C + - Critical Bearing Thermal Runaway: >= 105.0°C + - Nominal Vibration (RMS): 1.5 – 4.5 mm/s + - Warning Vibration (RMS): 7.5 – 12.9 mm/s + - Critical Bearing Seizure Vibration: >= 13.0 mm/s + - Minimum Safe Coolant Flow: 4.0 LPM + + ## Mandatory Autonomous Mitigation Rules + 1. **Thermal / Vibration Warning (`85°C <= bearingTempC < 105°C` OR `7.5 <= vibrationMmPerSec < 13.0`)**: + - Call tool `adjust_plc_setpoint` with `spindle_rpm_delta_pct: -25` and `enable_coolant_boost: true`. + - Record local work order priority `P2-WARNING`. + 2. **Critical Thermal Runaway or Bearing Seizure (`bearingTempC >= 105°C` OR `vibrationMmPerSec >= 13.0` OR `coolantFlowLpm < 2.0`)**: + - Immediately call tool `trigger_safety_interlock` with `spindle_rpm_delta_pct: -60`, `enable_coolant_boost: true`, and `interlock_mode: "SAFE_HOLD"`. + - Record local work order priority `P1-CRITICAL-INTERLOCK`. diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/04-airgap-policy/deny-external-egress.yaml b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/04-airgap-policy/deny-external-egress.yaml new file mode 100644 index 000000000..b5c4fa0d2 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/04-airgap-policy/deny-external-egress.yaml @@ -0,0 +1,56 @@ +# ============================================================================ +# Challenge 5: "Cut the Cord" — Sovereign Air-Gap NetworkPolicy +# Blocks all external internet / public Azure cloud egress from the +# 'edge-agents' namespace while permitting strictly local in-cluster communication: +# 1. CoreDNS (UDP/TCP 53 in kube-system) +# 2. Azure IoT Operations MQTT Broker (TCP 1883/18883 in azure-iot-operations) +# 3. Foundry Local Internal Gateway & Inference pods (TCP 80/443/8080 in foundry-local-operator & istio-system) +# ============================================================================ +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: enforce-airgap-sovereign-mode + namespace: edge-agents +spec: + podSelector: {} + policyTypes: + - Egress + egress: + # 1. Allow in-cluster CoreDNS resolution + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 + # 2. Allow local MQTT communication with Azure IoT Operations + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: azure-iot-operations + ports: + - protocol: TCP + port: 1883 + - protocol: TCP + port: 18883 + # 3. Allow local LLM inference calls to Foundry Local & Istio Internal Gateway + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: foundry-local-operator + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: istio-system + ports: + - protocol: TCP + port: 80 + - protocol: TCP + port: 443 + - protocol: TCP + port: 8080 + # 4. Allow intra-namespace communication within edge-agents + - to: + - podSelector: {} diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/04-airgap-policy/inject-critical-fault.yaml b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/04-airgap-policy/inject-critical-fault.yaml new file mode 100644 index 000000000..9def1b207 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/manifests/04-airgap-policy/inject-critical-fault.yaml @@ -0,0 +1,31 @@ +# ============================================================================ +# Challenge 5: Critical Bearing Thermal Runaway Fault Injector +# Runs inside the air-gapped 'edge-agents' namespace and publishes a critical +# emergency telemetry event to verify autonomous sub-3-second local actuation +# with zero public cloud connectivity. +# ============================================================================ +apiVersion: batch/v1 +kind: Job +metadata: + name: inject-critical-bearing-fault + namespace: edge-agents +spec: + ttlSecondsAfterFinished: 300 + template: + metadata: + labels: + app: fault-injector + spec: + restartPolicy: Never + containers: + - name: fault-injector + image: eclipse-mosquitto:2.0 + imagePullPolicy: IfNotPresent + command: + - /bin/sh + - -c + - | + PAYLOAD='{"machineId":"CNC-EXTRUDER-01","timestamp":"2026-10-18T12:00:00Z","tick":9999,"spindleRpm":4200,"bearingTempC":114.6,"vibrationMmPerSec":15.2,"coolantFlowLpm":1.4,"status":"CRITICAL_THERMAL_RUNAWAY_BEARING_SEIZURE"}' + echo "[FAULT-INJECTOR] Publishing critical fault to factory/line1/telemetry..." + mosquitto_pub -h aio-broker-internal.azure-iot-operations.svc.cluster.local -p 1883 -t "factory/line1/telemetry" -q 1 -m "$PAYLOAD" + echo "[FAULT-INJECTOR] Critical fault injected successfully." diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/src/edge-agent/Dockerfile b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/src/edge-agent/Dockerfile new file mode 100644 index 000000000..283157c86 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/src/edge-agent/Dockerfile @@ -0,0 +1,7 @@ +FROM python:3.12-slim +WORKDIR /app +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt +COPY agent.py . +USER 1000:1000 +CMD ["python", "/app/agent.py"] diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/src/edge-agent/agent.py b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/src/edge-agent/agent.py new file mode 100644 index 000000000..18211e7ac --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/src/edge-agent/agent.py @@ -0,0 +1,116 @@ +""" +Autonomous Edge Triage & Actuation Agent (Reference Source) +----------------------------------------------------------- +Connects Azure IoT Operations (MQTT Broker) to Foundry Local (/v1/chat/completions) +using Kubernetes Service Account Token (SAT) authentication and local SOP grounding. +""" +import json +import os +import ssl +import time +import urllib.request +from datetime import datetime, timezone +import paho.mqtt.client as mqtt + +MQTT_HOST = os.getenv("MQTT_HOST", "aio-broker-internal.azure-iot-operations.svc.cluster.local") +MQTT_PORT = int(os.getenv("MQTT_PORT", "1883")) +ANOMALIES_TOPIC = os.getenv("ANOMALIES_TOPIC", "factory/line1/anomalies") +COMMANDS_TOPIC = os.getenv("COMMANDS_TOPIC", "factory/line1/commands") +FOUNDRY_ENDPOINT = os.getenv( + "FOUNDRY_ENDPOINT", + "https://inference-internal-gateway-istio.foundry-local-operator.svc.cluster.local:443/phi-4-mini/v1/chat/completions", +) +MODEL_NAME = os.getenv("MODEL_NAME", "Phi-4-mini-instruct-generic-cpu:5") +SAT_TOKEN_PATH = os.getenv("SAT_TOKEN_PATH", "/var/run/secrets/foundry-local/token") +SOP_PATH = os.getenv("SOP_PATH", "/etc/edge-sop/sop.md") + + +def read_local_sop() -> str: + with open(SOP_PATH, "r", encoding="utf-8") as f: + return f.read() + + +def read_sat_token() -> str: + """Re-read the projected Kubernetes SAT token on every request to support automatic rotation.""" + with open(SAT_TOKEN_PATH, "r", encoding="utf-8") as f: + return f.read().strip() + + +def invoke_foundry_local(telemetry: dict, sop_text: str) -> dict: + token = read_sat_token() + system_prompt = ( + "You are an autonomous industrial edge AI agent running inside an air-gapped factory Kubernetes cluster.\n" + "Analyze the incoming machine anomaly against the following Local SOP Runbook and respond ONLY with a valid JSON object " + "matching this schema:\n" + '{"tool_name": "adjust_plc_setpoint" | "trigger_safety_interlock", ' + '"arguments": {"spindle_rpm_delta_pct": int, "enable_coolant_boost": bool, "interlock_mode": str, "priority": str}, ' + '"reasoning": str}\n\n' + f"--- LOCAL SOP RUNBOOK ---\n{sop_text}" + ) + request_body = { + "model": MODEL_NAME, + "messages": [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": f"Telemetry anomaly received: {json.dumps(telemetry)}"}, + ], + "temperature": 0.0, + "max_tokens": 256, + } + + ctx = ssl.create_default_context() + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + + req = urllib.request.Request( + FOUNDRY_ENDPOINT, + data=json.dumps(request_body).encode("utf-8"), + headers={ + "Content-Type": "application/json", + "Authorization": f"Bearer {token}", + }, + method="POST", + ) + start = time.perf_counter() + with urllib.request.urlopen(req, context=ctx, timeout=20) as resp: + raw = json.loads(resp.read().decode("utf-8")) + latency_ms = round((time.perf_counter() - start) * 1000, 1) + + content = raw["choices"][0]["message"]["content"].strip() + if "```" in content: + content = content.split("```")[1].replace("json", "", 1).strip() + decision = json.loads(content) + decision["inferenceLatencyMs"] = latency_ms + return decision + + +def evaluate_anomaly(client: mqtt.Client, telemetry: dict) -> None: + sop_text = read_local_sop() + decision = invoke_foundry_local(telemetry, sop_text) + command_event = { + "commandId": f"cmd-{int(time.time() * 1000)}", + "machineId": telemetry.get("machineId", "CNC-EXTRUDER-01"), + "timestamp": datetime.now(timezone.utc).isoformat(), + "authMethod": "Kubernetes-SAT (audience: foundry-local)", + "inferenceEngine": "foundry-local-onnx-genai", + "decision": decision, + } + client.publish(COMMANDS_TOPIC, json.dumps(command_event), qos=1) + print(f"[EDGE-AGENT] Published autonomous PLC command: {json.dumps(command_event)}", flush=True) + + +def on_connect(client, userdata, flags, reason_code, properties=None): + print(f"[EDGE-AGENT] Connected to AIO MQTT Broker at {MQTT_HOST}:{MQTT_PORT}", flush=True) + client.subscribe(ANOMALIES_TOPIC, qos=1) + + +def on_message(client, userdata, msg): + telemetry = json.loads(msg.payload.decode("utf-8")) + evaluate_anomaly(client, telemetry) + + +if __name__ == "__main__": + mqtt_client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id="autonomous-edge-agent-01") + mqtt_client.on_connect = on_connect + mqtt_client.on_message = on_message + mqtt_client.connect(MQTT_HOST, MQTT_PORT, keepalive=60) + mqtt_client.loop_forever() diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/src/edge-agent/requirements.txt b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/src/edge-agent/requirements.txt new file mode 100644 index 000000000..96a27161c --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/src/edge-agent/requirements.txt @@ -0,0 +1 @@ +paho-mqtt==2.1.0 diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/tests/Invoke-E2ELabValidation.ps1 b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/tests/Invoke-E2ELabValidation.ps1 new file mode 100644 index 000000000..f004fef87 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/tests/Invoke-E2ELabValidation.ps1 @@ -0,0 +1,170 @@ +#Requires -Version 7.0 +<# +.SYNOPSIS +End-to-End Automated Validation Runner for Air-Gapped Agentic AI: Orchestrating Local SLMs and IoT on Edge Kubernetes. +.DESCRIPTION +Supports two validation modes: + 1. -Mode Offline (Default, <10 seconds, zero Azure cost): + - Compiles resources/infra/main.bicep via 'az bicep build' + - Validates PowerShell syntax on labautomation/*.ps1 and JSON schema on lab-defaults.json + - Validates all relative Markdown links across challenges and walkthroughs + - Runs the Python closed-loop simulation test suite (test-local-simulation.py) + 2. -Mode LiveAzure (Full unattended cloud deployment + Challenge 01-05 execution): + - Registers required Azure Resource Providers + - Deploys resources/infra/main.bicep (fast-k3s profile) into -ResourceGroupName + - Uses 'az vm run-command invoke' to execute Challenges 01-05 unattended on the K3s VM + - Optionally cleans up the test Resource Group when -CleanupOnSuccess is set +.EXAMPLE +./Invoke-E2ELabValidation.ps1 -Mode Offline +.EXAMPLE +./Invoke-E2ELabValidation.ps1 -Mode LiveAzure -SubscriptionId -ResourceGroupName rg-edgeai-e2e -Location eastus2 -CleanupOnSuccess +#> +[CmdletBinding()] +param( + [ValidateSet('Offline', 'LiveAzure')] + [string]$Mode = 'Offline', + + [string]$SubscriptionId, + + [string]$ResourceGroupName = 'rg-edgeai-e2e-validation', + + [string]$Location = 'swedencentral', + + [switch]$CleanupOnSuccess +) + +$ErrorActionPreference = 'Stop' +$microHackRoot = Split-Path -Parent (Split-Path -Parent $PSScriptRoot) + +Write-Host "==================================================================" -ForegroundColor Cyan +Write-Host " Air-Gapped Agentic Edge AI — Automated Validation ($Mode Mode)" -ForegroundColor Cyan +Write-Host "==================================================================" -ForegroundColor Cyan + +# ---------------------------------------------------------------------------- +# Phase 1: Static & Offline Simulation Checks (Always Run) +# ---------------------------------------------------------------------------- +Write-Host "`n[1/4] Compiling Bicep template (resources/infra/main.bicep)..." -ForegroundColor Yellow +$bicepFile = Join-Path $microHackRoot 'resources\infra\main.bicep' +az bicep build --file $bicepFile --stdout | Out-Null +if ($LASTEXITCODE -ne 0) { + throw "Bicep compilation failed for $bicepFile" +} +Write-Host " -> Bicep template compiled with 0 errors." -ForegroundColor Green + +Write-Host "`n[2/4] Validating PowerShell & JSON labautomation artifacts..." -ForegroundColor Yellow +$psFiles = @( + Join-Path $microHackRoot 'labautomation\deploy-lab.ps1' + Join-Path $microHackRoot 'labautomation\shared-deploy-lab.ps1' +) +foreach ($psFile in $psFiles) { + $parseErrors = $null + [System.Management.Automation.Language.Parser]::ParseFile($psFile, [ref]$null, [ref]$parseErrors) | Out-Null + if ($parseErrors.Count -gt 0) { + throw "PowerShell syntax error in ${psFile}: $($parseErrors | Out-String)" + } +} +$labDefaults = Get-Content (Join-Path $microHackRoot 'labautomation\lab-defaults.json') -Raw | ConvertFrom-Json +if ($labDefaults.deploymentType -ne 'resourcegroup-with-subscriptionowner') { + throw "Expected deploymentType 'resourcegroup-with-subscriptionowner' in lab-defaults.json" +} +Write-Host " -> PowerShell scripts & lab-defaults.json valid." -ForegroundColor Green + +Write-Host "`n[3/5] Checking all relative Markdown links & PR file-structure compliance..." -ForegroundColor Yellow +$mdFiles = Get-ChildItem -Path $microHackRoot -Recurse -Filter *.md +$brokenLinks = @() +foreach ($md in $mdFiles) { + $content = Get-Content $md.FullName -Raw + $matches = [regex]::Matches($content, '\[[^\]]+\]\(([^)#]+)(?:#[^)]*)?\)') + foreach ($m in $matches) { + $target = $m.Groups[1].Value.Trim() + if ($target -notmatch '^(https?|mailto):') { + $resolved = Join-Path $md.DirectoryName $target + if (-not (Test-Path $resolved)) { + $brokenLinks += "$($md.Name) -> $target" + } + } + } +} +if ($brokenLinks.Count -gt 0) { + throw "Broken Markdown links detected:`n$($brokenLinks -join "`n")" +} +$repoRoot = Resolve-Path (Join-Path $microHackRoot '..\..\..') +$complianceScript = Join-Path $repoRoot '.github\scripts\check_file_structure.py' +if (Test-Path $complianceScript) { + $env:PYTHONIOENCODING = 'utf-8' + $relFiles = Get-ChildItem -Path $microHackRoot -Recurse -File | ForEach-Object { + [System.IO.Path]::GetRelativePath($repoRoot.Path, $_.FullName).Replace('\', '/') + } + $relFiles | python $complianceScript | Out-Null + if ($LASTEXITCODE -ne 0) { + throw "Repository PR compliance check (check_file_structure.py) failed." + } + $resultsReport = Join-Path (Get-Location).Path 'structure-check-results.md' + if (Test-Path $resultsReport) { Remove-Item $resultsReport -Force } +} +Write-Host " -> All relative links across $($mdFiles.Count) Markdown files & PR structure compliance verified." -ForegroundColor Green + +Write-Host "`n[4/5] Running Closed-Loop Edge Agent + SAT Auth + SOP Simulation Suite..." -ForegroundColor Yellow +$pyTestScript = Join-Path $PSScriptRoot 'test-local-simulation.py' +python $pyTestScript +if ($LASTEXITCODE -ne 0) { + throw "Python closed-loop simulation test suite failed." +} +Write-Host " -> Closed-loop simulation tests passed!" -ForegroundColor Green + +if ($Mode -eq 'Offline') { + Write-Host "`nSUCCESS: All Offline & Closed-Loop Simulation checks passed!" -ForegroundColor Green + return +} + +# ---------------------------------------------------------------------------- +# Phase 2: Live Azure Unattended E2E Provisioning & Challenge Runner +# ---------------------------------------------------------------------------- +if ([string]::IsNullOrWhiteSpace($SubscriptionId)) { + $SubscriptionId = (az account show --query id -o tsv) +} +Write-Host "`n[LiveAzure 1/4] Setting subscription $SubscriptionId and creating RG $ResourceGroupName in $Location..." -ForegroundColor Yellow +az account set --subscription $SubscriptionId +az group create --name $ResourceGroupName --location $Location --output none + +$adminPassword = "EdgeAiLab!$(Get-Random -Minimum 100000 -Maximum 999999)aA" +Write-Host "[LiveAzure 2/4] Deploying resources/infra/main.bicep (fast-k3s profile, deployBastion=false for unattended speed)..." -ForegroundColor Yellow +$deployJson = az deployment group create ` + --resource-group $ResourceGroupName ` + --template-file $bicepFile ` + --parameters edgeProfile=fast-k3s deployBastion=false adminPasswordOrKey=$adminPassword ` + --query "properties.outputs" -o json | ConvertFrom-Json + +$vmName = $deployJson.edgeVmName.value +$arcClusterName = $deployJson.arcClusterSuggestedName.value +$storageName = $deployJson.schemaStorageAccountName.value +$uamiClientId = $deployJson.userAssignedIdentityClientId.value + +Write-Host " -> Provisioned VM: $vmName | Suggested Arc Cluster: $arcClusterName | Storage: $storageName" -ForegroundColor Green + +$clOid = az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv 2>$null + +Write-Host "[LiveAzure 3/4] Uploading manifests & executing unattended Challenge 01-05 verification inside $vmName..." -ForegroundColor Yellow +$e2eShellScript = (Get-Content (Join-Path $PSScriptRoot 'run-challenges-e2e.sh') -Raw) -replace "`r`n", "`n" +$lfTempScript = Join-Path $env:TEMP 'run-challenges-e2e-lf.sh' +[System.IO.File]::WriteAllText($lfTempScript, $e2eShellScript, [System.Text.UTF8Encoding]::new($false)) + +$runResult = az vm run-command invoke ` + --resource-group $ResourceGroupName ` + --name $vmName ` + --command-id RunShellScript ` + --scripts "@$lfTempScript" ` + --parameters "SUBSCRIPTION_ID=$SubscriptionId" "RESOURCE_GROUP=$ResourceGroupName" "ARC_LOCATION=westeurope" "CLUSTER_NAME=$arcClusterName" "SCHEMA_STORAGE_ACCOUNT=$storageName" "UAMI_CLIENT_ID=$uamiClientId" "CUSTOM_LOCATIONS_OID=$clOid" ` + --query "value[0].message" -o tsv + +Write-Host $runResult +if ($runResult -notmatch 'E2E_ALL_CHALLENGES_PASSED') { + throw "LiveAzure E2E challenge execution did not report E2E_ALL_CHALLENGES_PASSED." +} + +if ($CleanupOnSuccess) { + Write-Host "[LiveAzure 4/4] Cleaning up Resource Group $ResourceGroupName (--no-wait)..." -ForegroundColor Yellow + az group delete --name $ResourceGroupName --yes --no-wait +} + +Write-Host "`nSUCCESS: LiveAzure End-to-End MicroHack validation passed!" -ForegroundColor Green diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/tests/README.md b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/tests/README.md new file mode 100644 index 000000000..4c9f861bf --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/tests/README.md @@ -0,0 +1,35 @@ +# Automated Validation & Testing Suite + +This folder provides a two-tier automated test harness for **Air-Gapped Agentic AI: Orchestrating Local SLMs and IoT on Edge Kubernetes**: + +## 1. Fast Offline & Closed-Loop Simulation (`-Mode Offline`, < 10 seconds, $0 cost) + +Runs locally on Windows, macOS, Linux, or in CI without provisioning Azure resources: + +```powershell +pwsh ./resources/tests/Invoke-E2ELabValidation.ps1 -Mode Offline +``` + +What it validates automatically: +1. **Bicep Compilation:** Compiles [`../infra/main.bicep`](../infra/main.bicep) with `az bicep build` and verifies zero errors/warnings. +2. **Lab Automation Contracts:** Parses [`../../labautomation/deploy-lab.ps1`](../../labautomation/deploy-lab.ps1), [`../../labautomation/shared-deploy-lab.ps1`](../../labautomation/shared-deploy-lab.ps1), and [`../../labautomation/lab-defaults.json`](../../labautomation/lab-defaults.json). +3. **Markdown Link Integrity:** Verifies all relative links across all 15 Markdown files. +4. **Live Closed-Loop Edge Agent + SAT Auth + Foundry Local Simulation ([`test-local-simulation.py`](test-local-simulation.py)):** + - Validates all 8 Kubernetes YAML manifests and cross-checks SAT `ServiceAccount` names (`inference-client`), token audience (`foundry-local`), and `NetworkPolicy` port rules. + - Starts an in-process mock Foundry Local `/phi-4-mini/v1/chat/completions` server that enforces `Bearer ` authorization and grounds responses on [`SOP-CNC-042`](../manifests/03-edge-agent/local-sop-configmap.yaml). + - Feeds both a **Warning telemetry anomaly** (`96.4°C`, `9.8 mm/s`) and the **Challenge 5 Air-Gapped Critical Fault** (`114.6°C`, `15.2 mm/s`, `1.4 LPM`) through the actual [`agent.py`](../src/edge-agent/agent.py) code and asserts the exact PLC actuation payloads (`adjust_plc_setpoint` with `-25%` RPM and `trigger_safety_interlock` with `-60%` RPM / `SAFE_HOLD`). + +--- + +## 2. Unattended Live Azure E2E Test (`-Mode LiveAzure`) + +Deploys the `fast-k3s` Ubuntu 24.04 VM into an Azure subscription and uses `az vm run-command invoke` to execute [`run-challenges-e2e.sh`](run-challenges-e2e.sh) directly inside the K3s host without manual SSH/Bastion interaction: + +```powershell +pwsh ./resources/tests/Invoke-E2ELabValidation.ps1 ` + -Mode LiveAzure ` + -SubscriptionId "" ` + -ResourceGroupName "rg-edgeai-e2e-test" ` + -Location "eastus2" ` + -CleanupOnSuccess +``` diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/tests/run-challenges-e2e.sh b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/tests/run-challenges-e2e.sh new file mode 100644 index 000000000..031bd18c6 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/tests/run-challenges-e2e.sh @@ -0,0 +1,166 @@ +#!/usr/bin/env bash +# ============================================================================ +# Full Unattended End-to-End Challenge 01–05 Runner (Executed inside K3s VM) +# Invoked automatically by Invoke-E2ELabValidation.ps1 -Mode LiveAzure +# Exercises the complete stack: +# - Challenge 01: Kernel sysctl, K3s readiness, Azure Arc connect + Custom Locations +# - Challenge 02: Schema Registry, Device Registry Namespace, az iot ops init/create, +# BrokerListener, Anomaly Dataflow, Telemetry Simulator +# - Challenge 03: Gateway API CRDs, Istio, cert-manager update, Microsoft.Foundry +# operator, phi-4-mini CPU ModelDeployment & StoreModel cache +# - Challenge 04: SAT token verification, SOP ConfigMap, Autonomous Edge Agent +# - Challenge 05: Air-Gap NetworkPolicy, blocked cloud check, Critical Fault +# injection, and verification of spindleRpm=1680 interlock +# ============================================================================ +set -euo pipefail + +for arg in "$@"; do + export "$arg" +done + +export HOME=/root +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +export AZURE_EXTENSION_DIR=/usr/local/lib/azure-cli-extensions +export ARC_LOCATION="${ARC_LOCATION:-westeurope}" + +echo "[E2E] Waiting for cloud-init bootstrap (/var/log/edge-bootstrap-complete.log)..." +for i in $(seq 1 60); do + if [ -f /var/log/edge-bootstrap-complete.log ]; then + break + fi + sleep 10 +done + +echo "[E2E - Challenge 01] Verifying kernel sysctl parameters and K3s node readiness..." +sysctl fs.inotify.max_user_instances fs.inotify.max_user_watches fs.file-max +kubectl wait --for=condition=Ready node --all --timeout=180s + +echo "[E2E - Challenge 01] Signing in with User-Assigned Managed Identity and connecting K3s to Azure Arc in $ARC_LOCATION..." +az login --identity --client-id "$UAMI_CLIENT_ID" --output none +az account set --subscription "$SUBSCRIPTION_ID" + +az connectedk8s connect \ + --name "$CLUSTER_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --location "$ARC_LOCATION" \ + --enable-oidc-issuer \ + --enable-workload-identity \ + --disable-auto-upgrade + +if [ -n "${CUSTOM_LOCATIONS_OID:-}" ]; then + az connectedk8s enable-features \ + --name "$CLUSTER_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --custom-locations-oid "$CUSTOM_LOCATIONS_OID" \ + --features cluster-connect custom-locations +fi + +echo "[E2E - Challenge 02] Creating Schema Registry, Device Registry Namespace, and deploying Azure IoT Operations..." +STORAGE_ACCOUNT_ID=$(az storage account show --name "$SCHEMA_STORAGE_ACCOUNT" --resource-group "$RESOURCE_GROUP" --query id -o tsv) +SCHEMA_REGISTRY_NAME="sr-${CLUSTER_NAME}" +SCHEMA_REGISTRY_NAMESPACE="srns-${CLUSTER_NAME}" +ADR_NAMESPACE_NAME="adrns-${CLUSTER_NAME}" +AIO_INSTANCE_NAME="aio-${CLUSTER_NAME}" +CUSTOM_LOCATION_NAME="cl-${CLUSTER_NAME}" + +az iot ops schema registry create \ + --name "$SCHEMA_REGISTRY_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --location "$ARC_LOCATION" \ + --registry-namespace "$SCHEMA_REGISTRY_NAMESPACE" \ + --sa-resource-id "$STORAGE_ACCOUNT_ID" + +az iot ops ns create \ + --name "$ADR_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --location "$ARC_LOCATION" + +az iot ops init \ + --cluster "$CLUSTER_NAME" \ + --resource-group "$RESOURCE_GROUP" + +SCHEMA_REGISTRY_ID=$(az iot ops schema registry show --name "$SCHEMA_REGISTRY_NAME" --resource-group "$RESOURCE_GROUP" --query id -o tsv) +ADR_NAMESPACE_ID=$(az iot ops ns show --name "$ADR_NAMESPACE_NAME" --resource-group "$RESOURCE_GROUP" --query id -o tsv) + +az iot ops create \ + --cluster "$CLUSTER_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --name "$AIO_INSTANCE_NAME" \ + --location "$ARC_LOCATION" \ + --custom-location "$CUSTOM_LOCATION_NAME" \ + --sr-resource-id "$SCHEMA_REGISTRY_ID" \ + --ns-resource-id "$ADR_NAMESPACE_ID" \ + --broker-frontend-replicas 1 \ + --broker-frontend-workers 1 \ + --broker-backend-part 1 \ + --broker-backend-workers 1 \ + --broker-backend-rf 2 \ + --broker-mem-profile Low + +kubectl apply -f /opt/edge-ai-lab/manifests/01-iot-sim/aio-anomaly-dataflow.yaml +kubectl apply -f /opt/edge-ai-lab/manifests/01-iot-sim/telemetry-simulator.yaml +kubectl rollout status deployment/manufacturing-telemetry-simulator -n edge-agents --timeout=120s + +echo "[E2E - Challenge 03] Installing Gateway API, Istio, updating cert-manager, and deploying Foundry Local..." +kubectl apply --server-side --force-conflicts -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml +kubectl apply --server-side --force-conflicts -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.5.0/manifests.yaml + +helm repo add istio https://istio-release.storage.googleapis.com/charts +helm repo update +helm upgrade --install istio-base istio/base -n istio-system --create-namespace --wait +helm upgrade --install istiod istio/istiod -n istio-system --set pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true --wait + +az k8s-extension update \ + --cluster-name "$CLUSTER_NAME" \ + --name "cert-manager" \ + --resource-group "$RESOURCE_GROUP" \ + --cluster-type connectedClusters \ + --config config.enableGatewayAPI=true \ + --config cert-manager.crds.keep=true \ + --config trust-manager.defaultPackage.enabled=false \ + --config trust-manager.secretTargets.enabled=true \ + --config trust-manager.secretTargets.authorizedSecretsAll=true \ + --yes + +az k8s-extension create \ + --resource-group "$RESOURCE_GROUP" \ + --cluster-name "$CLUSTER_NAME" \ + --name "inference-operator" \ + --extension-type Microsoft.Foundry \ + --scope cluster \ + --release-namespace "foundry-local-operator" \ + --cluster-type connectedClusters \ + --release-train stable \ + --config entraAuth.enabled=false \ + --config satAuth.enabled=true \ + --config satAuth.audience="foundry-local" \ + --config storeModel.cacheJob.resources.requests.memory="4Gi" \ + --config storeModel.cacheJob.resources.limits.memory="12Gi" + +kubectl wait --for=condition=complete job/catalog-sync-seed -n foundry-local-operator --timeout=120s +kubectl apply -f /opt/edge-ai-lab/manifests/02-foundry-local/caller-service-account.yaml +kubectl apply -f /opt/edge-ai-lab/manifests/02-foundry-local/model-deployment-cpu.yaml + +for i in $(seq 1 60); do + READY=$(kubectl get modeldeployment phi-4-mini -n foundry-local-operator -o jsonpath='{.status.deploymentReady}' 2>/dev/null || echo "false") + if [ "$READY" = "true" ]; then + break + fi + sleep 15 +done + +echo "[E2E - Challenge 04] Deploying Local SOP ConfigMap and Autonomous Edge Agent..." +kubectl apply -f /opt/edge-ai-lab/manifests/03-edge-agent/local-sop-configmap.yaml +kubectl apply -f /opt/edge-ai-lab/manifests/03-edge-agent/edge-agent-deployment.yaml +kubectl rollout status deployment/autonomous-edge-agent -n edge-agents --timeout=120s + +echo "[E2E - Challenge 05] Enforcing Air-Gap NetworkPolicy and injecting Critical Fault..." +kubectl apply -f /opt/edge-ai-lab/manifests/04-airgap-policy/deny-external-egress.yaml +kubectl delete job inject-critical-bearing-fault -n edge-agents --ignore-not-found +kubectl apply -f /opt/edge-ai-lab/manifests/04-airgap-policy/inject-critical-fault.yaml +kubectl wait --for=condition=complete job/inject-critical-bearing-fault -n edge-agents --timeout=60s +sleep 25 + +kubectl logs -n edge-agents deploy/autonomous-edge-agent --tail=20 +kubectl logs -n edge-agents deploy/manufacturing-telemetry-simulator --tail=15 | grep "spindleRpm=1680" +echo "E2E_ALL_CHALLENGES_PASSED" diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/tests/test-local-simulation.py b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/tests/test-local-simulation.py new file mode 100644 index 000000000..afc5b44fc --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/resources/tests/test-local-simulation.py @@ -0,0 +1,253 @@ +#!/usr/bin/env python3 +""" +Automated Offline & Closed-Loop Simulation Test Suite +for Air-Gapped Agentic AI: Orchestrating Local SLMs and IoT on Edge Kubernetes. + +What this tests (in <10 seconds, zero Azure cost): + 1. Kubernetes YAML manifest integrity across all 4 challenge manifest folders + (Namespace, ServiceAccount, BrokerListener, Dataflow, ModelDeployment, + ConfigMap, Deployment, NetworkPolicy, Job). + 2. Contract alignment between manifests: + - ServiceAccount 'inference-client' in 'edge-agents' matches ModelDeployment SAT authorizedSubjects + - Projected SAT token audience ('foundry-local') and mount path ('/var/run/secrets/foundry-local/token') + - Internal MQTT broker DNS ('aio-broker-internal.azure-iot-operations.svc.cluster.local:1883') + - NetworkPolicy 'enforce-airgap-sovereign-mode' blocks external internet while allowing + kube-system DNS (53), azure-iot-operations MQTT (1883/18883), and foundry-local-operator/istio-system (80/443/8080). + 3. Live Closed-Loop Simulation of the Edge Agent + Foundry Local HTTPS Gateway + PLC Simulator: + - Spins up a local mock of Foundry Local (/phi-4-mini/v1/chat/completions) that validates + the Kubernetes SAT Bearer token and evaluates SOP-CNC-042. + - Feeds both a Warning telemetry anomaly (96.4C, 9.8mm/s) and the Challenge 5 Critical fault + payload (114.6C, 15.2mm/s, 1.4 LPM) through the actual Edge Agent code (resources/src/edge-agent/agent.py). + - Verifies that the agent issues 'adjust_plc_setpoint' (-25% RPM) for warnings and + 'trigger_safety_interlock' (-60% RPM, SAFE_HOLD, P1-CRITICAL-INTERLOCK) for critical faults. +""" +from __future__ import annotations + +import http.server +import importlib.util +import json +import os +import re +import sys +import tempfile +import threading +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +MANIFESTS_DIR = ROOT / "resources" / "manifests" +AGENT_SRC = ROOT / "resources" / "src" / "edge-agent" / "agent.py" + + +def parse_simple_yaml_docs(file_path: Path) -> list[str]: + text = file_path.read_text(encoding="utf-8") + docs = [d.strip() for d in re.split(r"(?m)^---\s*$", text) if d.strip()] + return docs + + +class MockFoundryLocalHandler(http.server.BaseHTTPRequestHandler): + expected_token = "eyJhbGciOiJSUzI1NiIsImtpZCI6Imthc3MifQ.simulated-k8s-sat-token" + + def log_message(self, format: str, *args) -> None: + pass + + def do_POST(self) -> None: + auth_header = self.headers.get("Authorization", "") + if auth_header != f"Bearer {self.expected_token}": + self.send_response(401) + self.end_headers() + self.wfile.write(b'{"error":"Unauthorized SAT token"}') + return + + length = int(self.headers.get("Content-Length", "0")) + body = json.loads(self.rfile.read(length).decode("utf-8")) + user_msg = body["messages"][-1]["content"] + + match = re.search(r"\{.*\}", user_msg) + telemetry = json.loads(match.group(0)) if match else {} + + temp = telemetry.get("bearingTempC", 0.0) + vib = telemetry.get("vibrationMmPerSec", 0.0) + coolant = telemetry.get("coolantFlowLpm", 10.0) + + if temp >= 105.0 or vib >= 13.0 or coolant < 2.0: + tool_resp = { + "tool_name": "trigger_safety_interlock", + "arguments": { + "spindle_rpm_delta_pct": -60, + "enable_coolant_boost": True, + "interlock_mode": "SAFE_HOLD", + "priority": "P1-CRITICAL-INTERLOCK", + }, + "reasoning": f"Critical threshold breached (temp={temp}C, vib={vib}mm/s). Triggering SOP-CNC-042 interlock.", + } + else: + tool_resp = { + "tool_name": "adjust_plc_setpoint", + "arguments": { + "spindle_rpm_delta_pct": -25, + "enable_coolant_boost": True, + "interlock_mode": "CLOSED_LOOP_THROTTLE", + "priority": "P2-WARNING", + }, + "reasoning": f"Warning threshold breached (temp={temp}C, vib={vib}mm/s). Throttling RPM by 25%.", + } + + completion = { + "id": "chatcmpl-edge-001", + "object": "chat.completion", + "model": body.get("model", "Phi-4-mini-instruct"), + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": json.dumps(tool_resp), + }, + "finish_reason": "stop", + } + ], + } + payload = json.dumps(completion).encode("utf-8") + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(payload))) + self.end_headers() + self.wfile.write(payload) + + +class FakeMqttClient: + def __init__(self) -> None: + self.published: list[tuple[str, dict, int]] = [] + + def publish(self, topic: str, payload: str, qos: int = 0) -> None: + self.published.append((topic, json.loads(payload), qos)) + + +class AirGappedEdgeMicroHackTests(unittest.TestCase): + def test_01_all_manifests_exist_and_have_valid_kinds(self) -> None: + expected_files = [ + MANIFESTS_DIR / "01-iot-sim" / "telemetry-simulator.yaml", + MANIFESTS_DIR / "01-iot-sim" / "aio-anomaly-dataflow.yaml", + MANIFESTS_DIR / "02-foundry-local" / "caller-service-account.yaml", + MANIFESTS_DIR / "02-foundry-local" / "model-deployment-cpu.yaml", + MANIFESTS_DIR / "03-edge-agent" / "local-sop-configmap.yaml", + MANIFESTS_DIR / "03-edge-agent" / "edge-agent-deployment.yaml", + MANIFESTS_DIR / "04-airgap-policy" / "deny-external-egress.yaml", + MANIFESTS_DIR / "04-airgap-policy" / "inject-critical-fault.yaml", + ] + for f in expected_files: + self.assertTrue(f.exists(), f"Missing manifest file: {f}") + docs = parse_simple_yaml_docs(f) + self.assertGreaterEqual(len(docs), 1, f"Empty YAML manifest: {f}") + for doc in docs: + self.assertIn("apiVersion:", doc, f"Missing apiVersion in {f}") + self.assertIn("kind:", doc, f"Missing kind in {f}") + self.assertIn("metadata:", doc, f"Missing metadata in {f}") + + def test_02_contract_alignment_across_manifests(self) -> None: + sa_yaml = (MANIFESTS_DIR / "02-foundry-local" / "caller-service-account.yaml").read_text(encoding="utf-8") + md_yaml = (MANIFESTS_DIR / "02-foundry-local" / "model-deployment-cpu.yaml").read_text(encoding="utf-8") + agent_yaml = (MANIFESTS_DIR / "03-edge-agent" / "edge-agent-deployment.yaml").read_text(encoding="utf-8") + netpol_yaml = (MANIFESTS_DIR / "04-airgap-policy" / "deny-external-egress.yaml").read_text(encoding="utf-8") + + # ServiceAccount & SAT binding + self.assertIn("name: inference-client", sa_yaml) + self.assertIn("namespace: edge-agents", sa_yaml) + self.assertIn("serviceAccountName: inference-client", md_yaml) + self.assertIn("serviceAccountNamespace: edge-agents", md_yaml) + self.assertIn("role: GeneralInferenceUser", md_yaml) + self.assertIn("compute: cpu", md_yaml) + self.assertIn("runtime: onnx-genai", md_yaml) + + # Projected SAT token volume in Edge Agent deployment + self.assertIn("serviceAccountName: inference-client", agent_yaml) + self.assertIn("audience: foundry-local", agent_yaml) + self.assertIn("/var/run/secrets/foundry-local", agent_yaml) + + # Air-gap NetworkPolicy allows only DNS (53), AIO MQTT (1883/18883), and Foundry Local (80/443/8080) + self.assertIn("kind: NetworkPolicy", netpol_yaml) + self.assertIn("azure-iot-operations", netpol_yaml) + self.assertIn("foundry-local-operator", netpol_yaml) + self.assertIn("istio-system", netpol_yaml) + + def test_03_closed_loop_agent_with_mock_foundry_local_and_sat_auth(self) -> None: + with tempfile.TemporaryDirectory() as tmpdir: + tmp_path = Path(tmpdir) + sop_file = tmp_path / "sop.md" + sat_file = tmp_path / "token" + + sop_yaml = (MANIFESTS_DIR / "03-edge-agent" / "local-sop-configmap.yaml").read_text(encoding="utf-8") + sop_content = sop_yaml.split("sop.md: |", 1)[1] + sop_file.write_text(sop_content, encoding="utf-8") + sat_file.write_text(MockFoundryLocalHandler.expected_token, encoding="utf-8") + + server = http.server.HTTPServer(("127.0.0.1", 0), MockFoundryLocalHandler) + port = server.server_port + server_thread = threading.Thread(target=server.serve_forever, daemon=True) + server_thread.start() + + try: + os.environ["SOP_PATH"] = str(sop_file) + os.environ["SAT_TOKEN_PATH"] = str(sat_file) + os.environ["FOUNDRY_ENDPOINT"] = f"http://127.0.0.1:{port}/phi-4-mini/v1/chat/completions" + + if "paho" not in sys.modules: + import types + paho_mod = types.ModuleType("paho") + mqtt_mod = types.ModuleType("paho.mqtt") + client_mod = types.ModuleType("paho.mqtt.client") + class DummyClient: + pass + class DummyEnum: + VERSION2 = 2 + client_mod.Client = DummyClient + client_mod.CallbackAPIVersion = DummyEnum + sys.modules["paho"] = paho_mod + sys.modules["paho.mqtt"] = mqtt_mod + sys.modules["paho.mqtt.client"] = client_mod + + spec = importlib.util.spec_from_file_location("edge_agent_module", AGENT_SRC) + agent_mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(agent_mod) + + fake_mqtt = FakeMqttClient() + + # Case A: Warning anomaly (96.4C, 9.8mm/s) -> should trigger adjust_plc_setpoint (-25% RPM) + warning_telemetry = { + "machineId": "CNC-EXTRUDER-01", + "bearingTempC": 96.4, + "vibrationMmPerSec": 9.8, + "coolantFlowLpm": 6.2, + "status": "WARNING_THERMAL_VIBRATION_DRIFT", + } + agent_mod.evaluate_anomaly(fake_mqtt, warning_telemetry) + self.assertEqual(len(fake_mqtt.published), 1) + topic, cmd_warn, qos = fake_mqtt.published[0] + self.assertEqual(topic, "factory/line1/commands") + self.assertEqual(cmd_warn["decision"]["tool_name"], "adjust_plc_setpoint") + self.assertEqual(cmd_warn["decision"]["arguments"]["spindle_rpm_delta_pct"], -25) + + # Case B: Challenge 5 Air-Gapped Critical Fault (114.6C, 15.2mm/s, 1.4 LPM) -> trigger_safety_interlock (-60% RPM) + critical_telemetry = { + "machineId": "CNC-EXTRUDER-01", + "bearingTempC": 114.6, + "vibrationMmPerSec": 15.2, + "coolantFlowLpm": 1.4, + "status": "CRITICAL_THERMAL_RUNAWAY_BEARING_SEIZURE", + } + agent_mod.evaluate_anomaly(fake_mqtt, critical_telemetry) + self.assertEqual(len(fake_mqtt.published), 2) + topic, cmd_crit, qos = fake_mqtt.published[1] + self.assertEqual(topic, "factory/line1/commands") + self.assertEqual(cmd_crit["decision"]["tool_name"], "trigger_safety_interlock") + self.assertEqual(cmd_crit["decision"]["arguments"]["spindle_rpm_delta_pct"], -60) + self.assertEqual(cmd_crit["decision"]["arguments"]["interlock_mode"], "SAFE_HOLD") + self.assertEqual(cmd_crit["decision"]["arguments"]["priority"], "P1-CRITICAL-INTERLOCK") + finally: + server.shutdown() + server.server_close() + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-01/solution-01.md b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-01/solution-01.md new file mode 100644 index 000000000..5a02323e5 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-01/solution-01.md @@ -0,0 +1,174 @@ +# Solution 01 — Provision & Verify the Sovereign Edge Kubernetes Foundation + +**[Home](../../Readme.md)** - [Back to Challenge 01](../../challenges/challenge-01.md) - [Next Solution →](../challenge-02/solution-02.md) + +**Duration:** 30 minutes + +## Step 1: Connect to the Edge Host VM + +1. Open the [Azure Portal](https://portal.azure.com) and navigate to your lab Resource Group (`ResourceGroupName` from your lab credentials). +2. Select your **Edge Host VM** (`EdgeHostVmName`, default `edgeai-host-...`). +3. Select **Connect** > **Bastion**, enter `EdgeHostAdminUsername` (`edgeadmin`) and `EdgeHostAdminPassword`, and open a browser terminal session. + > **Where do I find `EdgeHostAdminUsername` and `EdgeHostAdminPassword`?** + > - **Hackbox / Workshop Portal:** Copy `EdgeHostAdminUsername` (`edgeadmin`) and `EdgeHostAdminPassword` (and the direct `AzurePortalEdgeVmUrl` link) from your **Lab Credentials** tab. + > - **Azure Portal Key Vault:** In your lab Resource Group, open the Key Vault (`edgeai-kv-...`) -> **Objects** > **Secrets**, where `EdgeHostAdminUsername` and `EdgeHostAdminPassword` are stored automatically by `main.bicep`. + > - **No-Password Alternative:** You can also select **Operations** > **Run command** > **RunShellScript** directly on the VM blade in the Azure Portal, or reset the password anytime via **Help** > **Reset password**. + > - **Manual / Self-Paced Deployment:** If you are running in your own subscription without pre-provisioned lab automation, deploy [`resources/infra/main.bicep`](../../resources/infra/main.bicep) with your own password: + > ```bash + > az deployment group create \ + > --resource-group \ + > --template-file resources/infra/main.bicep \ + > --parameters edgeProfile=fast-k3s adminPasswordOrKey='' + > ``` + +![Connecting to the Ubuntu 24.04 Edge Host VM via Azure Bastion](../../img/bastion-edge-host-login.png) + +--- + +## Step 2: Verify Kernel Limits, K3s, and CLI Tooling + +Inside the Edge Host VM terminal, verify that `cloud-init` has completed: + +```bash +cat /var/log/edge-bootstrap-complete.log +``` + +Verify the Linux kernel `inotify` and file-descriptor limits required by **Azure IoT Operations** and **Foundry Local**: + +```bash +sysctl fs.inotify.max_user_instances fs.inotify.max_user_watches fs.file-max +``` + +Expected output: + +```text +fs.inotify.max_user_instances = 8192 +fs.inotify.max_user_watches = 524288 +fs.file-max = 100000 +``` + +Verify that your single-node K3s cluster is `Ready` and clone the MicroHack repository onto the Edge Host if not already present: + +```bash +export KUBECONFIG=~/.kube/config +kubectl get nodes -o wide +helm version --short +az version +``` + +Clone the repository and navigate to this MicroHack folder: + +```bash +git clone https://github.com/microsoft/MicroHack.git ~/MicroHack +cd ~/MicroHack/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI +``` + +--- + +## Step 3: Sign In to Azure & Set Environment Variables + +Sign in to Azure on the Edge Host VM. You can sign in either with your workshop user account (`az login --use-device-code`) or directly via the Edge Host VM's User-Assigned Managed Identity (which is pre-authorized with `Contributor` and `Role Based Access Control Administrator` on your lab Resource Group): + +```bash +# Option A: Sign in with your workshop user account +az login --use-device-code + +# Option B (Fast Unattended Alternative): Sign in with the Edge Host VM's Managed Identity +# UAMI_CLIENT_ID=$(curl -s -H Metadata:true "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/" | jq -r .client_id) +# az login --identity --client-id "$UAMI_CLIENT_ID" +``` + +Next, create and source `~/edge-env.sh`. This block automatically queries the Azure Instance Metadata Service (`169.254.169.254`) on the Edge Host VM to discover your `SUBSCRIPTION_ID`, `RESOURCE_GROUP`, and `VM_LOCATION`—**no manual placeholder editing required**: + +```bash +cat << 'EOF' > ~/edge-env.sh +export SUBSCRIPTION_ID=$(curl -s -H Metadata:true "http://169.254.169.254/metadata/instance/compute/subscriptionId?api-version=2021-02-01&format=text") +export RESOURCE_GROUP=$(curl -s -H Metadata:true "http://169.254.169.254/metadata/instance/compute/resourceGroupName?api-version=2021-02-01&format=text") +export VM_LOCATION=$(curl -s -H Metadata:true "http://169.254.169.254/metadata/instance/compute/location?api-version=2021-02-01&format=text") +export ARC_LOCATION="westeurope" +export CLUSTER_NAME="edgeai-arck8s-$(echo -n "$RESOURCE_GROUP" | md5sum | cut -c1-6)" +export SCHEMA_STORAGE_ACCOUNT=$(az storage account list -g "$RESOURCE_GROUP" --query "[0].name" -o tsv) +export KUBECONFIG=~/.kube/config +EOF + +source ~/edge-env.sh +az account set --subscription "$SUBSCRIPTION_ID" +echo "Subscription: $SUBSCRIPTION_ID | RG: $RESOURCE_GROUP | VM Location: $VM_LOCATION | Arc/AIO Location: $ARC_LOCATION | Cluster: $CLUSTER_NAME | Storage: $SCHEMA_STORAGE_ACCOUNT" +``` + +> **Why is `ARC_LOCATION` set to `westeurope`?** +> Even when your Edge Host VM runs in `swedencentral` (for `Standard_D16s_v5` SKU availability), **Azure Device Registry** and **Azure IoT Operations** require an officially supported AIO control-plane region (`westeurope`, `northeurope`, `germanywestcentral`, `uksouth`, `eastus`, `eastus2`, `westus2`, `westus3`, `centralus`). Setting `ARC_LOCATION="westeurope"` ensures your `ConnectedCluster`, `CustomLocation`, `SchemaRegistry`, `DeviceRegistry Namespace`, and `IoTOperations` instance all share the same supported region. + +If resource providers have not yet been registered in your subscription, run the helper script: + +```bash +bash resources/infra/register-providers.sh +``` + +--- + +## Step 4: Connect the Kubernetes Cluster to Azure Arc (~4 minutes) + +Connect your K3s cluster to **Azure Arc** in `$ARC_LOCATION` with **OIDC Issuer** and **Workload Identity** enabled (required by Azure IoT Operations): + +```bash +az connectedk8s connect \ + --name "$CLUSTER_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --location "$ARC_LOCATION" \ + --subscription "$SUBSCRIPTION_ID" \ + --enable-oidc-issuer \ + --enable-workload-identity \ + --disable-auto-upgrade +``` + +Enable **Custom Locations** on the cluster using the Microsoft Entra Object ID of the `Microsoft.ExtendedLocation` service principal (or use `CustomLocationsRpObjectId` from your lab credentials dashboard if your account does not have Microsoft Graph directory-read permissions): + +```bash +CUSTOM_LOCATION_RP_OID=$(az ad sp show --id bc313c14-388c-4e7d-a58e-70017303ee3b --query id -o tsv 2>/dev/null || echo "") + +az connectedk8s enable-features \ + --name "$CLUSTER_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --custom-locations-oid "$CUSTOM_LOCATION_RP_OID" \ + --features cluster-connect custom-locations +``` + +--- + +## Step 5: Verify Azure Arc Connectivity + +Confirm that the cluster shows `Connected` in Azure and all `azure-arc` pods are `Running`: + +```bash +az connectedk8s show \ + --name "$CLUSTER_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --query "{Name:name, Status:connectivityStatus, K8sVersion:kubernetesVersion, OidcEnabled:oidcIssuerProfile.enabled}" \ + --output table + +kubectl get pods -n azure-arc +``` + +![Challenge 01 — Azure Portal Resource Group Overview](../../img/portal-resource-group-overview.png) + +![Challenge 01 — Azure Portal Arc-enabled Kubernetes Cluster Connected (K3s v1.33.13+k3s2)](../../img/portal-arc-cluster-connected.png) + +![Challenge 01 — CLI & Arc-enabled Kubernetes Verification](../../img/challenge-01-portal-and-arc.png) + +**Common stumbles:** `Unable to fetch the Custom Location OID` warning during `az connectedk8s enable-features` (harmless when `--custom-locations-oid` is explicitly supplied) · `kubectl` permission denied (ensure `export KUBECONFIG=~/.kube/config` is set). + +--- + +## Grounding Questions — Suggested Answers + +**1. Why must `fs.inotify.max_user_watches` and `fs.inotify.max_user_instances` be raised on the Linux host before deploying Azure IoT Operations and Foundry Local?** +Kubernetes operators, Istio sidecars, OpenTelemetry collectors, and Azure IoT Operations broker pods all watch ConfigMaps, Secrets, and projected SAT token files via Linux `inotify`. Default Ubuntu limits (`128` instances / `8192` watches) are quickly exhausted by multi-extension clusters, causing `too many open files` errors in kubelet and ConfigMap reloaders. + +**2. What is the role of an Azure Arc Custom Location (`Microsoft.ExtendedLocation/customLocations`), and how does it allow Azure Resource Manager (ARM) to deploy resources onto an on-premises Kubernetes cluster?** +A Custom Location maps an Azure Resource Manager target scope onto a specific Kubernetes namespace and set of Arc cluster extensions. When you create an Azure resource targeting that Custom Location, the `resource-sync-agent` pod inside `azure-arc` translates the ARM resource into Kubernetes Custom Resources (CRs) on the local cluster. + +**3. Why can the Edge Host VM reside in `swedencentral` while the Azure Arc `connectedCluster` resource is registered in `westeurope`?** +An Azure Arc `connectedCluster` is a metadata projection of an edge Kubernetes cluster that could physically sit in any factory worldwide. Its Azure region only determines which regional Azure Arc and Azure IoT Operations control-plane endpoints manage its ARM metadata. + +[Back to Challenge 01](../../challenges/challenge-01.md) | [Next Solution →](../challenge-02/solution-02.md) diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-02/solution-02.md b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-02/solution-02.md new file mode 100644 index 000000000..bb8ab1958 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-02/solution-02.md @@ -0,0 +1,166 @@ +# Solution 02 — Deploy Azure IoT Operations & Simulated Manufacturing Telemetry + +**[Home](../../Readme.md)** - [← Previous Solution](../challenge-01/solution-01.md) - [Back to Challenge 02](../../challenges/challenge-02.md) - [Next Solution →](../challenge-03/solution-03.md) + +**Duration:** 45 minutes + +## Step 1: Create the Schema Registry, Device Registry Namespace & Initialize AIO + +Source your environment variables from Challenge 01: + +```bash +source ~/edge-env.sh +cd ~/MicroHack/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI +``` + +Get the Resource ID of your HNS-enabled Storage Account, create the **Azure IoT Operations Schema Registry**, and create the **Azure Device Registry Namespace**: + +```bash +export STORAGE_ACCOUNT_ID=$(az storage account show \ + --name "$SCHEMA_STORAGE_ACCOUNT" \ + --resource-group "$RESOURCE_GROUP" \ + --query id -o tsv) + +export SCHEMA_REGISTRY_NAME="sr-${CLUSTER_NAME}" +export SCHEMA_REGISTRY_NAMESPACE="srns-${CLUSTER_NAME}" +export ADR_NAMESPACE_NAME="adrns-${CLUSTER_NAME}" + +az iot ops schema registry create \ + --name "$SCHEMA_REGISTRY_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --location "$ARC_LOCATION" \ + --registry-namespace "$SCHEMA_REGISTRY_NAMESPACE" \ + --sa-resource-id "$STORAGE_ACCOUNT_ID" + +az iot ops ns create \ + --name "$ADR_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --location "$ARC_LOCATION" +``` + +Initialize the cluster for **Azure IoT Operations** (this installs the foundational `cert-manager` and `azure-secret-store` Arc extensions): + +```bash +az iot ops init \ + --cluster "$CLUSTER_NAME" \ + --resource-group "$RESOURCE_GROUP" +``` + +--- + +## Step 2: Deploy the Azure IoT Operations Instance + +Create the Azure IoT Operations instance on your Arc-enabled Kubernetes cluster: + +```bash +export AIO_INSTANCE_NAME="aio-${CLUSTER_NAME}" +export CUSTOM_LOCATION_NAME="cl-${CLUSTER_NAME}" + +export SCHEMA_REGISTRY_ID=$(az iot ops schema registry show \ + --name "$SCHEMA_REGISTRY_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --query id -o tsv) + +export ADR_NAMESPACE_ID=$(az iot ops ns show \ + --name "$ADR_NAMESPACE_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --query id -o tsv) + +az iot ops create \ + --cluster "$CLUSTER_NAME" \ + --resource-group "$RESOURCE_GROUP" \ + --name "$AIO_INSTANCE_NAME" \ + --location "$ARC_LOCATION" \ + --custom-location "$CUSTOM_LOCATION_NAME" \ + --sr-resource-id "$SCHEMA_REGISTRY_ID" \ + --ns-resource-id "$ADR_NAMESPACE_ID" \ + --broker-frontend-replicas 1 \ + --broker-frontend-workers 1 \ + --broker-backend-part 1 \ + --broker-backend-workers 1 \ + --broker-backend-rf 2 \ + --broker-mem-profile Low +``` + +Verify that the Azure IoT Operations pods are running in the `azure-iot-operations` namespace: + +```bash +kubectl get pods -n azure-iot-operations +``` + +--- + +## Step 3: Configure the Internal MQTT BrokerListener and Anomaly Dataflow + +Apply [`resources/manifests/01-iot-sim/aio-anomaly-dataflow.yaml`](../../resources/manifests/01-iot-sim/aio-anomaly-dataflow.yaml) to expose the internal cluster MQTT listener (`aio-broker-internal.azure-iot-operations.svc.cluster.local:1883`) and create the `cnc-anomaly-router` Dataflow: + +```bash +kubectl apply -f resources/manifests/01-iot-sim/aio-anomaly-dataflow.yaml +``` + +Verify the `BrokerListener` service and `Dataflow`: + +```bash +kubectl get brokerlistener,dataflow -n azure-iot-operations +kubectl get svc aio-broker-internal -n azure-iot-operations +``` + +--- + +## Step 4: Deploy the Manufacturing Telemetry Simulator + +Apply [`resources/manifests/01-iot-sim/telemetry-simulator.yaml`](../../resources/manifests/01-iot-sim/telemetry-simulator.yaml) to create the `edge-agents` namespace and start publishing CNC-Extruder telemetry every 5 seconds (with an anomaly injected every 6th tick): + +```bash +kubectl apply -f resources/manifests/01-iot-sim/telemetry-simulator.yaml +kubectl rollout status deployment/manufacturing-telemetry-simulator -n edge-agents +``` + +Inspect the live simulator logs: + +```bash +kubectl logs -n edge-agents deploy/manufacturing-telemetry-simulator --tail=20 -f +``` + +You will see output similar to: + +```text +[SIMULATOR] Connected to AIO MQTT Broker at aio-broker-internal.azure-iot-operations.svc.cluster.local:1883 (rc=Success) +[SIMULATOR] >>> Published factory/line1/telemetry (tick=1, status=NOMINAL, temp=68.4C, vib=2.41mm/s) +... +[SIMULATOR] >>> Published factory/line1/telemetry (tick=6, status=WARNING_THERMAL_VIBRATION_DRIFT, temp=96.3C, vib=10.12mm/s) +``` + +--- + +## Step 5: Verify Anomaly Filtering on `factory/line1/anomalies` + +> **Important Technical Note:** In Azure IoT Operations Dataflows, `builtInTransformationSettings.filter` acts as a **drop filter**—when `expression` evaluates to `true`, the message is dropped. Therefore, `aio-anomaly-dataflow.yaml` uses `expression: "$1 <= 85.0 && $2 <= 7.5"` to drop normal messages and pass only threshold breaches (`bearingTempC > 85.0` or `vibrationMmPerSec > 7.5`) to `factory/line1/anomalies`. + +Verify that `cnc-anomaly-router` routes only anomalous events to `factory/line1/anomalies` using the `mosquitto_sub` client pre-installed on the Edge Host (connecting via `nodePort: 31883`): + +```bash +mosquitto_sub -h 127.0.0.1 -p 31883 -t "factory/line1/anomalies" -C 1 -v +``` + +Once a single anomaly message (`status: WARNING_THERMAL_VIBRATION_DRIFT`) is printed and `mosquitto_sub` exits, proceed to Challenge 03! + +![Challenge 02 — Azure Portal Azure IoT Operations Instance Overview](../../img/portal-aio-instance-overview.png) + +![Challenge 02 — Live Azure Bastion Terminal: Broker, Dataflow & Anomaly Stream](../../img/bastion-challenge-02-aio-anomaly.png) + +![Challenge 02 — Azure IoT Operations & Anomaly Dataflow Filtering](../../img/challenge-02-aio-and-dataflow.png) + +**Common stumbles:** `LocationNotAvailableForResourceType` when creating Schema Registry (ensure `--location "$ARC_LOCATION"` is set to `westeurope`) · `HostResourceRegionMismatch` during `az iot ops create` (the `connectedCluster` in Challenge 01 and `az iot ops create` in Challenge 02 must both use `westeurope`). + +--- + +## Grounding Questions — Suggested Answers + +**1. Why do we filter normal sensor ticks inside an Azure IoT Operations Dataflow rather than forwarding every 5-second MQTT message to the Edge AI Agent?** +Small Language Models running on CPU take several seconds per completion and consume significant CPU/memory bandwidth. Using the high-throughput Rust-based AIO Dataflow engine to drop normal readings at the broker ensures the SLM is only invoked when an actual threshold drift or fault occurs. + +**2. Why did `aio-broker-webhook` reject creating a second `BrokerListener` with `serviceType: ClusterIp`, and how does a `NodePort` service still provide an internal cluster DNS name (`aio-broker-internal.azure-iot-operations.svc.cluster.local:1883`)?** +Azure IoT Operations' admission webhook enforces at most one `BrokerListener` per Kubernetes Service type (`ClusterIp`, `NodePort`, `LoadBalancer`) per broker. Because every Kubernetes `NodePort` service automatically allocates an internal `ClusterIP` and CoreDNS record in addition to its host nodePort (`31883`), in-cluster pods can connect over `aio-broker-internal.azure-iot-operations.svc.cluster.local:1883` while host CLI tools can also test against `127.0.0.1:31883`. + +[← Previous Solution](../challenge-01/solution-01.md) | [Back to Challenge 02](../../challenges/challenge-02.md) | [Next Solution →](../challenge-03/solution-03.md) diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-03/solution-03.md b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-03/solution-03.md new file mode 100644 index 000000000..14a6e0483 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-03/solution-03.md @@ -0,0 +1,152 @@ +# Solution 03 — Deploy Foundry Local Operator & Serve a Quantized CPU SLM + +**[Home](../../Readme.md)** - [← Previous Solution](../challenge-02/solution-02.md) - [Back to Challenge 03](../../challenges/challenge-03.md) - [Next Solution →](../challenge-04/solution-04.md) + +**Duration:** 45 minutes + +## Step 1: Install Kubernetes Gateway API CRDs & Istio + +Foundry Local routes model inference traffic through the **Kubernetes Gateway API** using **Istio** (`istio-base` + `istiod`) as the Gateway controller. Install the Gateway API CRDs and Inference Extension CRDs **before** installing `istiod`: + +```bash +source ~/edge-env.sh +cd ~/MicroHack/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI + +# 1. Install Kubernetes Gateway API CRDs (v1.4.0 standard channel, reconciling with cert-manager's Helm fields) +kubectl apply --server-side --force-conflicts -f \ + https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml + +# 2. Install Gateway API Inference Extension CRDs (v1.5.0) +kubectl apply --server-side --force-conflicts -f \ + https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v1.5.0/manifests.yaml + +# 3. Install Istio base and istiod with Inference Extension support +helm repo add istio https://istio-release.storage.googleapis.com/charts +helm repo update + +helm upgrade --install istio-base istio/base \ + -n istio-system --create-namespace --wait + +helm upgrade --install istiod istio/istiod \ + -n istio-system \ + --set pilot.env.ENABLE_GATEWAY_API_INFERENCE_EXTENSION=true \ + --wait +``` + +Verify that the `istio` `GatewayClass` is `Accepted`: + +```bash +kubectl get gatewayclass istio +``` + +Expected output: + +```text +NAME CONTROLLER ACCEPTED AGE +istio istio.io/gateway-controller True 1m +``` + +--- + +## Step 2: Update `cert-manager` (`Microsoft.CertManagement`) for Foundry Local + +When you ran `az iot ops init` in Challenge 02, Azure IoT Operations automatically installed the **`cert-manager` (`microsoft.certmanagement`)** Arc extension, but configured it with `trust-manager.secretTargets.enabled=false`. + +Foundry Local (`Microsoft.Foundry`) requires `trust-manager` secret targets (`foundry-local-operator-ca-bundle`) and Gateway API certificate support to be enabled on `cert-manager`. Update the existing `cert-manager` extension in place **before** installing Foundry Local: + +```bash +az k8s-extension update \ + --cluster-name "$CLUSTER_NAME" \ + --name "cert-manager" \ + --resource-group "$RESOURCE_GROUP" \ + --cluster-type connectedClusters \ + --config config.enableGatewayAPI=true \ + --config cert-manager.crds.keep=true \ + --config trust-manager.defaultPackage.enabled=false \ + --config trust-manager.secretTargets.enabled=true \ + --config trust-manager.secretTargets.authorizedSecretsAll=true \ + --yes +``` + +--- + +## Step 3: Install the Foundry Local Extension (`Microsoft.Foundry`) + +Install the **Foundry Local** inference operator with: +- Kubernetes **Service Account Token (SAT)** authentication enabled (`satAuth.enabled=true`, `satAuth.audience=foundry-local`) for air-gapped in-cluster agents. +- Reduced model-cache job memory requests (`4Gi` request / `12Gi` limit) sized for quantized CPU SLMs on a 16-vCPU/64-GiB edge host. + +```bash +az k8s-extension create \ + --resource-group "$RESOURCE_GROUP" \ + --cluster-name "$CLUSTER_NAME" \ + --name "inference-operator" \ + --extension-type Microsoft.Foundry \ + --scope cluster \ + --release-namespace "foundry-local-operator" \ + --cluster-type connectedClusters \ + --release-train stable \ + --config entraAuth.enabled=false \ + --config satAuth.enabled=true \ + --config satAuth.audience="foundry-local" \ + --config storeModel.cacheJob.resources.requests.memory="4Gi" \ + --config storeModel.cacheJob.resources.limits.memory="12Gi" +``` + +Verify that the operator pods and catalog ConfigMap are ready in `foundry-local-operator` (waiting for the `catalog-sync-seed` job to finish populating the 177-model catalog): + +```bash +kubectl get pods -n foundry-local-operator +kubectl wait --for=condition=complete job/catalog-sync-seed -n foundry-local-operator --timeout=120s +kubectl get configmap foundry-local-catalog -n foundry-local-operator \ + -o jsonpath="{.data['catalog\.json']}" | jq '.models[] | select(.supportedCompute[] == "cpu") | {alias: .alias, displayName: .displayName, framework: .framework, compute: .supportedCompute}' +``` + +--- + +## Step 4: Deploy the Quantized CPU Small Language Model (`ModelDeployment`) + +First, create the `inference-client` `ServiceAccount` in `edge-agents` so it can be referenced in the `ModelDeployment` authorization list: + +```bash +kubectl apply -f resources/manifests/02-foundry-local/caller-service-account.yaml +``` + +Next, apply [`resources/manifests/02-foundry-local/model-deployment-cpu.yaml`](../../resources/manifests/02-foundry-local/model-deployment-cpu.yaml) to deploy **`phi-4-mini` (`spec.model.catalog.name: phi-4-mini`, which resolves to `Phi-4-mini-instruct-generic-cpu:5`, `compute: cpu`, `runtime: onnx-genai`, `exposure: internal`)**: + +```bash +kubectl apply -f resources/manifests/02-foundry-local/model-deployment-cpu.yaml +``` + +Monitor the `StoreModel` cache job (~6–7 minutes to download and validate the 5.15 GB ONNX INT4 weights into the local cluster registry) and the `ModelDeployment` status until `STATE` is `Running` and `READY` is `true`: + +```bash +kubectl get storemodel,modeldeployment -n foundry-local-operator -w +``` + +Once `phi-4-mini` reaches the `Running` state, inspect its internal cluster endpoint (`https://inference-internal-gateway-istio.foundry-local-operator.svc.cluster.local:443/phi-4-mini`): + +```bash +ENDPOINT=$(kubectl get modeldeployment phi-4-mini -n foundry-local-operator -o jsonpath='{.status.endpoints.internal}') +echo "Internal Foundry Local Endpoint: $ENDPOINT" +``` + +![Challenge 03 — Azure Portal Arc-enabled Kubernetes Extensions](../../img/portal-arc-extensions.png) + +![Challenge 03 — Live Azure Bastion Terminal: StoreModel Caching & Operator Pods](../../img/bastion-challenge-03-storing.png) + +![Challenge 03 — Foundry Local Extension, StoreModel Cache & Phi-4-mini CPU ModelDeployment](../../img/challenge-03-foundry-local-operator.png) + +**Common stumbles:** `inference-operator` fails with `secret "foundry-local-operator-ca-bundle" not found` (happens if Step 2 `az k8s-extension update --name cert-manager` was skipped before installing `Microsoft.Foundry`) · `ModelDeployment` rejected as unknown catalog model (use lowercase alias `phi-4-mini` in `spec.model.catalog.name`, which resolves to `Phi-4-mini-instruct-generic-cpu:5`). + +--- + +## Grounding Questions — Suggested Answers + +**1. How does Foundry Local's `StoreModel` + `inference-operator-model-store` architecture ensure that model pods can restart or scale when the cluster is air-gapped?** +When a `ModelDeployment` is created, the operator spawns a `StoreModel` cache job that downloads the ONNX model files once, verifies their cryptographic hashes and manifest completeness, and pushes them as an OCI artifact into the local in-cluster registry (`model-store.foundry-local-operator.svc.cluster.local:8443`) backed by a persistent volume. Inference pods pull weights exclusively from this local cluster registry, never from the internet. + +**2. Why did we set `entraAuth.enabled=false` and `satAuth.enabled=true` on the `inference-operator` extension for this sovereign edge scenario?** +In an air-gapped or WAN-disconnected factory, pods cannot reach public Microsoft Entra ID endpoints to issue tokens or validate RBAC assignments against Azure Resource Manager. Kubernetes Service Account Token (`satAuth`) authentication relies entirely on the local Kubernetes cluster's OIDC keyset, making it 100% resilient to internet outages. + +[← Previous Solution](../challenge-02/solution-02.md) | [Back to Challenge 03](../../challenges/challenge-03.md) | [Next Solution →](../challenge-04/solution-04.md) diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-04/solution-04.md b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-04/solution-04.md new file mode 100644 index 000000000..fe0787a0f --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-04/solution-04.md @@ -0,0 +1,111 @@ +# Solution 04 — Wire the Autonomous Edge Agent with Zero-Cloud SAT Authentication + +**[Home](../../Readme.md)** - [← Previous Solution](../challenge-03/solution-03.md) - [Back to Challenge 04](../../challenges/challenge-04.md) - [Next Solution →](../challenge-05/solution-05.md) + +**Duration:** 35 minutes + +## Step 1: Verify Kubernetes Service Account Token (SAT) Authentication + +Before deploying the Autonomous Edge Agent pod, verify that a short-lived Kubernetes Service Account Token minted for `inference-client` (`--audience foundry-local`) is accepted by the `phi-4-mini` `ModelDeployment`. + +```bash +source ~/edge-env.sh +cd ~/MicroHack/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI + +# 1. Mint a 10-minute SAT token from the local Kubernetes API server +TOKEN=$(kubectl create token inference-client \ + -n edge-agents \ + --audience foundry-local \ + --duration 10m) + +# 2. Retrieve the internal Gateway endpoint for phi-4-mini +ENDPOINT=$(kubectl get modeldeployment phi-4-mini \ + -n foundry-local-operator \ + -o jsonpath='{.status.endpoints.internal}') + +# 3. Call /v1/models and /v1/chat/completions from inside the cluster with the SAT token +kubectl exec -n edge-agents deploy/manufacturing-telemetry-simulator -- \ + python -c "import json, ssl, urllib.request +ctx=ssl.create_default_context(); ctx.check_hostname=False; ctx.verify_mode=ssl.CERT_NONE +body=json.dumps({'model':'Phi-4-mini-instruct-generic-cpu:5','messages':[{'role':'user','content':'Reply with READY.'}],'max_tokens':16}).encode() +req=urllib.request.Request('${ENDPOINT}/v1/chat/completions', data=body, headers={'Authorization': 'Bearer ${TOKEN}', 'Content-Type': 'application/json'}, method='POST') +print(urllib.request.urlopen(req, context=ctx, timeout=60).read().decode())" +``` + +You should receive a JSON chat completion response (`"content":"READY"`) from `Phi-4-mini-instruct-generic-cpu:5`. + +--- + +## Step 2: Deploy the Local SOP Runbook (`SOP-CNC-042`) + +Apply [`resources/manifests/03-edge-agent/local-sop-configmap.yaml`](../../resources/manifests/03-edge-agent/local-sop-configmap.yaml) so the Edge Agent has local access to the CNC-Extruder operating thresholds and mitigation rules without querying any external database: + +```bash +kubectl apply -f resources/manifests/03-edge-agent/local-sop-configmap.yaml +kubectl describe configmap local-sop-runbook -n edge-agents +``` + +--- + +## Step 3: Deploy the Autonomous Edge Triage & Actuation Agent + +Apply [`resources/manifests/03-edge-agent/edge-agent-deployment.yaml`](../../resources/manifests/03-edge-agent/edge-agent-deployment.yaml): + +```bash +kubectl apply -f resources/manifests/03-edge-agent/edge-agent-deployment.yaml +kubectl rollout status deployment/autonomous-edge-agent -n edge-agents +``` + +Verify that the projected Service Account Token volume is mounted inside the pod at `/var/run/secrets/foundry-local/token`: + +```bash +kubectl exec -n edge-agents deploy/autonomous-edge-agent -- ls -l /var/run/secrets/foundry-local/token +``` + +--- + +## Step 4: Observe the Closed-Loop Edge Workflow + +Follow the logs of the `autonomous-edge-agent` deployment: + +```bash +kubectl logs -n edge-agents deploy/autonomous-edge-agent --tail=30 -f +``` + +When the simulator emits its periodic warning anomaly (`bearingTempC > 85.0` or `vibrationMmPerSec > 7.5`), the AIO Dataflow routes it to `factory/line1/anomalies` and you will see: + +```text +[EDGE-AGENT] <<< Anomaly received on factory/line1/anomalies: machine=CNC-EXTRUDER-01 temp=95.4C vib=9.82mm/s +[EDGE-AGENT] >>> Published autonomous actuation to factory/line1/commands: tool=adjust_plc_setpoint latency=1420.5ms args={"spindle_rpm_delta_pct": -25, "enable_coolant_boost": true, "interlock_mode": "CLOSED_LOOP_THROTTLE", "priority": "P2-WARNING"} +``` + +Check the simulator logs to verify that the simulated PLC received the command on `factory/line1/commands` and throttled the spindle RPM: + +```bash +kubectl logs -n edge-agents deploy/manufacturing-telemetry-simulator --tail=15 +``` + +Expected output: + +```text +[SIMULATOR] <<< RECEIVED AUTONOMOUS PLC COMMAND on factory/line1/commands: action=adjust_plc_setpoint args={"spindle_rpm_delta_pct": -25, "enable_coolant_boost": true, ...} +[SIMULATOR] === PLC ACTUATED: spindleRpm=3150, coolantBoost=True === +``` + +![Challenge 04 — Live Azure Bastion Terminal: Projected Kubernetes SAT JWT & Live Phi-4-mini Completion](../../img/bastion-challenge-04-sat-and-inference.png) + +![Challenge 04 — Closed-Loop Edge Agent & PLC Actuation](../../img/challenge-05-airgap-interlock.png) + +**Common stumbles:** `401 Unauthorized` when calling `/v1/chat/completions` (ensure `--audience foundry-local` was passed to `kubectl create token` and matches `satAuth.audience`) · `404 Not Found` on model name (pass `"model": "Phi-4-mini-instruct-generic-cpu:5"`, the exact ID returned by `/v1/models`). + +--- + +## Grounding Questions — Suggested Answers + +**1. Why does `caller-service-account.yaml` set `automountServiceAccountToken: false` on the `inference-client` ServiceAccount when the pod still needs a Service Account Token?** +By default, Kubernetes mounts a general-purpose token targeted at the Kubernetes API server audience into every pod. Setting `automountServiceAccountToken: false` follows least-privilege security by disabling the default Kubernetes API token and mounting **only** a dedicated projected token bound strictly to `audience: foundry-local` with a 1-hour expiry. + +**2. How does the Foundry Local auth sidecar verify a Kubernetes Service Account Token without calling out to Microsoft Entra ID?** +Each `ModelDeployment` pod runs an authentication sidecar that validates the incoming `Bearer` JWT against the local Kubernetes API server's `TokenReview` / OIDC discovery keys, checks that `aud` equals `foundry-local`, and verifies that the caller's `sub` (`system:serviceaccount:edge-agents:inference-client`) is listed under `authorizedSubjects` with role `GeneralInferenceUser`. + +[← Previous Solution](../challenge-03/solution-03.md) | [Back to Challenge 04](../../challenges/challenge-04.md) | [Next Solution →](../challenge-05/solution-05.md) diff --git a/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-05/solution-05.md b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-05/solution-05.md new file mode 100644 index 000000000..f07b74673 --- /dev/null +++ b/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI/walkthrough/challenge-05/solution-05.md @@ -0,0 +1,108 @@ +# Solution 05 — "Cut the Cord": Air-Gapped Fault Simulation & Verification + +**[Home](../../Readme.md)** - [← Previous Solution](../challenge-04/solution-04.md) - [Back to Challenge 05](../../challenges/challenge-05.md) - [Finish →](../../challenges/finish.md) + +**Duration:** 30 minutes + +## Step 1: Apply the Sovereign Air-Gap `NetworkPolicy` + +Apply [`resources/manifests/04-airgap-policy/deny-external-egress.yaml`](../../resources/manifests/04-airgap-policy/deny-external-egress.yaml) to block all external internet and public Azure cloud egress from the `edge-agents` namespace while allowing only: +- In-cluster DNS (`kube-system` UDP/TCP 53) +- Local Azure IoT Operations MQTT (`azure-iot-operations` TCP 1883/18883) +- Local Foundry Local & Istio Internal Gateway (`foundry-local-operator` & `istio-system` TCP 80/443/8080) + +```bash +source ~/edge-env.sh +cd ~/MicroHack/03-Azure/01-04-AI/09_Air_Gapped_Agentic_Edge_AI + +kubectl apply -f resources/manifests/04-airgap-policy/deny-external-egress.yaml +kubectl describe networkpolicy enforce-airgap-sovereign-mode -n edge-agents +``` + +--- + +## Step 2: Prove That Public Cloud / Internet Egress Is Blocked + +Run an outbound connectivity check from inside the `autonomous-edge-agent` pod to `https://management.azure.com`: + +```bash +kubectl exec -n edge-agents deploy/autonomous-edge-agent -- \ + python -c "import urllib.request; urllib.request.urlopen('https://management.azure.com', timeout=5)" +``` + +Expected result: **Connection timed out / URLError**, confirming that the `edge-agents` namespace cannot reach Azure Resource Manager or public cloud APIs. + +Now verify that internal cluster resolution to the Azure IoT Operations MQTT Broker still succeeds: + +```bash +kubectl exec -n edge-agents deploy/autonomous-edge-agent -- \ + python -c "import socket; print('AIO Broker IP:', socket.gethostbyname('aio-broker-internal.azure-iot-operations.svc.cluster.local'))" +``` + +--- + +## Step 3: Inject a Critical Bearing Thermal-Runaway Fault While Air-Gapped + +Trigger the `inject-critical-bearing-fault` Kubernetes Job ([`resources/manifests/04-airgap-policy/inject-critical-fault.yaml`](../../resources/manifests/04-airgap-policy/inject-critical-fault.yaml)), which publishes a critical emergency reading (`bearingTempC: 114.6`, `vibrationMmPerSec: 15.2`, `coolantFlowLpm: 1.4`) to `factory/line1/telemetry`: + +```bash +kubectl delete job inject-critical-bearing-fault -n edge-agents --ignore-not-found +kubectl apply -f resources/manifests/04-airgap-policy/inject-critical-fault.yaml +kubectl logs -n edge-agents job/inject-critical-bearing-fault -f +``` + +--- + +## Step 4: Verify Autonomous Emergency Interlock Execution + +Inspect the `autonomous-edge-agent` logs to confirm that it processed the critical anomaly using **SOP-CNC-042** and **Foundry Local** (`trigger_safety_interlock`, `-60%` RPM, `P1-CRITICAL-INTERLOCK`): + +```bash +kubectl logs -n edge-agents deploy/autonomous-edge-agent --tail=20 +``` + +Expected output: + +```text +[EDGE-AGENT] <<< Anomaly received on factory/line1/anomalies: machine=CNC-EXTRUDER-01 temp=114.6C vib=15.2mm/s +[EDGE-AGENT] >>> Published autonomous actuation to factory/line1/commands: tool=trigger_safety_interlock latency=... args={"spindle_rpm_delta_pct": -60, "enable_coolant_boost": true, "interlock_mode": "SAFE_HOLD", "priority": "P1-CRITICAL-INTERLOCK"} +``` + +Inspect the `manufacturing-telemetry-simulator` logs to confirm that the spindle RPM was immediately reduced by 60% (`4200 RPM` -> `1680 RPM`) and coolant boost was engaged: + +```bash +kubectl logs -n edge-agents deploy/manufacturing-telemetry-simulator --tail=15 +``` + +Expected output: + +```text +[SIMULATOR] <<< RECEIVED AUTONOMOUS PLC COMMAND on factory/line1/commands: action=trigger_safety_interlock args={"spindle_rpm_delta_pct": -60, "enable_coolant_boost": true, "interlock_mode": "SAFE_HOLD", "priority": "P1-CRITICAL-INTERLOCK"} +[SIMULATOR] === PLC ACTUATED: spindleRpm=1680, coolantBoost=True === +``` + +![Challenge 05 — Live Azure Bastion Terminal: "Cut the Cord" NetworkPolicy & Autonomous PLC Interlock](../../img/bastion-challenge-05-cut-the-cord.png) + +![Challenge 05 — Air-Gapped NetworkPolicy & Emergency PLC Interlock Execution](../../img/challenge-05-airgap-interlock.png) + +--- + +## Troubleshooting + +| Symptom | Cause | Resolution | +| :--- | :--- | :--- | +| `ModelDeployment` stuck in `Pending` | Insufficient allocatable CPU/memory on the K3s node | Ensure the Edge Host VM is at least `Standard_D8s_v5` (`Standard_D16s_v5` default) or reduce `resources.requests.cpu` in `model-deployment-cpu.yaml` to `2`. | +| `401 Unauthorized` or `403 Forbidden` calling Foundry Local | Projected SAT token audience mismatch or missing `authorizedSubjects` | Verify `audience: foundry-local` in `edge-agent-deployment.yaml` and `serviceAccountName: inference-client` in `model-deployment-cpu.yaml`. | +| No messages arriving on `factory/line1/anomalies` | AIO Dataflow not bound to default Broker | Run `kubectl get dataflow cnc-anomaly-router -n azure-iot-operations -o yaml` and verify `status.conditions`. | + +--- + +## Grounding Questions — Suggested Answers + +**1. Why must the `NetworkPolicy` explicitly permit UDP/TCP port `53` to `kube-system` even when all services are inside the same Kubernetes cluster?** +In Kubernetes, once any `Egress` `NetworkPolicy` selects a pod, **all** egress traffic from that pod is denied unless explicitly allowed by a rule—including DNS lookups to `CoreDNS` in the `kube-system` namespace. Without allowing UDP/TCP 53 to `kube-system`, the Edge Agent could not resolve `aio-broker-internal.azure-iot-operations.svc.cluster.local` or `inference-internal-gateway-istio.foundry-local-operator.svc.cluster.local`. + +**2. In `agent.py`, why do we include a deterministic SOP safety guard (`local-sop-deterministic-guard`) alongside the SLM call for critical industrial equipment?** +In industrial control systems (OT), AI agents provide contextual reasoning and multi-signal diagnosis, but safety-critical interlocks must never have a single point of failure if a model pod is restarting or times out. Pairing local SLM reasoning with a deterministic SOP fallback guarantees bounded-time safety actuation under all operating conditions. + +[← Previous Solution](../challenge-04/solution-04.md) | [Back to Challenge 05](../../challenges/challenge-05.md) | [Finish →](../../challenges/finish.md) diff --git a/03-Azure/01-04-AI/Readme.md b/03-Azure/01-04-AI/Readme.md index f04846d4d..3d8606290 100644 --- a/03-Azure/01-04-AI/Readme.md +++ b/03-Azure/01-04-AI/Readme.md @@ -12,5 +12,6 @@ A collection of MicroHacks focused on AI on Azure and the GitHub + Azure agentic * [**Agentic Governance Hub (Citadel framework)**](06_AI_Governance/README.md) — Deploy a unified AI governance platform using the **Citadel Agentic Governance Hub** framework: onboard LLM backends through the hub's API Management gateway, configure access controls and policy enforcement, demonstrate usage tracking and observability, build governed agents on the spoke, and publish them behind unified infrastructure — all with Bicep infrastructure-as-code and Jupyter notebook walkthroughs. * [**Fraud Intelligence Multi-Agent System**](07_Fraud_Intelligence/README.md) — Build a Python-based, multi-agent AML investigation workflow with **Microsoft Foundry** and the **Microsoft Agent Framework**: enrich transactions with evidence from **Azure Cosmos DB** through MCP, assess global and country-specific policies with **Foundry IQ**, generate audit-ready reports and operational alerts, govern model and MCP traffic through **AI Gateway**, and monitor outcomes with **Application Insights** and **Grafana Dashboards**. * [**Predictive Maintenance Multi-Agent System**](08_Predictive_Maintenance/README.md) — Build an end-to-end multi-agent predictive maintenance workflow with **Microsoft Foundry**, **Microsoft Agent Framework**, **.NET Aspire**, and **Azure Cosmos DB**: classify telemetry anomalies, diagnose root causes via **Foundry IQ** knowledge bases and **API Management** MCP tools, plan repairs in C#, and orchestrate cross-language A2A agents into a sequential autonomous maintenance pipeline. +* [**Air-Gapped Agentic AI: Orchestrating Local SLMs and IoT on Edge Kubernetes**](09_Air_Gapped_Agentic_Edge_AI/Readme.md) — Build an end-to-end, disconnected sovereign edge AI and IoT workflow on **Azure Arc-enabled Kubernetes** (K3s on Ubuntu edge host): ingest and filter manufacturing sensor telemetry with **Azure IoT Operations** (MQTT Broker & Dataflows), serve a quantized CPU Small Language Model via **Foundry Local** (`Microsoft.Foundry` Arc extension + Gateway API/Istio + `onnx-genai`), authenticate in-cluster edge agents using **Kubernetes Service Account Tokens (SAT)**, and prove autonomous closed-loop actuation under strict air-gapped network policies. Collaborators wanted :) — see [CONTRIBUTING.md](../../CONTRIBUTING.md) to add your own. \ No newline at end of file