EKS + OpenSearch Observability Lab

A complete lab environment for deploying an EKS cluster with centralized logging via Fluent Bit → OpenSearch.

Goal: Get hands-on with EKS observability, then implement MCP (Model Context Protocol) on top to enable AI agents to query your logs.

Architecture

┌────────────────── EKS Cluster ──────────────────┐
│                                                   │
│  📦 Sample Microservices                          │
│  ┌───────┐ ┌─────────┐ ┌──────────┐             │
│  │ carts │ │ catalog │ │ checkout │             │
│  └───────┘ └─────────┘ └──────────┘             │
│  ┌────────┐ ┌────┐                               │
│  │ orders │ │ ui │                               │
│  └────────┘ └────┘                               │
│       │         │          │                      │
│       ▼         ▼          ▼                      │
│  🔄 Fluent Bit DaemonSet (every node)            │
│     (IRSA → OpenSearch write access)             │
└──────────────────┬────────────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────┐
│  🔍 Amazon OpenSearch (VPC mode)     │
│  • Fine-grained access control       │
│  • Index: eks-app-logs-YYYY.MM.DD    │
│  • OpenSearch Dashboards (Kibana)    │
└──────────────────────────────────────┘

Prerequisites

• AWS CLI v2 configured with appropriate permissions
• kubectl installed
• Bash shell

Quick Start

# Deploy everything (takes ~25 min)
chmod +x deploy.sh
./deploy.sh

# When done, clean up
chmod +x destroy.sh
./destroy.sh

Stack Details

Stack	Template	Resources
VPC	cloudformation/01-vpc.yaml	VPC, 2 public + 2 private subnets, NAT Gateway, IGW
EKS	cloudformation/02-eks.yaml	EKS Cluster, Managed Node Group, OIDC Provider
OpenSearch	cloudformation/03-opensearch.yaml	OpenSearch domain (VPC mode), fine-grained access
IRSA	cloudformation/04-irsa-fluentbit.yaml	IAM Role for Fluent Bit service account

Kubernetes Resources

Manifest	Description
kubernetes/fluent-bit.yaml	DaemonSet + ConfigMap shipping logs to OpenSearch
kubernetes/sample-app.yaml	5 microservices generating sample logs

Accessing OpenSearch Dashboards

OpenSearch runs in VPC mode. Use kubectl port-forward or SSH tunnel:

# Option 1: kubectl port-forward (if you have a bastion pod)
kubectl run bastion --image=alpine/socat -- tcp-listen:5601,fork tcp:<opensearch-endpoint>:443
kubectl port-forward pod/bastion 5601:5601

# Option 2: Use the deploy script output URL with VPN/DirectConnect

Default credentials: admin / Admin123! (change in production!)

Parameters

All stacks accept parameters for customization:

• ClusterName (default: eks-observability-lab)
• OpenSearchDomainName (default: eks-app-logs)
• NodeInstanceType (default: t3.medium)
• NodeGroupSize (default: 2)
• OpenSearchInstanceType (default: t3.small.search)

Next Steps: Implement MCP on Top 🚀

This lab gives you a working EKS + OpenSearch pipeline. The next challenge is to deploy an MCP server that lets AI agents query your logs:

1. Deploy OpenSearch MCP Server on Bedrock AgentCore
   • See: Hosting OpenSearch MCP Server with Amazon Bedrock AgentCore
2. Connect an AI Agent (Amazon Q, Strands, or any MCP-compatible agent)
3. Query logs with natural language — e.g., "Show me all errors from the checkout service in the last hour"

MCP Architecture (target state)

┌──────────────┐     MCP      ┌─────────────────────┐
│  AI Agent    │ ◄──────────► │ OpenSearch MCP Server│
│  (Q / Strands)│              │ (Bedrock AgentCore)  │
└──────────────┘              └──────────┬────────────┘
                                         │
                                         ▼
                              ┌─────────────────────┐
                              │  OpenSearch Cluster   │
                              │  (this lab's infra)   │
                              └─────────────────────┘

Cleanup

./destroy.sh

This deletes all CloudFormation stacks and Kubernetes resources in reverse order.

License

MIT