MCP Semantic Layer Extension

The first formal Model Context Protocol (MCP) extension for semantic layers - standardizing how AI agents discover, validate, and query governed data with built-in quality gates.

🎯 What is This?

Just as MCP Apps extended MCP with UI capabilities, this project extends MCP with semantic layer capabilities - enabling AI agents to:

• 🔍 Discover data by business terms (not technical names)
• 🛡️ Validate quality before querying (trust gates)
• 🤝 Find relationships automatically (FK + semantic)
• 📊 Access unified data (structured + unstructured)
• 🎯 Get query templates (never execute directly)

🚀 Quick Start

Option 1: With Amazon DataZone (Recommended for Production)

# 1. Install
pip install -r requirements.txt

# 2. Configure AWS + DataZone
export AWS_REGION=us-east-1
export GLUE_CATALOG_ID=123456789012
export DATAZONE_DOMAIN_ID=dzd_xxxxxxxxxxxxx
export S3_BUCKET=my-semantic-layer-bucket
export TRUST_THRESHOLD=0.85

# 3. Run MCP Server
python semantic_layer_server.py

# 4. Test
python test_semantic_server.py

See DATAZONE_DEPLOYMENT_GUIDE.md for complete setup.

Option 2: Glue-Only (Quick Testing)

# 1. Install
pip install -r requirements.txt

# 2. Configure AWS (no DataZone)
export AWS_REGION=us-east-1
export GLUE_CATALOG_ID=123456789012
export S3_BUCKET=my-semantic-layer-bucket
export TRUST_THRESHOLD=0.85

# 3. Run MCP Server
python semantic_layer_server.py

See AWS_DEPLOYMENT_GUIDE.md for testing without DataZone.

🏗️ Architecture

This is a protocol extension with a reference implementation:

┌─────────────────────────────────────────────────────────────┐
│                    MCP Protocol Extension                    │
│  • Standard tool signatures (semantic/discover, etc.)       │
│  • _meta.semantic response format                           │
│  • Trust Gate primitive                                      │
│  • Capability declaration                                    │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│              Reference Implementation (AWS)                  │
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐     │
│  │   Discovery  │  │  Trust Gate  │  │ Relationships│     │
│  │    Engine    │  │   Enforcer   │  │    Engine    │     │
│  └──────────────┘  └──────────────┘  └──────────────┘     │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐  │
│  │         Backend Abstraction Layer                     │  │
│  │  (IStructuredDataHandler, IUnstructuredDataHandler)  │  │
│  └──────────────────────────────────────────────────────┘  │
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐     │
│  │   DataZone   │  │  AWS Glue    │  │   Amazon S3  │     │
│  │  (Business)  │  │ (Technical)  │  │   + Docs     │     │
│  └──────────────┘  └──────────────┘  └──────────────┘     │
│                                                              │
│  ┌──────────────┐  ┌──────────────┐                        │
│  │  Textract    │  │  Comprehend  │                        │
│  │  (Optional)  │  │  (Optional)  │                        │
│  └──────────────┘  └──────────────┘                        │
└─────────────────────────────────────────────────────────────┘

Core AWS Services (Required):

• AWS Glue Data Catalog: Technical metadata, schema, partitions
• Amazon DataZone: Business glossaries, governance, subscriptions (RECOMMENDED)
• Amazon S3: Document storage

Optional AWS Services (Phase 2+):

• Amazon Textract: PDF text extraction (for content-based relationships)
• Amazon Comprehend: Entity extraction (for semantic relationships)

Note: Semantic layer returns query templates only - it does NOT execute queries. Query execution is the AI agent's responsibility.

✨ Core Principles

1. Protocol-Level Governance: Trust gates are part of the protocol, not application logic
2. Business-First API: Agents search by business terms ("customer revenue"), not technical names
3. Backend Agnostic: Clean abstraction layer - AWS is reference, works with any backend
4. Never Execute: Semantic layer returns templates/plans, never executes queries
5. Unified Data Model: Structured (tables) + Unstructured (documents) in one API

✨ v0.2.0 Features - AI-Powered Metadata

NEW: Inspired by Google Dataplex, Databricks, and Snowflake innovations:

• 🤖 AI-Generated Metadata: Auto-generate column descriptions, glossary terms, and LLM instructions using Amazon Bedrock
• 📊 Auto-Generate Semantics: Infer metrics and dimensions from data automatically
• 🎯 Verified Queries: Example queries that AI agents can trust
• 🏷️ Discovery Tags: Enhanced search with AI-suggested tags

See V0.2.0_ENHANCEMENTS.md for complete details.

🛠️ Standard Tools

The extension defines 8 standard tools that all semantic layer implementations should provide:

1. semantic/discover

Discover assets by business terms, not technical names.

{
  "search_query": "customer revenue",
  "glossary_terms": ["customer", "revenue"],
  "min_quality_score": 0.85
}

2. semantic/validate

Trust gate validation before querying.

{
  "asset_id": "sales_db.customer_orders",
  "operation": "read"
}

3. semantic/get_metadata

Complete business metadata for an asset.

{
  "asset_id": "sales_db.customer_orders",
  "include_lineage": true
}

4. semantic/query_template

Get query template with business context (never executes).

{
  "asset_id": "sales_db.customer_orders",
  "columns": ["customer_id", "total_amount"],
  "include_related": true
}

5. semantic/discover_relationships

Find relationships between assets (FK + semantic).

{
  "asset_id": "sales_db.customer_orders",
  "relationship_types": ["foreign_key", "related_to"],
  "max_depth": 2
}

6. semantic/search_unified

Search across structured and unstructured data.

{
  "query": "customer complaints",
  "asset_types": ["table", "document"],
  "include_relationships": true
}

7. semantic/generate_metadata (v0.2.0)

Generate metadata automatically using AI.

{
  "asset_id": "sales_db.customer_orders",
  "use_data_profile": true,
  "generate_llm_instructions": true
}

8. semantic/auto_generate (v0.2.0)

Auto-generate semantic definitions for entire database.

{
  "database": "sales_db",
  "include_metrics": true,
  "include_dimensions": true
}

📊 What Makes This Special?

1. Protocol Extension (Like MCP Apps)

• Standard tool signatures
• _meta.semantic response format
• Capability declaration
• Trust Gate primitive
• Standard error codes

2. Amazon DataZone Integration

Business context is first-class:

• Business glossaries with definitions
• Business names (not just technical)
• Subscription-based access control
• Quality scores tracked over time
• Data lineage and impact analysis

3. Backend Agnostic

Clean abstraction layer means:

• Easy to add Snowflake, Databricks, dbt
• Swap implementations without changing core
• Test with mock backends

4. Production Ready

• Comprehensive error handling
• Configuration management
• Type hints everywhere
• Full documentation
• Test suite included

5. Governance Native

• Trust gate is protocol-level
• Quality scores with every response
• Corrective actions when blocked
• Audit trail ready

🎯 Use Cases

Prevent AI Hallucinations

Agent: "Show me customer revenue"
Semantic Layer: "Found 2 assets, but 'old_customers' has 72% quality - BLOCKED"
Agent: "Using 'customers_v2' instead (95% quality)"

Business-First Discovery

Agent: "Find data about customer complaints"
Semantic Layer: Returns both:
  - complaints_table (structured)
  - complaint_docs/ (unstructured)

Automatic Relationships

Agent: "Get customer orders with product details"
Semantic Layer: "Found FK relationship, here's the JOIN template"

AI-Generated Metadata (v0.2.0)

Agent: "What is txn_amt column?"
Semantic Layer: "Transaction amount in USD, derived from daily sales feed"
Agent: "Show me example queries"
Semantic Layer: "1. Show total revenue by region, 2. What was revenue last quarter?"

📚 Documentation

• V0.2.0_ENHANCEMENTS.md - NEW: AI-powered metadata features
• DATAZONE_DEPLOYMENT_GUIDE.md - Production deployment with DataZone (RECOMMENDED)
• AWS_DEPLOYMENT_GUIDE.md - Testing without DataZone
• QUICKSTART.md - 5-minute setup guide
• MCP_SEMANTIC_LAYER_EXTENSION.md - Protocol specification
• semantic_layer/README.md - Implementation docs
• IMPLEMENTATION_COMPLETE.md - What we built
• CLOUD_PLATFORMS_SEMANTIC_LAYER_COMPARISON.md - Cloud platform analysis

🏗️ Project Structure

governed-truth-mcp/
├── semantic_layer/              # Core implementation
│   ├── models/                  # Data models
│   │   ├── asset.py            # SemanticAsset, StructuredAsset, UnstructuredAsset
│   │   ├── governance.py       # GovernanceMetadata, QualityRuleResult
│   │   ├── relationship.py     # AssetRelationship, RelationshipGraph
│   │   └── config.py           # SemanticLayerConfig
│   ├── backend/                 # Abstraction layer
│   │   └── interfaces.py       # IStructuredDataHandler, IUnstructuredDataHandler
│   ├── aws/                     # AWS implementation
│   │   ├── glue_handler.py     # Glue Data Catalog
│   │   ├── datazone_handler.py # DataZone business context
│   │   ├── s3_handler.py       # S3 + Textract + Comprehend
│   │   └── relationship_handler.py  # Relationship discovery
│   └── core/                    # Business logic
│       ├── discovery.py        # AssetDiscoveryEngine
│       ├── quality.py          # QualityAssessmentFramework
│       ├── trust_gate.py       # TrustGateEnforcer
│       ├── relationships.py    # RelationshipDiscoveryEngine
│       ├── search.py           # UnifiedSearchCoordinator
│       └── metadata_generator.py # AI-powered metadata (v0.2.0)
├── semantic_layer_server.py    # MCP server
├── test_semantic_server.py     # Test suite
└── .kiro/specs/                # Specifications
    └── mcp-semantic-extension/
        ├── requirements.md     # Requirements
        ├── design.md          # Design document
        └── tasks.md           # Implementation tasks

🔧 Configuration

Environment variables:

# AWS Configuration
AWS_REGION=us-east-1
GLUE_CATALOG_ID=123456789012
S3_BUCKET=my-semantic-layer-bucket

# Amazon DataZone (RECOMMENDED for production)
DATAZONE_DOMAIN_ID=dzd_xxxxxxxxxxxxx

# Semantic Layer Configuration
TRUST_THRESHOLD=0.85
ENABLE_VECTOR_SEARCH=false
LOG_LEVEL=INFO

Note: If DATAZONE_DOMAIN_ID is set, the semantic layer will use DataZone for business context, glossaries, and governance. Otherwise, it falls back to Glue-only mode.

🧪 Testing

# Run all tests
python test_semantic_server.py

# Test specific tool
python -c "from test_semantic_server import *; test_semantic_discover()"

# Test with mock backend (no AWS required)
MOCK_BACKEND=true python test_semantic_server.py

🚀 Deployment

Local Development

python semantic_layer_server.py

Docker

docker build -t semantic-layer-mcp .
docker run -e AWS_REGION=us-east-1 semantic-layer-mcp

AWS Lambda (coming soon)

Deploy as Lambda function for serverless operation.

🤝 Contributing

This is a reference implementation of the MCP Semantic Layer Extension. Contributions welcome:

1. Additional backend implementations (Snowflake, Databricks, dbt)
2. Enhanced relationship discovery algorithms
3. Vector embedding support
4. Performance optimizations
5. Additional quality metrics

📄 License

MIT License - see LICENSE file

🎤 Presentations

• AWS Tel Aviv Summit 2026: "Governed AI: MCP Semantic Layer Extension for Trusted Data Access"
• See AWS_SUMMIT_ABSTRACT.md for details

🙏 Acknowledgments

• Model Context Protocol - The foundation
• MCP Apps - Inspiration for protocol extensions
• AWS Glue, DataZone, S3, Textract, Comprehend - Reference implementation

📞 Contact

• GitHub Issues: For bugs and feature requests
• Discussions: For questions and community support

---

Built with ❤️ for the AI + Data Governance community