TwinTrace - Distributed Tracing & Observability Demo

A comprehensive demonstration of distributed tracing and observability in microservices architecture using .NET 8, OpenTelemetry, and modern observability tools.

🎯 Overview

TwinTrace showcases a complete observability stack for monitoring distributed applications. It consists of two microservices that communicate with each other, with full instrumentation for traces, metrics, and logs using OpenTelemetry standards.

Architecture Components

• ApiOne - Data Collection Service (Port 7160)
• ApiTwo - Analytics Engine Service (Port 7161)
• OpenTelemetry Collector - Centralized telemetry collection
• Prometheus - Metrics storage and querying
• Grafana - Metrics visualization and dashboards
• Jaeger - Distributed tracing visualization

🚀 Features

• Distributed Tracing: End-to-end request tracing across microservices
• Metrics Collection: Application and infrastructure metrics
• Centralized Logging: Structured logging with correlation IDs
• Observability Stack: Complete monitoring solution with Prometheus, Grafana, and Jaeger
• Docker Compose: Easy deployment and development setup
• OpenTelemetry Integration: Standards-based instrumentation

📋 Prerequisites

• .NET 8.0 SDK
• Docker Desktop
• Docker Compose

🛠️ Quick Start

1. Clone the Repository

git clone https://github.com/yourusername/TwinTrace.git
cd TwinTrace

2. Start the Observability Stack

docker-compose up -d

This will start:

• OpenTelemetry Collector (Ports: 4317, 4318, 4319)
• Prometheus (Port: 9090)
• Grafana (Port: 3000) - Username: admin, Password: admin
• Jaeger (Port: 16686)

3. Run the Applications

Option A: Using Docker (Recommended)

# Build and run both APIs
docker-compose -f docker-compose.yaml up --build

Option B: Using .NET CLI

# Terminal 1 - Run ApiOne
cd ApiOne
dotnet run

# Terminal 2 - Run ApiTwo  
cd ApiTwo
dotnet run

4. Test the Application

Send a Test Event

curl -X POST "https://localhost:7160/api/Event/SendEvent" \
  -H "Content-Type: application/json" \
  -d '{
    "userId": "12345",
    "eventSource": "WebApp",
    "eventType": "PageView",
    "eventData": "HomePage",
    "eventTime": "2024-01-15T10:30:00Z"
  }'

📊 Observability Dashboards

Access Points

• Grafana: http://localhost:3000 (admin/admin)
• Jaeger: http://localhost:16686
• Prometheus: http://localhost:9090
• ApiOne Swagger: https://localhost:7160/swagger
• ApiTwo Swagger: https://localhost:7161/swagger

Key Metrics to Monitor

• Request duration and throughput
• Error rates and status codes
• HTTP client metrics
• Custom business metrics

🏗️ Project Structure

TwinTrace/
├── ApiOne/                    # Data Collection Service
│   ├── Controllers/          # API endpoints
│   ├── Models/               # Data models
│   ├── Services/             # Business logic
│   ├── Interfaces/           # Service contracts
│   └── Program.cs            # Application entry point
├── ApiTwo/                   # Analytics Engine Service
│   ├── Controllers/          # API endpoints
│   ├── Models/               # Data models
│   ├── Services/             # Business logic
│   ├── Interfaces/           # Service contracts
│   └── Program.cs            # Application entry point
├── docker-compose.yaml       # Container orchestration
├── otel-collector-config.yaml # OpenTelemetry configuration
├── prometheus.yml            # Prometheus configuration
└── TwinTrace.sln             # Visual Studio solution

🔧 Configuration

OpenTelemetry Collector

The collector is configured to:

• Receive OTLP traces, metrics, and logs
• Export traces to Jaeger
• Export metrics to Prometheus
• Provide debug output for development

Prometheus

Configured to scrape metrics from the OpenTelemetry Collector every 15 seconds.

Application Configuration

Both APIs are configured with:

• OpenTelemetry instrumentation for ASP.NET Core
• HTTP client instrumentation
• Console and OTLP exporters
• Structured logging

🧪 Testing

Manual Testing

1. Send Events: Use the /api/Event/SendEvent endpoint in ApiOne
2. View Traces: Check Jaeger UI for distributed traces
3. Monitor Metrics: View Prometheus metrics and Grafana dashboards
4. Check Logs: Monitor application logs in the console

API Endpoints

ApiOne (Data Collector)

• POST /api/Event/SendEvent - Send user events to ApiTwo

ApiTwo (Analytics Engine)

• POST /api/Collector/ReceiveEvents - Receive and process events

🔍 Troubleshooting

Common Issues

1. Port Conflicts: Ensure ports 7160, 7161, 9090, 3000, 16686 are available
2. Docker Issues: Make sure Docker Desktop is running
3. SSL Certificates: For HTTPS, you may need to trust development certificates

Development Certificates

dotnet dev-certs https --trust

📈 Monitoring Best Practices

1. Trace Correlation: All requests are correlated using trace IDs
2. Structured Logging: Logs include structured data for better analysis
3. Metrics Aggregation: Metrics are collected at regular intervals
4. Error Handling: Comprehensive error tracking and logging

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• OpenTelemetry for the observability framework
• Prometheus for metrics collection
• Grafana for visualization
• Jaeger for distributed tracing

📞 Support

For questions and support:

• Open an issue on GitHub
• Check the documentation
• Review the configuration files

---

Happy Observing! 🔍✨