A self-built observability pipeline for collecting, buffering, storing, querying, and visualizing application logs and metrics.
The project implements the core pieces of a small observability platform from scratch using FastAPI, Redis Streams, TimescaleDB, PostgreSQL, Next.js, React, and Recharts.
It is designed as a hands-on systems/backend engineering project for understanding how telemetry moves from applications into a durable time-series store and finally into a live dashboard.
- I do not have good knowledge of frontend and frontend was made with the help of AI. At least for a moment I am not intersted in frontend stuff only due to my development experience while working with frontend. I just wanted a project with a frontend.
- This project is built and tested on Linux. It should work well with MacOS but I cann't guarantee it especially for windows.
The system accepts structured telemetry from applications and processes it through a decoupled ingestion pipeline.
For each log or metric event, the system:
- Receives the event through a FastAPI ingestion endpoint.
- Validates it using shared Pydantic models.
- Generates an event UUID and UTC timestamp when they are not supplied.
- Publishes the event to a Redis Stream.
- Lets a separate worker consume queued events.
- Writes events to TimescaleDB in batches.
- Acknowledges Redis messages only after database persistence.
- Exposes stored telemetry through a separate query API.
- Aggregates metric values into time buckets using TimescaleDB.
- Displays recent logs, latency, and error-rate metrics in a Next.js dashboard.
- Includes a traffic generator that simulates healthy services and temporary incidents.
This keeps telemetry ingestion separate from database persistence and visualization.
flowchart TD
A[Applications <br>Traffic Generator] --> |HTTP POST <br>/logs <br>POST /metrics| B[Ingestion API <br>FastAPI <br>Port 8000]
B --> |XADD| C[Redis Streams: <br>logs_stream<br>metrics_stream]
C --> |XREADGROUP <br>consumer group: workers| D[Worker <br>Validate event <br>Batch DB insert <br>ACK after persistence]
D --> |SQL| E[TimescaleDB PostgreSQL-compatible:<br> logs hypertable<br> metrics hypertable]
E -->|"SQL queries<br>time_bucket()"| F[Query API FastAPI Port 8001]
F --> |HTTP GET| G[Dashboard Next.js + React Port 3000<br><br>Latency chart<br>Error-rate chart<br>Recent logs]
A telemetry system has a different workload from a normal CRUD application.
Applications may suddenly emit hundreds or thousands of telemetry events during an incident. Writing each incoming event directly to the database would tightly couple application ingestion speed to database performance.
This project therefore introduces Redis Streams as a buffer between ingestion and persistence.
flowchart LR
A[Application] --> B[Ingestion API]
B --> C[Redis Stream]
C --> D[Worker]
D --> E[TimescaleDB]
The ingestion API only needs to validate the event and enqueue it.
The worker can then process telemetry independently.
This separation provides several useful system-design properties:
- ingestion and database persistence are decoupled;
- short traffic bursts can be absorbed by Redis;
- database writes can be grouped into batches;
- workers can acknowledge events after persistence;
- the API layer remains small;
- telemetry storage is optimized for time-based queries;
- the dashboard never talks directly to the database;
- ingestion and querying can evolve independently.
flowchart TD
A[Application] -->|POST /logs| B[Ingestion API]
B -->|Pydantic validation<br>UUID + UTC timestamp| C[Redis]
C -->|XADD logs_stream| D[logs_stream]
D -->|XREADGROUP| E[Worker]
E -->|LogEvent validation<br>batch INSERT| F[TimescaleDB]
F --> G[logs hypertable]
G -->|SELECT| H[Query API]
H -->|GET /logs| I[Dashboard]
flowchart TD
A["Application"] -->|"POST /metrics"| B["Ingestion API"]
B -->|"Pydantic validation"| C["Redis"]
C -->|"XADD metrics_stream"| D["metrics_stream"]
D -->|"XREADGROUP"| E["Worker"]
E -->|"MetricEvent validation<br>batch INSERT"| F["TimescaleDB"]
F --> G["metrics hypertable"]
G -->|"time_bucket()<br>AVG / MIN / MAX"| H["Query API"]
H -->|"GET /metrics"| I["Dashboard"]
I --> J["Request Latency"]
I --> K["Error Rate"]
| Component | Technology | Responsibility |
|---|---|---|
| Backend language | Python | Ingestion, worker, querying, simulation |
| API framework | FastAPI | HTTP ingestion and query APIs |
| Validation | Pydantic | Structured telemetry models |
| Settings | pydantic-settings | Database and Redis configuration |
| Message buffer | Redis Streams | Temporary telemetry queue |
| Redis client | redis-py | Stream publishing and consumption |
| Database | TimescaleDB / PostgreSQL | Durable log and metric storage |
| Database client | Psycopg 3 | PostgreSQL access |
| Time-series aggregation | TimescaleDB time_bucket() |
Metric aggregation |
| HTTP client | HTTPX | Synthetic traffic generation |
| Dashboard | Next.js | Web application |
| UI | React | Dashboard components |
| Language | TypeScript | Dashboard implementation |
| Charts | Recharts | Metric visualization |
| Styling | Tailwind CSS | Dashboard styling |
| Infrastructure | Docker Compose | Local Redis and TimescaleDB |
observability-tool/
│
├── common/
│ ├── __init__.py
│ ├── config.py
│ ├── models.py
│ └── redis_client.py
│
├── ingestion_api/
│ └── main.py
│
├── worker/
│ ├── db.py
│ └── main.py
│
├── query_api/
│ ├── __init__.py
│ ├── db.py
│ └── main.py
│
├── traffic_generator/
│ └── main.py
│
├── dashboard/
│ ├── app/
│ │ ├── dashboard-client.tsx
│ │ ├── globals.css
│ │ ├── layout.tsx
│ │ └── page.tsx
│ │
│ ├── components/
│ │ ├── LogTable.tsx
│ │ └── MetricChart.tsx
│ │
│ ├── lib/
│ │ └── api.ts
│ │
│ ├── public/
│ ├── package.json
│ ├── package-lock.json
│ ├── next.config.ts
│ ├── postcss.config.mjs
│ ├── eslint.config.mjs
│ └── tsconfig.json
│
├── sql/
│ └── complete_setup.sql
│
├── docker-compose.yml
├── requirements.txt
├── .gitignore
├── license.txt
└── README.md
| Component | Responsibility |
|---|---|
common/models.py |
Defines the common log and metric event schemas |
common/config.py |
Loads Redis and database configuration |
common/redis_client.py |
Publishes telemetry into Redis Streams |
ingestion_api/main.py |
HTTP interface for accepting telemetry |
worker/main.py |
Reads telemetry from Redis consumer groups |
worker/db.py |
Persists batches of events into TimescaleDB |
query_api/main.py |
Public read API for logs and metrics |
query_api/db.py |
Implements database queries and metric aggregation |
traffic_generator/main.py |
Generates realistic synthetic telemetry |
sql/complete_setup.sql |
Creates database tables, hypertables, and indexes |
dashboard/lib/api.ts |
Client for communicating with the query API |
dashboard/app/page.tsx |
Loads initial dashboard telemetry |
dashboard/app/dashboard-client.tsx |
Periodically refreshes live dashboard data |
dashboard/components/LogTable.tsx |
Displays recent log events |
dashboard/components/MetricChart.tsx |
Displays time-series metric charts |
docker-compose.yml |
Starts Redis and TimescaleDB locally |
common/models.py defines two event types.
class LogEvent(BaseModel):
event_id: UUID
time: datetime
service: str
level: str
message: str
metadata: Optional[dict]event_id and time are automatically generated when they are not supplied.
Each event therefore receives:
- a UUID;
- a UTC timestamp;
- a service identifier;
- structured event information.
Example:
{
"service": "api-gateway",
"level": "info",
"message": "Request completed",
"metadata": {
"route": "/checkout",
"status": 200
}
}class MetricEvent(BaseModel):
event_id: UUID
time: datetime
service: str
metric_name: str
value: float
tags: Optional[dict]Example:
{
"service": "api-gateway",
"metric_name": "request_latency_ms",
"value": 47.2,
"tags": {
"route": "/checkout"
}
}The ingestion service is implemented in:
ingestion_api/main.py
It exposes two telemetry endpoints.
POST /logs
POST /metrics
The API validates incoming JSON using the shared Pydantic models.
After validation, the event is forwarded to Redis rather than written directly to PostgreSQL.
flowchart LR
A[POST /logs] --> B[LogEvent validation]
B --> C["<code>publish_log#40;#41;</code>"]
C --> D[XADD logs_stream]
flowchart LR
A[POST /metrics] --> B[MetricEvent validation]
B --> C["<code>publish_metric#40;#41;</code>"]
C --> D[XADD metrics_stream]
Successful ingestion returns:
202 Acceptedwith the generated event ID.
Example response:
{
"event_id": "c0402645-6434-44ac-a83c-a26b757ae522"
}If the event cannot be queued in Redis, the ingestion API returns:
503 Service UnavailableInvalid request bodies are handled by FastAPI/Pydantic and normally return:
422 Unprocessable EntityRedis is used as a durable intermediate stream between the ingestion API and database worker.
There are two streams:
logs_stream
metrics_stream
Publishing is handled by:
common/redis_client.py
Conceptually:
XADD logs_stream ...
XADD metrics_stream ...Separating logs and metrics makes it possible to process the two telemetry types independently.
The worker is implemented in:
worker/main.py
It creates the Redis consumer group:
workers
for both telemetry streams.
The current consumer is named:
worker-1
The worker uses Redis XREADGROUP to receive events.
flowchart TD
A[Redis Stream] --> |XREADGROUP| B[Worker]
B --> C[1. deserialize JSON]
B --> D[2. validate Pydantic model]
B --> E[3. insert database batch]
B --> F[4. XACK]
The worker requests up to:
100 messages
per Redis read and blocks for:
500 ms
while waiting for telemetry.
An important property of the worker is the order of operations.
flowchart TD
A["Read message"] --> B["Validate"]
B --> C["Write to DB"]
C --> D["Commit"]
D --> E["XACK"]
Redis messages are acknowledged after database insertion succeeds.
This avoids intentionally acknowledging an event before attempting durable persistence.
Database persistence is implemented in:
worker/db.py
Logs and metrics are inserted using Psycopg executemany().
For logs:
INSERT INTO logs (...)
VALUES (...)
ON CONFLICT (event_id, time) DO NOTHING;Metrics use the same duplicate-protection strategy.
The (event_id, time) unique indexes allow an identical event to be safely ignored if the same serialized telemetry event is inserted again.
Telemetry is stored in TimescaleDB.
TimescaleDB is PostgreSQL-compatible but adds functionality intended for time-series workloads.
The project creates two hypertables:
logs
metrics
The timestamp column is:
time
for both tables.
This allows telemetry to be efficiently organized and queried by time.
| Column | Type | Purpose |
|---|---|---|
event_id |
UUID | Event identifier |
time |
TIMESTAMPTZ | Event time |
service |
TEXT | Producing service |
level |
TEXT | Log severity |
message |
TEXT | Log message |
metadata |
JSONB | Optional structured metadata |
The table is converted into a TimescaleDB hypertable using:
SELECT create_hypertable('logs', 'time');Index:
(service, time DESC)
Unique index:
(event_id, time)
| Column | Type | Purpose |
|---|---|---|
event_id |
UUID | Metric event identifier |
time |
TIMESTAMPTZ | Metric timestamp |
service |
TEXT | Producing service |
metric_name |
TEXT | Metric identifier |
value |
DOUBLE PRECISION | Numeric metric value |
tags |
JSONB | Optional structured labels |
The table becomes a TimescaleDB hypertable using:
SELECT create_hypertable('metrics', 'time');Index:
(service, metric_name, time DESC)
Unique index:
(event_id, time)
The query layer is intentionally separated from telemetry ingestion.
It is implemented in:
query_api/
and normally runs on:
http://localhost:8001
The dashboard only communicates with this API.
It does not directly connect to TimescaleDB.
Endpoint:
GET /logsSupported query parameters:
| Parameter | Description | Default |
|---|---|---|
service |
Filter by service | none |
level |
Filter by log level | none |
since_minutes |
Look-back period | 60 |
since |
Explicit timestamp | none |
limit |
Maximum log records | 100 |
Example:
curl "http://localhost:8001/logs?service=api-gateway&level=error&since_minutes=60&limit=100"The database query orders logs by newest first.
ORDER BY time DESCEndpoint:
GET /metricsSupported parameters:
| Parameter | Description | Default |
|---|---|---|
service |
Filter by service | none |
metric_name |
Filter by metric | none |
since_minutes |
Look-back period | 60 |
bucket_minutes |
Aggregation interval | 1 |
Example:
curl "http://localhost:8001/metrics?service=api-gateway&metric_name=request_latency_ms&since_minutes=60&bucket_minutes=1"TimescaleDB performs metric aggregation using:
time_bucket()For every time bucket, the query calculates:
AVG(value)
MAX(value)
MIN(value)
Conceptually:
SELECT
time_bucket(bucket_width, time) AS bucket,
avg(value) AS avg_value,
max(value) AS max_value,
min(value) AS min_value
FROM metrics
...
GROUP BY bucket
ORDER BY bucket;The API therefore returns data similar to:
[
{
"bucket": "2026-09-04T03:20:00Z",
"avg_value": 48.3,
"max_value": 63.2,
"min_value": 39.4
}
]This means the browser does not need to download every raw metric point to create its charts.
The web dashboard is located in:
dashboard/
and is built using:
Next.js
React
TypeScript
Tailwind CSS
Recharts
The dashboard currently displays:
┌──────────────────────────────────────────────┐
│ Observability Dashboard │
├──────────────────────┬───────────────────────┤
│ │ │
│ Request Latency │ Error Rate │
│ Chart │ Chart │
│ │ │
├──────────────────────┴───────────────────────┤
│ │
│ Recent Logs │
│ │
│ Time Service Level Message │
│ ... api-gateway INFO ... │
│ ... payments ERROR ... │
│ │
└──────────────────────────────────────────────┘
When the dashboard page loads, it retrieves:
- the last 60 minutes of logs;
- the last 60 minutes of
request_latency_ms; - the last 60 minutes of
error_rate.
These are loaded on the server before the client dashboard component is rendered.
After the initial load, the client refreshes data every:
5 seconds
New logs are requested using the timestamp of the previous fetch.
The client keeps up to:
200 recent logs
in memory.
Metric charts are refreshed using approximately the latest:
10 minutes
of data with:
1-minute buckets
Two metric charts currently exist.
Metric:
request_latency_ms
Display:
Request Latency (ms)
Metric:
error_rate
Display:
Error Rate
The charts currently plot:
avg_value
for each TimescaleDB time bucket.
The project includes:
traffic_generator/main.py
to demonstrate the platform without requiring external applications.
It simulates three services:
auth-service
payments-service
api-gateway
For each service it continuously produces:
logs
request_latency_ms
error_rate
and sends them to:
http://localhost:8000
During normal operation, generated telemetry approximately represents:
Log levels:
mostly info
occasional warn
Latency:
around 50 ms
Error rate:
approximately 0–2%
The generator occasionally places a simulated service into an incident state.
During an incident it generates behavior such as:
Log levels:
warn / error
Latency:
approximately 400 ms
Error rate:
approximately 10–40%
Traffic interval:
faster than normal
This creates visible changes in the dashboard and provides a convenient way to test the telemetry pipeline.
Install:
- Git
- Python 3.10 or newer
- Docker with Docker Compose
- Node.js 20.9 or newer
- npm
The dashboard uses Next.js 16, which requires Node.js 20.9+.
git clone https://github.com/CowGivesMilk/observability-tool.git
cd observability-toolpython3 -m venv .venv
source .venv/bin/activatepython -m venv .venv
.\.venv\Scripts\Activate.ps1pip install -r requirements.txtdocker compose up -dThe Compose configuration starts:
| Service | Host port |
|---|---|
| TimescaleDB | 5432 |
| Redis | 6379 |
Check them with:
docker compose psThe database configured by Docker Compose is:
Database: observability
Username: obs
Password: obs
Port: 5432
Ensure the TimescaleDB extension is enabled:
docker compose exec timescaledb \
psql -U obs -d observability \
-c "CREATE EXTENSION IF NOT EXISTS timescaledb;"Then apply the schema.
docker compose exec -T timescaledb \
psql -U obs -d observability \
< sql/complete_setup.sqlGet-Content .\sql\complete_setup.sql |
docker compose exec -T timescaledb psql -U obs -d observabilityThe setup script creates:
logs hypertable
metrics hypertable
service/time indexes
event UUID columns
event uniqueness indexes
complete_setup.sqlis intended for initial database setup and is not currently written as a fully idempotent migration script. Re-running it against an already initialized database may report that tables or indexes already exist.
Configuration is loaded from:
common/config.py
Default values already match the included Docker Compose configuration.
You can therefore run the project without a .env file when using the defaults.
To override configuration, create:
.env
at the project root.
Example:
DB_URL=postgresql://obs:obs@localhost:5432/observability
REDIS_HOST=localhost
REDIS_PORT=6379The defaults are equivalent to:
PostgreSQL:
postgresql://obs:obs@localhost:5432/observability
Redis:
localhost:6379
Open a terminal at the repository root with the Python environment activated.
Run:
python -m uvicorn ingestion_api.main:app --reload --port 8000The ingestion API is now available at:
http://localhost:8000
Interactive FastAPI documentation:
http://localhost:8000/docs
Open another terminal.
Activate the same Python virtual environment and run:
python -m worker.mainThe worker will:
connect to Redis
│
├── create/check consumer groups
│
├── read logs_stream
│
├── read metrics_stream
│
├── validate events
│
├── persist batches
│
└── acknowledge persisted Redis messages
Open another terminal and run:
python -m uvicorn query_api.main:app --reload --port 8001The API is available at:
http://localhost:8001
Interactive API documentation:
http://localhost:8001/docs
The current dashboard expects the query API at this port.
Open another terminal:
cd dashboard
npm ciIf you are intentionally modifying dependencies instead of reproducing the lock file, use:
npm installFrom the dashboard directory:
npm run devOpen:
http://localhost:3000
The Query API currently allows CORS requests from this origin.
Return to the repository root in another terminal.
Run:
python -m traffic_generator.mainThe generator will continuously submit telemetry to:
http://localhost:8000
You should begin seeing:
- logs appear in the dashboard;
- request-latency data populate;
- error-rate data populate;
- occasional simulated incidents.
When everything is running, the system should look like this:
Terminal 1
──────────
docker compose
├── Redis :6379
└── TimescaleDB :5432
Terminal 2
──────────
python -m uvicorn ingestion_api.main:app --reload --port 8000
Terminal 3
──────────
python -m worker.main
Terminal 4
──────────
python -m uvicorn query_api.main:app --reload --port 8001
Terminal 5
──────────
cd dashboard
npm run dev
→ http://localhost:3000
Terminal 6
──────────
python -m traffic_generator.main
You do not need to run the traffic generator to test the pipeline.
Telemetry can be sent manually with curl.
curl -X POST http://localhost:8000/logs \
-H "Content-Type: application/json" \
-d '{
"service": "api-gateway",
"level": "info",
"message": "Request completed",
"metadata": {
"route": "/checkout",
"status": 200
}
}'Expected response:
{
"event_id": "<generated-uuid>"
}HTTP status:
202 Accepted
curl -X POST http://localhost:8000/metrics \
-H "Content-Type: application/json" \
-d '{
"service": "api-gateway",
"metric_name": "request_latency_ms",
"value": 47.2,
"tags": {
"route": "/checkout"
}
}'curl -X POST http://localhost:8000/metrics \
-H "Content-Type: application/json" \
-d '{
"service": "api-gateway",
"metric_name": "error_rate",
"value": 0.03
}'curl "http://localhost:8001/logs"curl "http://localhost:8001/logs?service=api-gateway"curl "http://localhost:8001/logs?level=error"curl "http://localhost:8001/logs?service=payments-service&level=error&since_minutes=30"curl "http://localhost:8001/metrics?metric_name=request_latency_ms&since_minutes=60&bucket_minutes=1"curl "http://localhost:8001/metrics?service=api-gateway&metric_name=request_latency_ms&since_minutes=60&bucket_minutes=5"You can inspect the streams directly.
Enter Redis CLI:
docker compose exec redis redis-cliCheck log-stream length:
XLEN logs_stream
Check metric-stream length:
XLEN metrics_stream
Inspect recent log events:
XREVRANGE logs_stream + - COUNT 5
Inspect the worker consumer group:
XINFO GROUPS logs_stream
Inspect consumers:
XINFO CONSUMERS logs_stream workers
Open PostgreSQL:
docker compose exec timescaledb psql -U obs -d observabilityCheck logs:
SELECT
time,
service,
level,
message
FROM logs
ORDER BY time DESC
LIMIT 20;Check metrics:
SELECT
time,
service,
metric_name,
value
FROM metrics
ORDER BY time DESC
LIMIT 20;Count telemetry:
SELECT COUNT(*) FROM logs;
SELECT COUNT(*) FROM metrics;Inspect hypertables:
SELECT *
FROM timescaledb_information.hypertables;Exit:
\q
The Python services currently share the following configuration.
| Setting | Default | Purpose |
|---|---|---|
DB_URL |
postgresql://obs:obs@localhost:5432/observability |
PostgreSQL/TimescaleDB connection |
REDIS_HOST |
localhost |
Redis hostname |
REDIS_PORT |
6379 |
Redis port |
Some application addresses are currently defined directly in code.
| Component | Current address |
|---|---|
| Ingestion API | http://localhost:8000 |
| Query API | http://localhost:8001 |
| Dashboard | http://localhost:3000 |
| TimescaleDB | localhost:5432 |
| Redis | localhost:6379 |
The traffic generator currently sends events to port 8000, and the dashboard currently queries port 8001.
The current implementation contains several useful reliability mechanisms.
The ingestion endpoint does not synchronously insert telemetry into TimescaleDB.
Redis Streams absorb telemetry before persistence.
The worker reads Redis through a consumer group rather than simply polling the complete stream.
This provides the foundation for coordinating stream consumers.
Events are acknowledged after database insertion succeeds.
flowchart LR
A[Redis Event] --> B[Database insert]
B --> C[Commit]
C --> D[XACK]
The database uses:
ON CONFLICT (event_id, time) DO NOTHINGtogether with unique indexes on:
(event_id, time)
so replaying the same serialized telemetry event does not create an identical duplicate row.
The current dashboard intentionally presents an overall system-level view.
It queries:
request_latency_ms
error_rate
without selecting an individual service.
As a result, values from matching metrics across services are aggregated together into their respective time buckets.
This makes the current dashboard useful for viewing the overall behavior of the synthetic system.
Service-specific dashboard filtering can be added later using the existing service query parameter supported by the Query API.
The project does not currently include a committed automated backend test suite.
Useful development checks include the following.
From the project root:
python -m compileall \
common \
ingestion_api \
worker \
query_api \
traffic_generatorcd dashboard
npm run lintnpm run buildA practical system test is:
1. Start Redis and TimescaleDB
2. Start ingestion API
3. Start worker
4. Start query API
5. POST a log
6. POST a metric
7. Query the events from port 8001
8. Confirm the dashboard displays them
This exercises the complete telemetry path:
HTTP
→ FastAPI
→ Redis
→ Worker
→ TimescaleDB
→ Query API
→ Dashboard
Stop the Python services and dashboard with:
Ctrl+C
Stop infrastructure:
docker compose downThis keeps the Redis and TimescaleDB volumes.
To completely remove local database and Redis data:
docker compose down -v
-vpermanently deletes the local Docker volumes and all telemetry stored in them.
If you want a completely fresh environment:
docker compose down -v
docker compose up -dThen enable TimescaleDB again if required:
docker compose exec timescaledb \
psql -U obs -d observability \
-c "CREATE EXTENSION IF NOT EXISTS timescaledb;"and re-run:
sql/complete_setup.sql
Ensure the TimescaleDB extension is enabled:
docker compose exec timescaledb \
psql -U obs -d observability \
-c "CREATE EXTENSION IF NOT EXISTS timescaledb;"Check Redis:
docker compose psTest it:
docker compose exec redis redis-cli PINGExpected:
PONG
Check:
Redis is running
TimescaleDB is running
database schema has been initialized
worker is running
DB_URL is correct
Inspect Redis:
docker compose exec redis redis-cli XLEN logs_streamIf events are in Redis but not PostgreSQL, inspect the worker terminal for database or event-validation errors.
The traffic generator expects the ingestion API at:
http://localhost:8000
Start it with:
python -m uvicorn ingestion_api.main:app --reload --port 8000The dashboard currently expects the query API at:
http://localhost:8001
Start:
python -m uvicorn query_api.main:app --reload --port 8001Then verify:
http://localhost:8001/logs
The Query API currently permits the dashboard origin:
http://localhost:3000
If you run the dashboard on a different host or port, update the Query API CORS configuration.
complete_setup.sql is currently an initialization script rather than a versioned migration system.
If the database is already initialized, either continue using the existing schema or reset the Docker database volume before applying the script again.
This repository currently focuses on demonstrating the fundamental observability pipeline rather than providing a production monitoring platform.
The worker reads new messages using a Redis consumer group and acknowledges them after persistence, but it does not currently implement explicit stale pending-message recovery using mechanisms such as XAUTOCLAIM.
A worker failure after reading an event but before acknowledging it can therefore leave that event in the pending entries list until recovery logic is added.
The consumer name is currently:
worker-1
Running multiple workers should eventually use unique consumer names.
Events that repeatedly fail validation or database persistence do not currently have a dedicated dead-letter stream.
Docker Compose currently starts only:
TimescaleDB
Redis
The Python APIs, worker, traffic generator, and Next.js dashboard are started separately on the host.
The dashboard query URL and traffic-generator ingestion URL currently use localhost addresses directly in application code.
These should become environment-configurable before containerized or remote deployment.
The Query API currently allows:
http://localhost:3000
for development.
Production deployments would require appropriate origin configuration.
Neither ingestion nor query endpoints currently require authentication.
Any client with network access can submit or retrieve telemetry.
Events are separated by service, but there is currently no concept of:
organization
tenant
project
environment
API key
TimescaleDB retention is not currently configured.
Telemetry will continue accumulating until it is manually removed.
TimescaleDB compression policies are not currently configured.
Log querying currently supports:
service
level
time
limit
but not full-text message searching.
The current telemetry model supports:
logs
metrics
but not distributed traces or spans.
The project visualizes incidents but does not yet implement alert rules or notifications.
The Query API supports service filtering, but the current dashboard shows aggregate metric behavior across services.
The repository does not currently include:
TLS termination
reverse proxy
authentication
container images for application services
orchestration manifests
production secrets management
The current repository does not yet contain a dedicated unit/integration/end-to-end backend test suite.
Natural next steps for the project include:
-
Pending-event recovery
- implement
XAUTOCLAIM; - retry abandoned Redis events;
- introduce a dead-letter stream.
- implement
-
Horizontal worker scaling
- dynamic consumer names;
- multiple worker instances;
- worker health monitoring.
-
Containerize every service
- ingestion API;
- worker;
- query API;
- dashboard;
- traffic generator.
-
Centralized environment configuration
- API URLs;
- ports;
- CORS;
- Redis streams;
- worker identifiers.
-
TimescaleDB lifecycle policies
- retention;
- compression;
- continuous aggregates.
-
Dashboard filtering
- service selector;
- time range;
- log level;
- metric selector.
-
Log search
- message search;
- metadata filtering;
- pagination.
-
Alerting
- latency thresholds;
- error-rate thresholds;
- alert cooldowns;
- webhook notifications.
-
Distributed tracing
- trace IDs;
- span IDs;
- OpenTelemetry ingestion;
- log/trace correlation.
-
Authentication and multi-tenancy
- API keys;
- project ownership;
- tenant isolation.
-
Automated testing
- Pydantic model tests;
- ingestion API tests;
- Redis integration tests;
- database integration tests;
- query API tests;
- worker failure/recovery tests;
- full pipeline tests.
-
CI/CD
- Python linting;
- TypeScript linting;
- frontend build;
- automated tests;
- container builds.
-
Operational health endpoints
/health;- Redis connectivity;
- database connectivity;
- worker status.
-
Observing the observability platform
- ingestion throughput;
- Redis queue depth;
- worker processing rate;
- database-write latency;
- failed-event count.
This project is intentionally small enough to understand end-to-end while still demonstrating several important backend and distributed-system concepts:
Structured telemetry
Pydantic validation
FastAPI service separation
Asynchronous ingestion
Redis Streams
Consumer groups
Buffered event processing
Batch persistence
Idempotent database writes
Time-series databases
TimescaleDB hypertables
Time-bucket aggregation
Service-oriented architecture
Separation of writes and reads
Live dashboard polling
Synthetic fault generation
The important part of the project is therefore not only the dashboard, but the entire telemetry pipeline.
It demonstrates the full telemetry lifecycle:
flowchart TD
A["Generate"] --> B["Ingest"]
B --> C["Validate"]
C --> D["Buffer"]
D --> E["Consume"]
E --> F["Persist"]
F --> G["Aggregate"]
G --> H["Query"]
H --> I["Visualize"]
This project is licensed under the MIT License.
Copyright 2026 Nimesh Poudel
See:
license.txt
for the full license text.