Persistent log storage for PipelineRun/TaskRun logs using OCI Object Storage

[vdemeester]

Problem

When TaskRun/PipelineRun pods are garbage collected (pruner TTL is 3-7 days depending on namespace), their logs are permanently lost. This makes it impossible to debug past failures or audit CI/CD runs.

The Tekton Dashboard is configured with --external-logs=https://logs.infra.tekton.dev/logs, but this endpoint is backed by the old pipelinerun-logs service which reads from GCP Stackdriver — a service we no longer use since migrating to OCI. Effectively, logs for completed runs are unavailable in the Dashboard today.

Additionally, Tekton Results (v0.16.0) is deployed but has no log backend configured, so it only stores run metadata, not actual log output.

Related: #3119 (Phase 2 observability)

Proposal

Use OCI Object Storage (S3-compatible API) as the persistent log store, integrating with both the Tekton Dashboard and Tekton Results.

Architecture

TaskRun/PipelineRun Pods
    │
    ├─ [Fluentd DaemonSet] ──► OCI Object Storage (S3 API)
    │   filters: app.kubernetes.io/         bucket: tekton-logs
    │            managed-by=tekton-pipelines path: <ns>/<pod>/<container>/
    │
    ├─ [Logs Server (Go)] ◄── reads from same bucket
    │   serves: GET /logs/<ns>/<pod>/<container>
    │   replaces: old Stackdriver-based pipelinerun-logs
    │         │
    │         ▼
    │   Tekton Dashboard (--external-logs)
    │
    └─ [Tekton Results] ◄── reads from same bucket
        LOGS_TYPE=Blob, S3 backend

Why OCI Object Storage?

• S3-compatible API — works out of the box with Fluentd S3 output, Tekton Results Blob backend, and standard AWS SDKs
• Cheap — ~$0.0255/GB/month (Standard), ~$0.003/GB/month (Archive). For ~50 GB/month of CI logs, that's ~$1.28/month
• Lifecycle policies — auto-archive logs older than 30 days to reduce cost further
• 20 GB Always Free

Key components

1. Fluentd log collection — Banzaicloud logging-operator deploying Fluentd as DaemonSet. Filters pods with the app.kubernetes.io/managed-by: tekton-pipelines label. Writes to OCI Object Storage in <namespace>/<pod>/<container>/<timestamp>.log path format. This is the exact pattern documented in the Tekton Dashboard walkthrough.

2. Logs server — Lightweight Go HTTP server replacing the current Stackdriver-based pipelinerun-logs/ app. Reads from OCI Object Storage and serves logs at GET /logs/<namespace>/<pod>/<container>, matching the Tekton Dashboard --external-logs contract. Backs logs.infra.tekton.dev.

3. Tekton Results S3 Blob backend — Configure Results with LOGS_TYPE=Blob pointing at the same bucket, so the Results API can serve logs alongside run records. Results supports this natively.

Alternatives considered

Option	Verdict
OCI Log Analytics	Too expensive ($372/unit/month for 300 GB). Overkill for CI/CD logs.
OCI Logging service	$0.05/GB/month, but no S3 API and harder to integrate with Dashboard/Results.
Loki	Good option but adds operational overhead (deploying + maintaining Loki stack).
Store logs via Tekton Results directly	Explicitly discouraged by Results docs — "Storing of data via Tekton Results is inefficient and bad for performance. It's better to use forwarders like Vector, Promtail, Fluentd."

Namespaces covered

tekton-ci, tekton-nightly, bastion-p, bastion-z, automated-releases

/kind feature