Skip to content

Full rewrite of collective.prometheus - #12

Open
Sakoes wants to merge 2 commits into
collective:masterfrom
Sakoes:rewrite
Open

Full rewrite of collective.prometheus#12
Sakoes wants to merge 2 commits into
collective:masterfrom
Sakoes:rewrite

Conversation

@Sakoes

@Sakoes Sakoes commented Jun 1, 2026

Copy link
Copy Markdown

A full rewrite of the @@metrics endpoint, modernizing it for current Prometheus practices.

What changed:

  • Uses prometheus_client for exposition — proper Content-Type, Cache-Control, and OpenMetrics negotiation. Also mounts the standard PROCESS_COLLECTOR / PLATFORM_COLLECTOR.
  • ZODB load/store metrics are now real counters (zodb_object_loads_total, zodb_object_stores_total), backed by a monotonic accumulator instead of a 15-second sampling window. Resets only on Zope restart — use rate() in dashboards.
  • Cleaner labels: per-database series use a db label (not _<dbname> name suffixes); per-connection series use a connection label (fixes a bug where connection 0 collided with later samples under the old thread label).
  • Renames: zope_cache_memoryzope_cache_objects (it always counted objects, not bytes).
  • New gauges: zope_busy_threads, zope_request_queue_length.
  • Concurrency fix: collector is built per-request instead of stashing request/context on a process-wide singleton, which raced under concurrent scrapes. Activity totals live in a module-level dict behind a lock.
  • Robustness: each sub-collector is isolated — one failure no longer blanks the whole scrape.
  • Cleanup: drops pre-2.5 threadframe fallback; tightens setup.py classifiers (Plone-only, not raw Zope2).

Breaking for anyone with existing dashboards — metric names, labels, and types have changed. Tagged as 2.0.0 (UNRELEASED).

Example output

curl http://localhost:8082/@@metrics
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 5.33516288e+08
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.18329088e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.78032678674e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 12.71
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 24.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="11",patchlevel="13",version="3.11.13"} 1.0
# HELP zope_total_objects Number of objects in the ZODB.
# TYPE zope_total_objects gauge
zope_total_objects{db="main"} 23393.0
# HELP zope_cache_objects Objects currently held in the ZODB cache.
# TYPE zope_cache_objects gauge
zope_cache_objects{db="main"} 11768.0
# HELP zope_cache_size Configured maximum objects per ZODB cache.
# TYPE zope_cache_size gauge
zope_cache_size{db="main"} 15000.0
# HELP zodb_connections Open ZODB connections.
# TYPE zodb_connections gauge
zodb_connections{db="main"} 0.0
# HELP zodb_object_loads_total Total ZODB object loads.
# TYPE zodb_object_loads_total counter
zodb_object_loads_total{db="main"} 3816.0
# HELP zodb_object_stores_total Total ZODB object stores.
# TYPE zodb_object_stores_total counter
zodb_object_stores_total{db="main"} 7937.0
# HELP zope_connection_active_objects Active (non-ghost) objects in a ZODB connection cache.
# TYPE zope_connection_active_objects gauge
zope_connection_active_objects{connection="0",db="main"} 196.0
zope_connection_active_objects{connection="1",db="main"} 178.0
zope_connection_active_objects{connection="2",db="main"} 182.0
zope_connection_active_objects{connection="3",db="main"} 11187.0
zope_connection_active_objects{connection="4",db="main"} 25.0
# HELP zope_connection_total_objects Total objects in a ZODB connection cache.
# TYPE zope_connection_total_objects gauge
zope_connection_total_objects{connection="0",db="main"} 1296.0
zope_connection_total_objects{connection="1",db="main"} 1238.0
zope_connection_total_objects{connection="2",db="main"} 1238.0
zope_connection_total_objects{connection="3",db="main"} 12496.0
zope_connection_total_objects{connection="4",db="main"} 531.0

Sakoes added 2 commits June 1, 2026 17:26
refactor: per-request _ZopeCollector; lock module-level activity accumulator

Removes request-scoped mutable state on the module-level collector
singleton: the prior design stashed self._request / self._context on a
process-wide _COLLECTOR via set_request(), which races under concurrent
/@@metrics scrapes.

- _ZopeCollector now requires (request, context) at construction; one
  instance is built per request inside Prometheus.__call__.
- Per-request CollectorRegistry; PROCESS_COLLECTOR and PLATFORM_COLLECTOR
  are re-registered each scrape (cheap; keeps the registry self-contained).
- Monotonic ZODB activity totals move to a module-level _ACTIVITY dict
  guarded by _ACTIVITY_LOCK so counters survive across scrapes but the
  read-modify-write is serialized per db.

refactor: table-driven collect() and yield from in _safe

style: split multi-statement yield lines and getattr chain; drop redundant or ''

docs: note counter-reset semantics in HISTORY; add overrides.zcml example to README

Also records (no code change) that setup.py classifier drop of
'Framework :: Zope2' is intentional: collective.prometheus targets
Plone, not raw Zope2 (Zope is a transitive concern of Plone).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant