Expose native CouchDB Prometheus endpoint

[mrjones-plip]

What feature do you want to improve?
Monitoring all the things via Watchdog has proven to be amazing. But there's no way to easily access the CouchDB Prometheus endpoint at  /_node/_local/_prometheus - as you need to be logged in (likely as an admin user).

Describe the improvement you'd like
Expose this /_node/_local/_prometheus via a reverse proxy under something like /api/v1/couchdb_prometheus`. No manipulation, just a 1:1 reverse proxy which bypasses the need for authentication. There's already a complimentary ticket on Watchdog to add the ingestion feature once this endpoint exists.

Describe alternatives you've considered
manually set it up, storing the medic password in the watchdog config

Additional context
The _local part of the URL is actually specifying the node. By default, for a single couch node deployment, _local will be fine. For a multinode this would need to do something different - maybe pass in an argument of which node to query? Given how rare multi-node deployments are, I would suggest an MVP that naively defaults to _local and we can enhance it to be multi-node should the need arise later.

There was a question:

there might be some pii leaked. we really need to check the contents

I audited a sample output and didn't see anything. As well, I've asked on CouchDB slack just to double check if there's reasons to not expose this!

sample output of endpoint from a docker helper instance

# HELP couchdb_couch_log_requests_total number of logged messages
# TYPE couchdb_couch_log_requests_total counter
couchdb_couch_log_requests_total{level="alert"} 0
couchdb_couch_log_requests_total{level="critical"} 0
couchdb_couch_log_requests_total{level="debug"} 0
couchdb_couch_log_requests_total{level="emergency"} 0
couchdb_couch_log_requests_total{level="error"} 0
couchdb_couch_log_requests_total{level="info"} 256
couchdb_couch_log_requests_total{level="notice"} 391
couchdb_couch_log_requests_total{level="report"} 0
couchdb_couch_log_requests_total{level="report_error"} 0
couchdb_couch_log_requests_total{level="warning"} 0

# HELP couchdb_couch_replicator_changes_manager_deaths_total number of failed replicator changes managers
# TYPE couchdb_couch_replicator_changes_manager_deaths_total counter
couchdb_couch_replicator_changes_manager_deaths_total 0

# HELP couchdb_couch_replicator_changes_queue_deaths_total number of failed replicator changes work queues
# TYPE couchdb_couch_replicator_changes_queue_deaths_total counter
couchdb_couch_replicator_changes_queue_deaths_total 0

# HELP couchdb_couch_replicator_changes_read_failures_total number of failed replicator changes read failures
# TYPE couchdb_couch_replicator_changes_read_failures_total counter
couchdb_couch_replicator_changes_read_failures_total 0

# HELP couchdb_couch_replicator_changes_reader_deaths_total number of failed replicator changes readers
# TYPE couchdb_couch_replicator_changes_reader_deaths_total counter
couchdb_couch_replicator_changes_reader_deaths_total 0

# HELP couchdb_couch_replicator_checkpoints_failure_total number of failed checkpoint saves
# TYPE couchdb_couch_replicator_checkpoints_failure_total counter
couchdb_couch_replicator_checkpoints_failure_total 0

# HELP couchdb_couch_replicator_checkpoints_total number of checkpoint saves
# TYPE couchdb_couch_replicator_checkpoints_total counter
couchdb_couch_replicator_checkpoints_total 0

# HELP couchdb_couch_replicator_connection_acquires_total number of times connections are shared
# TYPE couchdb_couch_replicator_connection_acquires_total counter
couchdb_couch_replicator_connection_acquires_total 0

# HELP couchdb_couch_replicator_connection_closes_total number of times a worker is gracefully shut down
# TYPE couchdb_couch_replicator_connection_closes_total counter
couchdb_couch_replicator_connection_closes_total 0

# HELP couchdb_couch_replicator_connection_creates_total number of connections created
# TYPE couchdb_couch_replicator_connection_creates_total counter
couchdb_couch_replicator_connection_creates_total 0

# HELP couchdb_couch_replicator_connection_owner_crashes_total number of times a connection owner crashes while owning at least one connection
# TYPE couchdb_couch_replicator_connection_owner_crashes_total counter
couchdb_couch_replicator_connection_owner_crashes_total 0

# HELP couchdb_couch_replicator_connection_releases_total number of times ownership of a connection is released
# TYPE couchdb_couch_replicator_connection_releases_total counter
couchdb_couch_replicator_connection_releases_total 0

# HELP couchdb_couch_replicator_connection_worker_crashes_total number of times a worker unexpectedly terminates
# TYPE couchdb_couch_replicator_connection_worker_crashes_total counter
couchdb_couch_replicator_connection_worker_crashes_total 0

# HELP couchdb_couch_replicator_db_scans_total number of times replicator db scans have been started
# TYPE couchdb_couch_replicator_db_scans_total counter
couchdb_couch_replicator_db_scans_total 1

# HELP couchdb_couch_replicator_docs_completed_state_updates_total number of 'completed' state document updates
# TYPE couchdb_couch_replicator_docs_completed_state_updates_total counter
couchdb_couch_replicator_docs_completed_state_updates_total 0

# HELP couchdb_couch_replicator_docs_db_changes_total number of db changes processed by replicator doc processor
# TYPE couchdb_couch_replicator_docs_db_changes_total counter
couchdb_couch_replicator_docs_db_changes_total 0

# HELP couchdb_couch_replicator_docs_dbs_deleted_total number of db shard deletions seen by replicator doc processor
# TYPE couchdb_couch_replicator_docs_dbs_deleted_total counter
couchdb_couch_replicator_docs_dbs_deleted_total 0

# HELP couchdb_couch_replicator_docs_dbs_found_total number of db shard found by replicator doc processor
# TYPE couchdb_couch_replicator_docs_dbs_found_total counter
couchdb_couch_replicator_docs_dbs_found_total 12

# HELP couchdb_couch_replicator_docs_failed_state_updates_total number of 'failed' state document updates
# TYPE couchdb_couch_replicator_docs_failed_state_updates_total counter
couchdb_couch_replicator_docs_failed_state_updates_total 0

# HELP couchdb_couch_replicator_failed_starts_total number of replications that have failed to start
# TYPE couchdb_couch_replicator_failed_starts_total counter
couchdb_couch_replicator_failed_starts_total 0

# HELP couchdb_couch_replicator_jobs_adds_total number of jobs added to replicator scheduler
# TYPE couchdb_couch_replicator_jobs_adds_total counter
couchdb_couch_replicator_jobs_adds_total 0

# HELP couchdb_couch_replicator_jobs_crashed replicator scheduler crashed jobs
# TYPE couchdb_couch_replicator_jobs_crashed gauge
couchdb_couch_replicator_jobs_crashed 0

# HELP couchdb_couch_replicator_jobs_crashes_total number of job crashed noticed by replicator scheduler
# TYPE couchdb_couch_replicator_jobs_crashes_total counter
couchdb_couch_replicator_jobs_crashes_total 0

# HELP couchdb_couch_replicator_jobs_duplicate_adds_total number of duplicate jobs added to replicator scheduler
# TYPE couchdb_couch_replicator_jobs_duplicate_adds_total counter
couchdb_couch_replicator_jobs_duplicate_adds_total 0

# HELP couchdb_couch_replicator_jobs_pending replicator scheduler pending jobs
# TYPE couchdb_couch_replicator_jobs_pending gauge
couchdb_couch_replicator_jobs_pending 0

# HELP couchdb_couch_replicator_jobs_removes_total number of jobs removed from replicator scheduler
# TYPE couchdb_couch_replicator_jobs_removes_total counter
couchdb_couch_replicator_jobs_removes_total 0

# HELP couchdb_couch_replicator_jobs_running replicator scheduler running jobs
# TYPE couchdb_couch_replicator_jobs_running gauge
couchdb_couch_replicator_jobs_running 0

# HELP couchdb_couch_replicator_jobs_starts_total number of jobs started by replicator scheduler
# TYPE couchdb_couch_replicator_jobs_starts_total counter
couchdb_couch_replicator_jobs_starts_total 0

# HELP couchdb_couch_replicator_jobs_stops_total number of jobs stopped by replicator scheduler
# TYPE couchdb_couch_replicator_jobs_stops_total counter
couchdb_couch_replicator_jobs_stops_total 0

# HELP couchdb_couch_replicator_jobs_total total number of replicator scheduler jobs
# TYPE couchdb_couch_replicator_jobs_total gauge
couchdb_couch_replicator_jobs_total 0

# HELP couchdb_couch_replicator_requests_total number of HTTP requests made by the replicator
# TYPE couchdb_couch_replicator_requests_total counter
couchdb_couch_replicator_requests_total 0

# HELP couchdb_couch_replicator_responses_failure_total number of failed HTTP responses received by the replicator
# TYPE couchdb_couch_replicator_responses_failure_total counter
couchdb_couch_replicator_responses_failure_total 0

# HELP couchdb_couch_replicator_responses_total number of HTTP responses received by the replicator
# TYPE couchdb_couch_replicator_responses_total counter
couchdb_couch_replicator_responses_total 0

# HELP couchdb_couch_replicator_stream_responses_failure_total number of failed streaming HTTP responses received by the replicator
# TYPE couchdb_couch_replicator_stream_responses_failure_total counter
couchdb_couch_replicator_stream_responses_failure_total 0

# HELP couchdb_couch_replicator_stream_responses_total number of streaming HTTP responses received by the replicator
# TYPE couchdb_couch_replicator_stream_responses_total counter
couchdb_couch_replicator_stream_responses_total 0

# HELP couchdb_couch_replicator_worker_deaths_total number of failed replicator workers
# TYPE couchdb_couch_replicator_worker_deaths_total counter
couchdb_couch_replicator_worker_deaths_total 0

# HELP couchdb_couch_replicator_workers_started_total number of replicator workers started
# TYPE couchdb_couch_replicator_workers_started_total counter
couchdb_couch_replicator_workers_started_total 0

# HELP couchdb_auth_cache_requests_total number of authentication cache requests
# TYPE couchdb_auth_cache_requests_total counter
couchdb_auth_cache_requests_total 382

# HELP couchdb_auth_cache_misses_total number of authentication cache misses
# TYPE couchdb_auth_cache_misses_total counter
couchdb_auth_cache_misses_total 0

# HELP couchdb_coalesced_updates_interactive_total number of coalesced interactive updates
# TYPE couchdb_coalesced_updates_interactive_total counter
couchdb_coalesced_updates_interactive_total 0

# HELP couchdb_coalesced_updates_replicated_total number of coalesced replicated updates
# TYPE couchdb_coalesced_updates_replicated_total counter
couchdb_coalesced_updates_replicated_total 0

# HELP couchdb_collect_results_time_seconds microsecond latency for calls to couch_db:collect_results/3
# TYPE couchdb_collect_results_time_seconds summary
couchdb_collect_results_time_seconds{quantile="0.5"} 0.0
couchdb_collect_results_time_seconds{quantile="0.75"} 0.0
couchdb_collect_results_time_seconds{quantile="0.9"} 0.0
couchdb_collect_results_time_seconds{quantile="0.95"} 0.0
couchdb_collect_results_time_seconds{quantile="0.99"} 0.0
couchdb_collect_results_time_seconds{quantile="0.999"} 0.0
couchdb_collect_results_time_seconds_sum 0.0
couchdb_collect_results_time_seconds_count 0

# HELP couchdb_commits_total number of commits performed
# TYPE couchdb_commits_total counter
couchdb_commits_total 340

# HELP couchdb_couch_server_lru_skip_total number of couch_server LRU operations skipped
# TYPE couchdb_couch_server_lru_skip_total counter
couchdb_couch_server_lru_skip_total 0

# HELP couchdb_database_purges_total number of times a database was purged
# TYPE couchdb_database_purges_total counter
couchdb_database_purges_total 0

# HELP couchdb_database_reads_total number of times a document was read from a database
# TYPE couchdb_database_reads_total counter
couchdb_database_reads_total 2716

# HELP couchdb_database_writes_total number of times a database was changed
# TYPE couchdb_database_writes_total counter
couchdb_database_writes_total 156

# HELP couchdb_db_open_time_seconds milliseconds required to open a database
# TYPE couchdb_db_open_time_seconds summary
couchdb_db_open_time_seconds{quantile="0.5"} 0.0
couchdb_db_open_time_seconds{quantile="0.75"} 0.0
couchdb_db_open_time_seconds{quantile="0.9"} 0.0
couchdb_db_open_time_seconds{quantile="0.95"} 0.0
couchdb_db_open_time_seconds{quantile="0.99"} 0.0
couchdb_db_open_time_seconds{quantile="0.999"} 0.0
couchdb_db_open_time_seconds_sum 0.0
couchdb_db_open_time_seconds_count 0

# HELP couchdb_dbinfo_seconds distribution of latencies for calls to retrieve DB info
# TYPE couchdb_dbinfo_seconds summary
couchdb_dbinfo_seconds{quantile="0.5"} 0.0
couchdb_dbinfo_seconds{quantile="0.75"} 0.0
couchdb_dbinfo_seconds{quantile="0.9"} 0.0
couchdb_dbinfo_seconds{quantile="0.95"} 0.0
couchdb_dbinfo_seconds{quantile="0.99"} 0.0
couchdb_dbinfo_seconds{quantile="0.999"} 0.0
couchdb_dbinfo_seconds_sum 0.0
couchdb_dbinfo_seconds_count 0

# HELP couchdb_document_inserts_total number of documents inserted
# TYPE couchdb_document_inserts_total counter
couchdb_document_inserts_total 7

# HELP couchdb_document_purges_failure_total number of failed document purge operations
# TYPE couchdb_document_purges_failure_total counter
couchdb_document_purges_failure_total 0

# HELP couchdb_document_purges_success_total number of successful document purge operations
# TYPE couchdb_document_purges_success_total counter
couchdb_document_purges_success_total 0

# HELP couchdb_document_purges_total number of total document purge operations
# TYPE couchdb_document_purges_total counter
couchdb_document_purges_total 0

# HELP couchdb_document_writes_total number of document write operations
# TYPE couchdb_document_writes_total counter
couchdb_document_writes_total 88

# HELP couchdb_httpd_abandoned_streaming_requests_total number of abandoned streaming requests
# TYPE couchdb_httpd_abandoned_streaming_requests_total counter
couchdb_httpd_abandoned_streaming_requests_total 0

# HELP couchdb_httpd_aborted_requests_total number of aborted requests
# TYPE couchdb_httpd_aborted_requests_total counter
couchdb_httpd_aborted_requests_total 0

# HELP couchdb_httpd_all_docs_timeouts_total number of HTTP all_docs timeouts
# TYPE couchdb_httpd_all_docs_timeouts_total counter
couchdb_httpd_all_docs_timeouts_total 0

# HELP couchdb_httpd_bulk_docs_seconds distribution of the number of docs in _bulk_docs requests
# TYPE couchdb_httpd_bulk_docs_seconds summary
couchdb_httpd_bulk_docs_seconds{quantile="0.5"} 0.0
couchdb_httpd_bulk_docs_seconds{quantile="0.75"} 0.0
couchdb_httpd_bulk_docs_seconds{quantile="0.9"} 0.0
couchdb_httpd_bulk_docs_seconds{quantile="0.95"} 0.0
couchdb_httpd_bulk_docs_seconds{quantile="0.99"} 0.0
couchdb_httpd_bulk_docs_seconds{quantile="0.999"} 0.0
couchdb_httpd_bulk_docs_seconds_sum 0.0
couchdb_httpd_bulk_docs_seconds_count 0

# HELP couchdb_httpd_bulk_requests_total number of bulk requests
# TYPE couchdb_httpd_bulk_requests_total counter
couchdb_httpd_bulk_requests_total 22

# HELP couchdb_httpd_clients_requesting_changes number of clients for continuous _changes
# TYPE couchdb_httpd_clients_requesting_changes gauge
couchdb_httpd_clients_requesting_changes 5

# HELP couchdb_httpd_explain_timeouts_total number of HTTP _explain timeouts
# TYPE couchdb_httpd_explain_timeouts_total counter
couchdb_httpd_explain_timeouts_total 0

# HELP couchdb_httpd_find_timeouts_total number of HTTP find timeouts
# TYPE couchdb_httpd_find_timeouts_total counter
couchdb_httpd_find_timeouts_total 0

# HELP couchdb_httpd_partition_all_docs_requests_total number of partition HTTP _all_docs requests
# TYPE couchdb_httpd_partition_all_docs_requests_total counter
couchdb_httpd_partition_all_docs_requests_total 0

# HELP couchdb_httpd_partition_all_docs_timeouts_total number of partition HTTP all_docs timeouts
# TYPE couchdb_httpd_partition_all_docs_timeouts_total counter
couchdb_httpd_partition_all_docs_timeouts_total 0

# HELP couchdb_httpd_partition_explain_requests_total number of partition HTTP _explain requests
# TYPE couchdb_httpd_partition_explain_requests_total counter
couchdb_httpd_partition_explain_requests_total 0

# HELP couchdb_httpd_partition_explain_timeouts_total number of partition HTTP _explain timeouts
# TYPE couchdb_httpd_partition_explain_timeouts_total counter
couchdb_httpd_partition_explain_timeouts_total 0

# HELP couchdb_httpd_partition_find_requests_total number of partition HTTP _find requests
# TYPE couchdb_httpd_partition_find_requests_total counter
couchdb_httpd_partition_find_requests_total 0

# HELP couchdb_httpd_partition_find_timeouts_total number of partition HTTP find timeouts
# TYPE couchdb_httpd_partition_find_timeouts_total counter
couchdb_httpd_partition_find_timeouts_total 0

# HELP couchdb_httpd_partition_view_requests_total number of partition HTTP view requests
# TYPE couchdb_httpd_partition_view_requests_total counter
couchdb_httpd_partition_view_requests_total 0

# HELP couchdb_httpd_partition_view_timeouts_total number of partition HTTP view timeouts
# TYPE couchdb_httpd_partition_view_timeouts_total counter
couchdb_httpd_partition_view_timeouts_total 0

# HELP couchdb_httpd_purge_requests_total number of purge requests
# TYPE couchdb_httpd_purge_requests_total counter
couchdb_httpd_purge_requests_total 0

# HELP couchdb_httpd_requests_total number of HTTP requests
# TYPE couchdb_httpd_requests_total counter
couchdb_httpd_requests_total 380

# HELP couchdb_httpd_temporary_view_reads_total number of temporary view reads
# TYPE couchdb_httpd_temporary_view_reads_total counter
couchdb_httpd_temporary_view_reads_total 0

# HELP couchdb_httpd_view_reads_total number of view reads
# TYPE couchdb_httpd_view_reads_total counter
couchdb_httpd_view_reads_total 55

# HELP couchdb_httpd_view_timeouts_total number of HTTP view timeouts
# TYPE couchdb_httpd_view_timeouts_total counter
couchdb_httpd_view_timeouts_total 0

# HELP couchdb_httpd_request_methods number of HTTP requests by method
# TYPE couchdb_httpd_request_methods counter
couchdb_httpd_request_methods{method="COPY"} 0
couchdb_httpd_request_methods{method="DELETE"} 0
couchdb_httpd_request_methods{method="GET"} 313
couchdb_httpd_request_methods{method="HEAD"} 0
couchdb_httpd_request_methods{method="OPTIONS"} 0
couchdb_httpd_request_methods{method="POST"} 54
couchdb_httpd_request_methods{method="PUT"} 19

# HELP couchdb_httpd_status_codes number of HTTP responses by status code
# TYPE couchdb_httpd_status_codes counter
couchdb_httpd_status_codes{code="200"} 325
couchdb_httpd_status_codes{code="201"} 38
couchdb_httpd_status_codes{code="202"} 11
couchdb_httpd_status_codes{code="204"} 0
couchdb_httpd_status_codes{code="206"} 0
couchdb_httpd_status_codes{code="301"} 0
couchdb_httpd_status_codes{code="302"} 0
couchdb_httpd_status_codes{code="304"} 0
couchdb_httpd_status_codes{code="400"} 0
couchdb_httpd_status_codes{code="401"} 3
couchdb_httpd_status_codes{code="403"} 0
couchdb_httpd_status_codes{code="404"} 5
couchdb_httpd_status_codes{code="405"} 0
couchdb_httpd_status_codes{code="406"} 0
couchdb_httpd_status_codes{code="409"} 0
couchdb_httpd_status_codes{code="412"} 3
couchdb_httpd_status_codes{code="413"} 0
couchdb_httpd_status_codes{code="414"} 0
couchdb_httpd_status_codes{code="415"} 0
couchdb_httpd_status_codes{code="416"} 0
couchdb_httpd_status_codes{code="417"} 0
couchdb_httpd_status_codes{code="500"} 0
couchdb_httpd_status_codes{code="501"} 0
couchdb_httpd_status_codes{code="503"} 0
couchdb_httpd_status_codes{code="507"} 0

# HELP couchdb_io_queue_search_total Search IO directly triggered by client requests
# TYPE couchdb_io_queue_search_total counter
couchdb_io_queue_search_total 0

# HELP couchdb_io_queue2_search_count_total Search IO directly triggered by client requests
# TYPE couchdb_io_queue2_search_count_total counter
couchdb_io_queue2_search_count_total 0

# HELP couchdb_legacy_checksums_total number of legacy checksums found in couch_file instances
# TYPE couchdb_legacy_checksums_total counter
couchdb_legacy_checksums_total 0

# HELP couchdb_local_document_writes_total number of _local document write operations
# TYPE couchdb_local_document_writes_total counter
couchdb_local_document_writes_total 89

# HELP couchdb_mrview_emits_total number of invocations of `emit' in map functions in the view server
# TYPE couchdb_mrview_emits_total counter
couchdb_mrview_emits_total 22

# HELP couchdb_mrview_map_doc_total number of documents mapped in the view server
# TYPE couchdb_mrview_map_doc_total counter
couchdb_mrview_map_doc_total 74

# HELP couchdb_open_databases number of open databases
# TYPE couchdb_open_databases gauge
couchdb_open_databases 96

# HELP couchdb_open_os_files number of file descriptors CouchDB has open
# TYPE couchdb_open_os_files gauge
couchdb_open_os_files 277

# HELP couchdb_password_hashing_fast_total number of fast password hashing operations
# TYPE couchdb_password_hashing_fast_total counter
couchdb_password_hashing_fast_total 383

# HELP couchdb_password_hashing_slow_total number of slow password hashing operations
# TYPE couchdb_password_hashing_slow_total counter
couchdb_password_hashing_slow_total 1

# HELP couchdb_query_server_acquired_processes_total number of acquired external processes
# TYPE couchdb_query_server_acquired_processes_total counter
couchdb_query_server_acquired_processes_total 299

# HELP couchdb_query_server_calls_add_fun_total number of add_fun requests
# TYPE couchdb_query_server_calls_add_fun_total counter
couchdb_query_server_calls_add_fun_total 969

# HELP couchdb_query_server_calls_ddoc_filter_total number of filter requests
# TYPE couchdb_query_server_calls_ddoc_filter_total counter
couchdb_query_server_calls_ddoc_filter_total 0

# HELP couchdb_query_server_calls_ddoc_new_total number of ddoc new requests
# TYPE couchdb_query_server_calls_ddoc_new_total counter
couchdb_query_server_calls_ddoc_new_total 16

# HELP couchdb_query_server_calls_ddoc_other_total number of other ddoc requests
# TYPE couchdb_query_server_calls_ddoc_other_total counter
couchdb_query_server_calls_ddoc_other_total 0

# HELP couchdb_query_server_calls_ddoc_vdu_total number of vdu requests
# TYPE couchdb_query_server_calls_ddoc_vdu_total counter
couchdb_query_server_calls_ddoc_vdu_total 24

# HELP couchdb_query_server_calls_map_total number of map_doc requests
# TYPE couchdb_query_server_calls_map_total counter
couchdb_query_server_calls_map_total 74

# HELP couchdb_query_server_calls_other_total number of other requests
# TYPE couchdb_query_server_calls_other_total counter
couchdb_query_server_calls_other_total 30

# HELP couchdb_query_server_calls_reduce_total number of reduce requests
# TYPE couchdb_query_server_calls_reduce_total counter
couchdb_query_server_calls_reduce_total 30

# HELP couchdb_query_server_calls_reset_total number of reset requests
# TYPE couchdb_query_server_calls_reset_total counter
couchdb_query_server_calls_reset_total 299

# HELP couchdb_query_server_calls_spawn_proc_total number of spawned query processes
# TYPE couchdb_query_server_calls_spawn_proc_total counter
couchdb_query_server_calls_spawn_proc_total 27

# HELP couchdb_query_server_process_error_exits_total number of error OS process exits
# TYPE couchdb_query_server_process_error_exits_total counter
couchdb_query_server_process_error_exits_total 0

# HELP couchdb_query_server_process_errors_total number of OS process errors
# TYPE couchdb_query_server_process_errors_total counter
couchdb_query_server_process_errors_total 0

# HELP couchdb_query_server_process_exits_total number of normal OS process exits
# TYPE couchdb_query_server_process_exits_total counter
couchdb_query_server_process_exits_total 0

# HELP couchdb_query_server_process_prompts_total number of successful OS process prompts
# TYPE couchdb_query_server_process_prompts_total counter
couchdb_query_server_process_prompts_total 1442

# HELP couchdb_query_server_process_starts_total number of OS process starts
# TYPE couchdb_query_server_process_starts_total counter
couchdb_query_server_process_starts_total 27

# HELP couchdb_query_server_time_add_fun_total accumulated number of microseconds spent processing add_fun requests
# TYPE couchdb_query_server_time_add_fun_total counter
couchdb_query_server_time_add_fun_total 4334674

# HELP couchdb_query_server_time_ddoc_filter_total accumulated number of microseconds spent processing filter requests
# TYPE couchdb_query_server_time_ddoc_filter_total counter
couchdb_query_server_time_ddoc_filter_total 0

# HELP couchdb_query_server_time_ddoc_new_total accumulated number of microseconds spent processing ddoc new requests
# TYPE couchdb_query_server_time_ddoc_new_total counter
couchdb_query_server_time_ddoc_new_total 187735

# HELP couchdb_query_server_time_ddoc_other_total accumulated number of microseconds spent processing other ddoc requests
# TYPE couchdb_query_server_time_ddoc_other_total counter
couchdb_query_server_time_ddoc_other_total 0

# HELP couchdb_query_server_time_ddoc_vdu_total accumulated number of microseconds spent processing vdu requests
# TYPE couchdb_query_server_time_ddoc_vdu_total counter
couchdb_query_server_time_ddoc_vdu_total 212929

# HELP couchdb_query_server_time_map_total accumulated number of microseconds spent processing map_doc requests
# TYPE couchdb_query_server_time_map_total counter
couchdb_query_server_time_map_total 468300

# HELP couchdb_query_server_time_other_total accumulated number of microseconds spent processing other requests
# TYPE couchdb_query_server_time_other_total counter
couchdb_query_server_time_other_total 230653

# HELP couchdb_query_server_time_reduce_total accumulated number of microseconds spent processing reduce requests
# TYPE couchdb_query_server_time_reduce_total counter
couchdb_query_server_time_reduce_total 41721

# HELP couchdb_query_server_time_reset_total accumulated number of microseconds spent processing reset requests
# TYPE couchdb_query_server_time_reset_total counter
couchdb_query_server_time_reset_total 2200034

# HELP couchdb_query_server_time_spawn_proc_total accumulated number of microseconds spent spawning os query processes
# TYPE couchdb_query_server_time_spawn_proc_total counter
couchdb_query_server_time_spawn_proc_total 38705

# HELP couchdb_query_server_vdu_process_time_seconds duration of validate_doc_update function calls
# TYPE couchdb_query_server_vdu_process_time_seconds summary
couchdb_query_server_vdu_process_time_seconds{quantile="0.5"} 0.0
couchdb_query_server_vdu_process_time_seconds{quantile="0.75"} 0.0
couchdb_query_server_vdu_process_time_seconds{quantile="0.9"} 0.0
couchdb_query_server_vdu_process_time_seconds{quantile="0.95"} 0.0
couchdb_query_server_vdu_process_time_seconds{quantile="0.99"} 0.0
couchdb_query_server_vdu_process_time_seconds{quantile="0.999"} 0.0
couchdb_query_server_vdu_process_time_seconds_sum 0.0
couchdb_query_server_vdu_process_time_seconds_count 0

# HELP couchdb_query_server_vdu_rejects_total number of rejections by validate_doc_update function
# TYPE couchdb_query_server_vdu_rejects_total counter
couchdb_query_server_vdu_rejects_total 0

# HELP couchdb_request_time_seconds length of a request inside CouchDB without MochiWeb
# TYPE couchdb_request_time_seconds summary
couchdb_request_time_seconds{quantile="0.5"} 5.15625e-4
couchdb_request_time_seconds{quantile="0.75"} 5.390624999999999e-4
couchdb_request_time_seconds{quantile="0.9"} 5.531249999999999e-4
couchdb_request_time_seconds{quantile="0.95"} 5.578124999999999e-4
couchdb_request_time_seconds{quantile="0.99"} 5.615624999999999e-4
couchdb_request_time_seconds{quantile="0.999"} 5.624062499999999e-4
couchdb_request_time_seconds_sum 0.001359375
couchdb_request_time_seconds_count 3

# HELP couchdb_ddoc_cache_requests_total number of design doc cache requests
# TYPE couchdb_ddoc_cache_requests_total counter
couchdb_ddoc_cache_requests_total 748

# HELP couchdb_ddoc_cache_requests_failures_total number of design doc cache misses
# TYPE couchdb_ddoc_cache_requests_failures_total counter
couchdb_ddoc_cache_requests_failures_total 95

# HELP couchdb_ddoc_cache_requests_recovery_total number of design doc cache recoveries
# TYPE couchdb_ddoc_cache_requests_recovery_total counter
couchdb_ddoc_cache_requests_recovery_total 0

# HELP couchdb_dreyfus_httpd_search_seconds Distribution of overall search request latency as experienced by the end user
# TYPE couchdb_dreyfus_httpd_search_seconds summary
couchdb_dreyfus_httpd_search_seconds{quantile="0.5"} 0.0
couchdb_dreyfus_httpd_search_seconds{quantile="0.75"} 0.0
couchdb_dreyfus_httpd_search_seconds{quantile="0.9"} 0.0
couchdb_dreyfus_httpd_search_seconds{quantile="0.95"} 0.0
couchdb_dreyfus_httpd_search_seconds{quantile="0.99"} 0.0
couchdb_dreyfus_httpd_search_seconds{quantile="0.999"} 0.0
couchdb_dreyfus_httpd_search_seconds_sum 0.0
couchdb_dreyfus_httpd_search_seconds_count 0

# HELP couchdb_dreyfus_index_await_seconds length of an dreyfus_index await request
# TYPE couchdb_dreyfus_index_await_seconds summary
couchdb_dreyfus_index_await_seconds{quantile="0.5"} 0.0
couchdb_dreyfus_index_await_seconds{quantile="0.75"} 0.0
couchdb_dreyfus_index_await_seconds{quantile="0.9"} 0.0
couchdb_dreyfus_index_await_seconds{quantile="0.95"} 0.0
couchdb_dreyfus_index_await_seconds{quantile="0.99"} 0.0
couchdb_dreyfus_index_await_seconds{quantile="0.999"} 0.0
couchdb_dreyfus_index_await_seconds_sum 0.0
couchdb_dreyfus_index_await_seconds_count 0

# HELP couchdb_dreyfus_index_group1_seconds length of an dreyfus_index group1 request
# TYPE couchdb_dreyfus_index_group1_seconds summary
couchdb_dreyfus_index_group1_seconds{quantile="0.5"} 0.0
couchdb_dreyfus_index_group1_seconds{quantile="0.75"} 0.0
couchdb_dreyfus_index_group1_seconds{quantile="0.9"} 0.0
couchdb_dreyfus_index_group1_seconds{quantile="0.95"} 0.0
couchdb_dreyfus_index_group1_seconds{quantile="0.99"} 0.0
couchdb_dreyfus_index_group1_seconds{quantile="0.999"} 0.0
couchdb_dreyfus_index_group1_seconds_sum 0.0
couchdb_dreyfus_index_group1_seconds_count 0

# HELP couchdb_dreyfus_index_group2_seconds length of an dreyfus_index group2 request
# TYPE couchdb_dreyfus_index_group2_seconds summary
couchdb_dreyfus_index_group2_seconds{quantile="0.5"} 0.0
couchdb_dreyfus_index_group2_seconds{quantile="0.75"} 0.0
couchdb_dreyfus_index_group2_seconds{quantile="0.9"} 0.0
couchdb_dreyfus_index_group2_seconds{quantile="0.95"} 0.0
couchdb_dreyfus_index_group2_seconds{quantile="0.99"} 0.0
couchdb_dreyfus_index_group2_seconds{quantile="0.999"} 0.0
couchdb_dreyfus_index_group2_seconds_sum 0.0
couchdb_dreyfus_index_group2_seconds_count 0

# HELP couchdb_dreyfus_index_info_seconds length of an dreyfus_index info request
# TYPE couchdb_dreyfus_index_info_seconds summary
couchdb_dreyfus_index_info_seconds{quantile="0.5"} 0.0
couchdb_dreyfus_index_info_seconds{quantile="0.75"} 0.0
couchdb_dreyfus_index_info_seconds{quantile="0.9"} 0.0
couchdb_dreyfus_index_info_seconds{quantile="0.95"} 0.0
couchdb_dreyfus_index_info_seconds{quantile="0.99"} 0.0
couchdb_dreyfus_index_info_seconds{quantile="0.999"} 0.0
couchdb_dreyfus_index_info_seconds_sum 0.0
couchdb_dreyfus_index_info_seconds_count 0

# HELP couchdb_dreyfus_index_search_seconds length of an dreyfus_index search request
# TYPE couchdb_dreyfus_index_search_seconds summary
couchdb_dreyfus_index_search_seconds{quantile="0.5"} 0.0
couchdb_dreyfus_index_search_seconds{quantile="0.75"} 0.0
couchdb_dreyfus_index_search_seconds{quantile="0.9"} 0.0
couchdb_dreyfus_index_search_seconds{quantile="0.95"} 0.0
couchdb_dreyfus_index_search_seconds{quantile="0.99"} 0.0
couchdb_dreyfus_index_search_seconds{quantile="0.999"} 0.0
couchdb_dreyfus_index_search_seconds_sum 0.0
couchdb_dreyfus_index_search_seconds_count 0

# HELP couchdb_dreyfus_rpc_group1_seconds length of a group1 RPC worker
# TYPE couchdb_dreyfus_rpc_group1_seconds summary
couchdb_dreyfus_rpc_group1_seconds{quantile="0.5"} 0.0
couchdb_dreyfus_rpc_group1_seconds{quantile="0.75"} 0.0
couchdb_dreyfus_rpc_group1_seconds{quantile="0.9"} 0.0
couchdb_dreyfus_rpc_group1_seconds{quantile="0.95"} 0.0
couchdb_dreyfus_rpc_group1_seconds{quantile="0.99"} 0.0
couchdb_dreyfus_rpc_group1_seconds{quantile="0.999"} 0.0
couchdb_dreyfus_rpc_group1_seconds_sum 0.0
couchdb_dreyfus_rpc_group1_seconds_count 0

# HELP couchdb_dreyfus_rpc_group2_seconds length of a group2 RPC worker
# TYPE couchdb_dreyfus_rpc_group2_seconds summary
couchdb_dreyfus_rpc_group2_seconds{quantile="0.5"} 0.0
couchdb_dreyfus_rpc_group2_seconds{quantile="0.75"} 0.0
couchdb_dreyfus_rpc_group2_seconds{quantile="0.9"} 0.0
couchdb_dreyfus_rpc_group2_seconds{quantile="0.95"} 0.0
couchdb_dreyfus_rpc_group2_seconds{quantile="0.99"} 0.0
couchdb_dreyfus_rpc_group2_seconds{quantile="0.999"} 0.0
couchdb_dreyfus_rpc_group2_seconds_sum 0.0
couchdb_dreyfus_rpc_group2_seconds_count 0

# HELP couchdb_dreyfus_rpc_info_seconds length of an info RPC worker
# TYPE couchdb_dreyfus_rpc_info_seconds summary
couchdb_dreyfus_rpc_info_seconds{quantile="0.5"} 0.0
couchdb_dreyfus_rpc_info_seconds{quantile="0.75"} 0.0
couchdb_dreyfus_rpc_info_seconds{quantile="0.9"} 0.0
couchdb_dreyfus_rpc_info_seconds{quantile="0.95"} 0.0
couchdb_dreyfus_rpc_info_seconds{quantile="0.99"} 0.0
couchdb_dreyfus_rpc_info_seconds{quantile="0.999"} 0.0
couchdb_dreyfus_rpc_info_seconds_sum 0.0
couchdb_dreyfus_rpc_info_seconds_count 0

# HELP couchdb_dreyfus_rpc_search_seconds length of a search RPC worker
# TYPE couchdb_dreyfus_rpc_search_seconds summary
couchdb_dreyfus_rpc_search_seconds{quantile="0.5"} 0.0
couchdb_dreyfus_rpc_search_seconds{quantile="0.75"} 0.0
couchdb_dreyfus_rpc_search_seconds{quantile="0.9"} 0.0
couchdb_dreyfus_rpc_search_seconds{quantile="0.95"} 0.0
couchdb_dreyfus_rpc_search_seconds{quantile="0.99"} 0.0
couchdb_dreyfus_rpc_search_seconds{quantile="0.999"} 0.0
couchdb_dreyfus_rpc_search_seconds_sum 0.0
couchdb_dreyfus_rpc_search_seconds_count 0

# HELP couchdb_fabric_doc_update_errors_total number of document update errors
# TYPE couchdb_fabric_doc_update_errors_total counter
couchdb_fabric_doc_update_errors_total 0

# HELP couchdb_fabric_doc_update_mismatched_errors_total number of document update errors with multiple error types
# TYPE couchdb_fabric_doc_update_mismatched_errors_total counter
couchdb_fabric_doc_update_mismatched_errors_total 0

# HELP couchdb_fabric_doc_update_write_quorum_errors_total number of write quorum errors
# TYPE couchdb_fabric_doc_update_write_quorum_errors_total counter
couchdb_fabric_doc_update_write_quorum_errors_total 0

# HELP couchdb_fabric_open_shard_timeouts_total number of open shard timeouts
# TYPE couchdb_fabric_open_shard_timeouts_total counter
couchdb_fabric_open_shard_timeouts_total 0

# HELP couchdb_fabric_read_repairs_failures_total number of failed read repair operations
# TYPE couchdb_fabric_read_repairs_failures_total counter
couchdb_fabric_read_repairs_failures_total 0

# HELP couchdb_fabric_read_repairs_total number of fabric read repairs
# TYPE couchdb_fabric_read_repairs_total counter
couchdb_fabric_read_repairs_total 0

# HELP couchdb_fabric_worker_timeouts_total number of worker timeouts
# TYPE couchdb_fabric_worker_timeouts_total counter
couchdb_fabric_worker_timeouts_total 0

# HELP couchdb_fsync_count_total number of fsync calls
# TYPE couchdb_fsync_count_total counter
couchdb_fsync_count_total 1168

# HELP couchdb_fsync_time_seconds microseconds to call fsync
# TYPE couchdb_fsync_time_seconds summary
couchdb_fsync_time_seconds{quantile="0.5"} 0.0
couchdb_fsync_time_seconds{quantile="0.75"} 0.0
couchdb_fsync_time_seconds{quantile="0.9"} 0.0
couchdb_fsync_time_seconds{quantile="0.95"} 0.0
couchdb_fsync_time_seconds{quantile="0.99"} 0.0
couchdb_fsync_time_seconds{quantile="0.999"} 0.0
couchdb_fsync_time_seconds_sum 0.0
couchdb_fsync_time_seconds_count 0

# HELP couchdb_global_changes_db_writes_total number of db writes performed by global changes
# TYPE couchdb_global_changes_db_writes_total counter
couchdb_global_changes_db_writes_total 15

# HELP couchdb_global_changes_event_doc_conflict_total number of conflicted event docs encountered by global changes
# TYPE couchdb_global_changes_event_doc_conflict_total counter
couchdb_global_changes_event_doc_conflict_total 0

# HELP couchdb_global_changes_listener_pending_updates number of global changes updates pending writes in global_changes_listener
# TYPE couchdb_global_changes_listener_pending_updates gauge
couchdb_global_changes_listener_pending_updates 0

# HELP couchdb_global_changes_rpcs_total number of rpc operations performed by global_changes
# TYPE couchdb_global_changes_rpcs_total counter
couchdb_global_changes_rpcs_total 17

# HELP couchdb_global_changes_server_pending_updates number of global changes updates pending writes in global_changes_server
# TYPE couchdb_global_changes_server_pending_updates gauge
couchdb_global_changes_server_pending_updates 0

# HELP couchdb_mango_docs_examined_total number of documents examined by mango queries coordinated by this node
# TYPE couchdb_mango_docs_examined_total counter
couchdb_mango_docs_examined_total 0

# HELP couchdb_mango_evaluate_selector_total number of mango selector evaluations
# TYPE couchdb_mango_evaluate_selector_total counter
couchdb_mango_evaluate_selector_total 0

# HELP couchdb_mango_keys_examined_total number of keys examined by mango queries coordinated by this node
# TYPE couchdb_mango_keys_examined_total counter
couchdb_mango_keys_examined_total 0

# HELP couchdb_mango_query_invalid_index_total number of mango queries that generated an invalid index warning
# TYPE couchdb_mango_query_invalid_index_total counter
couchdb_mango_query_invalid_index_total 0

# HELP couchdb_mango_query_time_seconds length of time processing a mango query
# TYPE couchdb_mango_query_time_seconds summary
couchdb_mango_query_time_seconds{quantile="0.5"} 0.0
couchdb_mango_query_time_seconds{quantile="0.75"} 0.0
couchdb_mango_query_time_seconds{quantile="0.9"} 0.0
couchdb_mango_query_time_seconds{quantile="0.95"} 0.0
couchdb_mango_query_time_seconds{quantile="0.99"} 0.0
couchdb_mango_query_time_seconds{quantile="0.999"} 0.0
couchdb_mango_query_time_seconds_sum 0.0
couchdb_mango_query_time_seconds_count 0

# HELP couchdb_mango_quorum_docs_examined_total number of documents examined by mango queries, using cluster quorum
# TYPE couchdb_mango_quorum_docs_examined_total counter
couchdb_mango_quorum_docs_examined_total 0

# HELP couchdb_mango_results_returned_total number of rows returned by mango queries
# TYPE couchdb_mango_results_returned_total counter
couchdb_mango_results_returned_total 0

# HELP couchdb_mango_too_many_docs_scanned_total number of mango queries that generated an index scan warning
# TYPE couchdb_mango_too_many_docs_scanned_total counter
couchdb_mango_too_many_docs_scanned_total 0

# HELP couchdb_mango_unindexed_queries_total number of mango queries that could not use an index
# TYPE couchdb_mango_unindexed_queries_total counter
couchdb_mango_unindexed_queries_total 0

# HELP couchdb_mem3_shard_cache_eviction_total number of shard cache evictions
# TYPE couchdb_mem3_shard_cache_eviction_total counter
couchdb_mem3_shard_cache_eviction_total 0

# HELP couchdb_mem3_shard_cache_hit_total number of shard cache hits
# TYPE couchdb_mem3_shard_cache_hit_total counter
couchdb_mem3_shard_cache_hit_total 5932

# HELP couchdb_mem3_shard_cache_miss_total number of shard cache misses
# TYPE couchdb_mem3_shard_cache_miss_total counter
couchdb_mem3_shard_cache_miss_total 15

# HELP couchdb_nouveau_active_searches_total number of active search requests
# TYPE couchdb_nouveau_active_searches_total counter
couchdb_nouveau_active_searches_total 0

# HELP couchdb_nouveau_connection_closed_errors_total number of times we've retried a request after a connection_closed response
# TYPE couchdb_nouveau_connection_closed_errors_total counter
couchdb_nouveau_connection_closed_errors_total 0

# HELP couchdb_nouveau_search_latency_seconds Distribution of overall search request latency as experienced by the end user
# TYPE couchdb_nouveau_search_latency_seconds summary
couchdb_nouveau_search_latency_seconds{quantile="0.5"} 0.0
couchdb_nouveau_search_latency_seconds{quantile="0.75"} 0.0
couchdb_nouveau_search_latency_seconds{quantile="0.9"} 0.0
couchdb_nouveau_search_latency_seconds{quantile="0.95"} 0.0
couchdb_nouveau_search_latency_seconds{quantile="0.99"} 0.0
couchdb_nouveau_search_latency_seconds{quantile="0.999"} 0.0
couchdb_nouveau_search_latency_seconds_sum 0.0
couchdb_nouveau_search_latency_seconds_count 0

# HELP couchdb_nouveau_update_latency_seconds Distribution of time during search waiting for the index to be updated
# TYPE couchdb_nouveau_update_latency_seconds summary
couchdb_nouveau_update_latency_seconds{quantile="0.5"} 0.0
couchdb_nouveau_update_latency_seconds{quantile="0.75"} 0.0
couchdb_nouveau_update_latency_seconds{quantile="0.9"} 0.0
couchdb_nouveau_update_latency_seconds{quantile="0.95"} 0.0
couchdb_nouveau_update_latency_seconds{quantile="0.99"} 0.0
couchdb_nouveau_update_latency_seconds{quantile="0.999"} 0.0
couchdb_nouveau_update_latency_seconds_sum 0.0
couchdb_nouveau_update_latency_seconds_count 0

# HELP couchdb_pread_exceed_eof_total number of the attempts to read beyond end of db file
# TYPE couchdb_pread_exceed_eof_total counter
couchdb_pread_exceed_eof_total 0

# HELP couchdb_pread_exceed_limit_total number of the attempts to read beyond set limit
# TYPE couchdb_pread_exceed_limit_total counter
couchdb_pread_exceed_limit_total 0

# HELP couchdb_rexi_buffered_total number of rexi messages buffered
# TYPE couchdb_rexi_buffered_total counter
couchdb_rexi_buffered_total 0

# HELP couchdb_rexi_down_total number of rexi_DOWN messages handled
# TYPE couchdb_rexi_down_total counter
couchdb_rexi_down_total 0

# HELP couchdb_rexi_dropped_total number of rexi messages dropped from buffers
# TYPE couchdb_rexi_dropped_total counter
couchdb_rexi_dropped_total 0

# HELP couchdb_rexi_streams_timeout_total number of rexi stream timeouts
# TYPE couchdb_rexi_streams_timeout_total counter
couchdb_rexi_streams_timeout_total{stage="init_stream"} 0

# HELP couchdb_rexi_streams_timeout_stream_total number of rexi stream timeouts
# TYPE couchdb_rexi_streams_timeout_stream_total counter
couchdb_rexi_streams_timeout_stream_total 0

# HELP couchdb_rexi_streams_timeout_wait_for_ack_total number of rexi stream timeouts while waiting for acks
# TYPE couchdb_rexi_streams_timeout_wait_for_ack_total counter
couchdb_rexi_streams_timeout_wait_for_ack_total 0

# HELP couchdb_uptime_seconds couchdb uptime
# TYPE couchdb_uptime_seconds counter
couchdb_uptime_seconds 54

# HELP couchdb_erlang_io_recv_bytes_total the total number of bytes received through ports
# TYPE couchdb_erlang_io_recv_bytes_total counter
couchdb_erlang_io_recv_bytes_total 1531948

# HELP couchdb_erlang_io_sent_bytes_total the total number of bytes output to ports
# TYPE couchdb_erlang_io_sent_bytes_total counter
couchdb_erlang_io_sent_bytes_total 27941698

# HELP couchdb_erlang_message_queues total size of all message queues
# TYPE couchdb_erlang_message_queues gauge
couchdb_erlang_message_queues 0

# HELP couchdb_erlang_message_queue_min minimum size across all message queues
# TYPE couchdb_erlang_message_queue_min gauge
couchdb_erlang_message_queue_min 0

# HELP couchdb_erlang_message_queue_max maximum size across all message queues
# TYPE couchdb_erlang_message_queue_max gauge
couchdb_erlang_message_queue_max 0

# HELP couchdb_erlang_message_queue_size size of message queue
# TYPE couchdb_erlang_message_queue_size gauge
couchdb_erlang_message_queue_size{queue_name="couch_server"} 0
couchdb_erlang_message_queue_size{queue_name="index_server"} 0
couchdb_erlang_message_queue_size{queue_name="rexi_server"} 0
couchdb_erlang_message_queue_size{queue_name="rexi_buffer"} 0
couchdb_erlang_message_queue_size{queue_name="couch_stats_process_tracker"} 0
couchdb_erlang_message_queue_size{queue_name="timer_server"} 0
couchdb_erlang_message_queue_size{queue_name="couch_epi_data_gen_dreyfus_black_list"} 0
couchdb_erlang_message_queue_size{queue_name="mem3_reshard_dbdoc"} 0
couchdb_erlang_message_queue_size{queue_name="ddoc_cache_lru"} 0
couchdb_erlang_message_queue_size{queue_name="nouveau_index_manager"} 0
couchdb_erlang_message_queue_size{queue_name="ssl_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_13"} 0
couchdb_erlang_message_queue_size{queue_name="couch_disk_monitor"} 0
couchdb_erlang_message_queue_size{queue_name="tls_server_session_ticket_sup"} 0
couchdb_erlang_message_queue_size{queue_name="custodian_db_checker"} 0
couchdb_erlang_message_queue_size{queue_name="couch_epi_functions_gen_chttpd_handlers"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_14"} 0
couchdb_erlang_message_queue_size{queue_name="ken_server"} 0
couchdb_erlang_message_queue_size{queue_name="couch_event_server"} 0
couchdb_erlang_message_queue_size{queue_name="config"} 0
couchdb_erlang_message_queue_size{queue_name="net_kernel"} 0
couchdb_erlang_message_queue_size{queue_name="socket_registry"} 0
couchdb_erlang_message_queue_size{queue_name="couch_epi_sup"} 0
couchdb_erlang_message_queue_size{queue_name="httpc_profile_sup"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_1"} 0
couchdb_erlang_message_queue_size{queue_name="chttpd_auth_cache_lru"} 0
couchdb_erlang_message_queue_size{queue_name="couch_replicator_scheduler"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_1"} 0
couchdb_erlang_message_queue_size{queue_name="mem3_nodes"} 0
couchdb_erlang_message_queue_size{queue_name="couch_scanner_rate_limiter"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_8"} 0
couchdb_erlang_message_queue_size{queue_name="alarm_handler"} 0
couchdb_erlang_message_queue_size{queue_name="rexi_server_mon"} 0
couchdb_erlang_message_queue_size{queue_name="mochiweb_clock"} 0
couchdb_erlang_message_queue_size{queue_name="couch_peruser"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_9"} 0
couchdb_erlang_message_queue_size{queue_name="jwtf_keystore"} 0
couchdb_erlang_message_queue_size{queue_name="couch_secondary_services"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_16"} 0
couchdb_erlang_message_queue_size{queue_name="ssl_admin_sup"} 0
couchdb_erlang_message_queue_size{queue_name="tls_sup"} 0
couchdb_erlang_message_queue_size{queue_name="mem3_distribution"} 0
couchdb_erlang_message_queue_size{queue_name="custodian_server"} 0
couchdb_erlang_message_queue_size{queue_name="couch_epi_functions_gen_couch_db"} 0
couchdb_erlang_message_queue_size{queue_name="ibrowse"} 0
couchdb_erlang_message_queue_size{queue_name="ssl_manager"} 0
couchdb_erlang_message_queue_size{queue_name="mem3_sup"} 0
couchdb_erlang_message_queue_size{queue_name="smoosh_server"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_3"} 0
couchdb_erlang_message_queue_size{queue_name="dtls_connection_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_cfile"} 0
couchdb_erlang_message_queue_size{queue_name="httpc_sup"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_4"} 0
couchdb_erlang_message_queue_size{queue_name="ssl_connection_sup"} 0
couchdb_erlang_message_queue_size{queue_name="mem3_reshard"} 0
couchdb_erlang_message_queue_size{queue_name="mango_sup"} 0
couchdb_erlang_message_queue_size{queue_name="init"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_11"} 0
couchdb_erlang_message_queue_size{queue_name="erl_epmd"} 0
couchdb_erlang_message_queue_size{queue_name="standard_error_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_epi_functions_gen_chttpd"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_12"} 0
couchdb_erlang_message_queue_size{queue_name="user_drv_writer"} 0
couchdb_erlang_message_queue_size{queue_name="mem3_reshard_sup"} 0
couchdb_erlang_message_queue_size{queue_name="application_controller"} 0
couchdb_erlang_message_queue_size{queue_name="httpc_handler_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_uuids"} 0
couchdb_erlang_message_queue_size{queue_name="couch_replicator_rate_limiter"} 0
couchdb_erlang_message_queue_size{queue_name="disksup"} 0
couchdb_erlang_message_queue_size{queue_name="kernel_safe_sup"} 0
couchdb_erlang_message_queue_size{queue_name="config_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_6"} 0
couchdb_erlang_message_queue_size{queue_name="rexi_buffer_couchdb@127.0.0.1"} 0
couchdb_erlang_message_queue_size{queue_name="dtls_listener_sup"} 0
couchdb_erlang_message_queue_size{queue_name="rex"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_7"} 0
couchdb_erlang_message_queue_size{queue_name="couch_stats_server"} 0
couchdb_erlang_message_queue_size{queue_name="ddoc_cache_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_14"} 0
couchdb_erlang_message_queue_size{queue_name="dreyfus_sup"} 0
couchdb_erlang_message_queue_size{queue_name="tls_server_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_epi_functions_gen_global_changes"} 0
couchdb_erlang_message_queue_size{queue_name="user_drv"} 0
couchdb_erlang_message_queue_size{queue_name="erts_code_purger"} 0
couchdb_erlang_message_queue_size{queue_name="release_handler"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_15"} 0
couchdb_erlang_message_queue_size{queue_name="ken_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_event_sup2"} 0
couchdb_erlang_message_queue_size{queue_name="sasl_sup"} 0
couchdb_erlang_message_queue_size{queue_name="ssl_listen_tracker_sup"} 0
couchdb_erlang_message_queue_size{queue_name="mem3_seeds"} 0
couchdb_erlang_message_queue_size{queue_name="inet_gethost_native"} 0
couchdb_erlang_message_queue_size{queue_name="code_server"} 0
couchdb_erlang_message_queue_size{queue_name="couch_replication"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_2"} 0
couchdb_erlang_message_queue_size{queue_name="global_changes_server"} 0
couchdb_erlang_message_queue_size{queue_name="global_group"} 0
couchdb_erlang_message_queue_size{queue_name="erl_prim_loader"} 0
couchdb_erlang_message_queue_size{queue_name="couch_scanner_server"} 0
couchdb_erlang_message_queue_size{queue_name="ibrowse_sup"} 0
couchdb_erlang_message_queue_size{queue_name="config_event"} 0
couchdb_erlang_message_queue_size{queue_name="ssl_server_session_cache_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_proc_manager"} 0
couchdb_erlang_message_queue_size{queue_name="global_name_server"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_9"} 0
couchdb_erlang_message_queue_size{queue_name="couch_log_server"} 0
couchdb_erlang_message_queue_size{queue_name="dtls_server_session_cache_sup"} 0
couchdb_erlang_message_queue_size{queue_name="rexi_server_sup"} 0
couchdb_erlang_message_queue_size{queue_name="sasl_safe_sup"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_10"} 0
couchdb_erlang_message_queue_size{queue_name="erl_signal_server"} 0
couchdb_erlang_message_queue_size{queue_name="kernel_refc"} 0
couchdb_erlang_message_queue_size{queue_name="jwtf_sup"} 0
couchdb_erlang_message_queue_size{queue_name="mem3_reshard_job_sup"} 0
couchdb_erlang_message_queue_size{queue_name="rexi_buffer_sup"} 0
couchdb_erlang_message_queue_size{queue_name="nouveau_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_index_sup"} 0
couchdb_erlang_message_queue_size{queue_name="custodian_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_epi_functions_gen_chttpd_auth"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_4"} 0
couchdb_erlang_message_queue_size{queue_name="standard_error"} 0
couchdb_erlang_message_queue_size{queue_name="ssl_pem_cache"} 0
couchdb_erlang_message_queue_size{queue_name="mem3_sync"} 0
couchdb_erlang_message_queue_size{queue_name="couch_replicator_connection"} 0
couchdb_erlang_message_queue_size{queue_name="smoosh_sup"} 0
couchdb_erlang_message_queue_size{queue_name="kernel_sup"} 0
couchdb_erlang_message_queue_size{queue_name="chttpd_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_replicator_sup"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_5"} 0
couchdb_erlang_message_queue_size{queue_name="httpd_sup"} 0
couchdb_erlang_message_queue_size{queue_name="ioq_sup"} 0
couchdb_erlang_message_queue_size{queue_name="dtls_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_12"} 0
couchdb_erlang_message_queue_size{queue_name="couch_plugin"} 0
couchdb_erlang_message_queue_size{queue_name="error_logger"} 0
couchdb_erlang_message_queue_size{queue_name="couch_epi_functions_gen_feature_flags"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_13"} 0
couchdb_erlang_message_queue_size{queue_name="couch_prometheus_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_lockout_lru"} 0
couchdb_erlang_message_queue_size{queue_name="httpc_manager"} 0
couchdb_erlang_message_queue_size{queue_name="tls_connection_sup"} 0
couchdb_erlang_message_queue_size{queue_name="mem3_events"} 0
couchdb_erlang_message_queue_size{queue_name="logger_proxy"} 0
couchdb_erlang_message_queue_size{queue_name="memsup"} 0
couchdb_erlang_message_queue_size{queue_name="chttpd"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_7"} 0
couchdb_erlang_message_queue_size{queue_name="global_group_check"} 0
couchdb_erlang_message_queue_size{queue_name="mem3_sync_nodes"} 0
couchdb_erlang_message_queue_size{queue_name="logger"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_8"} 0
couchdb_erlang_message_queue_size{queue_name="couch_stats_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_task_status"} 0
couchdb_erlang_message_queue_size{queue_name="couch_primary_services"} 0
couchdb_erlang_message_queue_size{queue_name="ssl_upgrade_server_session_cache_sup"} 0
couchdb_erlang_message_queue_size{queue_name="logger_std_h_ssl_handler"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_15"} 0
couchdb_erlang_message_queue_size{queue_name="couch_epi_data_gen_flags_config"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_16"} 0
couchdb_erlang_message_queue_size{queue_name="os_mon_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_2"} 0
couchdb_erlang_message_queue_size{queue_name="file_server_2"} 0
couchdb_erlang_message_queue_size{queue_name="mem3_shards"} 0
couchdb_erlang_message_queue_size{queue_name="logger_sup"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_3"} 0
couchdb_erlang_message_queue_size{queue_name="global_changes_sup"} 0
couchdb_erlang_message_queue_size{queue_name="inet_gethost_native_sup"} 0
couchdb_erlang_message_queue_size{queue_name="chttpd_auth_cache"} 0
couchdb_erlang_message_queue_size{queue_name="couch_scanner_sup"} 0
couchdb_erlang_message_queue_size{queue_name="ioq"} 0
couchdb_erlang_message_queue_size{queue_name="auth"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_10"} 0
couchdb_erlang_message_queue_size{queue_name="dreyfus_index_manager"} 0
couchdb_erlang_message_queue_size{queue_name="couch_log_sup"} 0
couchdb_erlang_message_queue_size{queue_name="dtls_server_sup"} 0
couchdb_erlang_message_queue_size{queue_name="rexi_sup"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_11"} 0
couchdb_erlang_message_queue_size{queue_name="couch_peruser_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_httpd_vhost"} 0
couchdb_erlang_message_queue_size{queue_name="couch_passwords_cache_lru"} 0
couchdb_erlang_message_queue_size{queue_name="user"} 0
couchdb_erlang_message_queue_size{queue_name="inet_db"} 0
couchdb_erlang_message_queue_size{queue_name="inets_sup"} 0
couchdb_erlang_message_queue_size{queue_name="setup_sup"} 0
couchdb_erlang_message_queue_size{queue_name="tls_client_ticket_store"} 0
couchdb_erlang_message_queue_size{queue_name="rexi_buffer_mon"} 0
couchdb_erlang_message_queue_size{queue_name="couch_replicator_pg"} 0
couchdb_erlang_message_queue_size{queue_name="logger_handler_watcher"} 0
couchdb_erlang_message_queue_size{queue_name="cpu_sup"} 0
couchdb_erlang_message_queue_size{queue_name="couch_password_hasher"} 0
couchdb_erlang_message_queue_size{queue_name="couch_epi_functions_gen_couch_index"} 0
couchdb_erlang_message_queue_size{queue_name="couch_server_5"} 0
couchdb_erlang_message_queue_size{queue_name="rexi_server_couchdb@127.0.0.1"} 0
couchdb_erlang_message_queue_size{queue_name="couch_replicator_doc_processor"} 0
couchdb_erlang_message_queue_size{queue_name="net_sup"} 0
couchdb_erlang_message_queue_size{queue_name="runtime_tools_sup"} 0
couchdb_erlang_message_queue_size{queue_name="index_server_6"} 0

# HELP couchdb_erlang_message_queue_couch_file size of message queue across couch_file processes
# TYPE couchdb_erlang_message_queue_couch_file summary
couchdb_erlang_message_queue_couch_file{quantile="0.5"} 0
couchdb_erlang_message_queue_couch_file{quantile="0.9"} 0
couchdb_erlang_message_queue_couch_file{quantile="0.99"} 0
couchdb_erlang_message_queue_couch_file_sum 0
couchdb_erlang_message_queue_couch_file_count 277
couchdb_erlang_message_queue_couch_file_min 0
couchdb_erlang_message_queue_couch_file_max 0

# HELP couchdb_erlang_message_queue_couch_db_updater size of message queue across couch_db_updater processes
# TYPE couchdb_erlang_message_queue_couch_db_updater summary
couchdb_erlang_message_queue_couch_db_updater{quantile="0.5"} 0
couchdb_erlang_message_queue_couch_db_updater{quantile="0.9"} 0
couchdb_erlang_message_queue_couch_db_updater{quantile="0.99"} 0
couchdb_erlang_message_queue_couch_db_updater_sum 0
couchdb_erlang_message_queue_couch_db_updater_count 96
couchdb_erlang_message_queue_couch_db_updater_min 0
couchdb_erlang_message_queue_couch_db_updater_max 0

# HELP couchdb_erlang_scheduler_queues the total size of all normal run queues
# TYPE couchdb_erlang_scheduler_queues gauge
couchdb_erlang_scheduler_queues 0

# HELP couchdb_erlang_dirty_cpu_scheduler_queues the total size of all dirty CPU scheduler run queues
# TYPE couchdb_erlang_dirty_cpu_scheduler_queues gauge
couchdb_erlang_dirty_cpu_scheduler_queues 0

# HELP couchdb_erlang_memory_bytes size of memory (in bytes) dynamically allocated by the Erlang emulator
# TYPE couchdb_erlang_memory_bytes gauge
couchdb_erlang_memory_bytes{memory_type="total"} 91791864
couchdb_erlang_memory_bytes{memory_type="processes"} 34718624
couchdb_erlang_memory_bytes{memory_type="processes_used"} 34687400
couchdb_erlang_memory_bytes{memory_type="system"} 57073240
couchdb_erlang_memory_bytes{memory_type="atom"} 680161
couchdb_erlang_memory_bytes{memory_type="atom_used"} 653773
couchdb_erlang_memory_bytes{memory_type="binary"} 3369160
couchdb_erlang_memory_bytes{memory_type="code"} 15008790
couchdb_erlang_memory_bytes{memory_type="ets"} 2364096

# HELP couchdb_erlang_gc_collections_total number of garbage collections by the Erlang emulator
# TYPE couchdb_erlang_gc_collections_total counter
couchdb_erlang_gc_collections_total 66866

# HELP couchdb_erlang_gc_words_reclaimed_total number of words reclaimed by garbage collections
# TYPE couchdb_erlang_gc_words_reclaimed_total counter
couchdb_erlang_gc_words_reclaimed_total 77688284

# HELP couchdb_erlang_context_switches_total total number of context switches
# TYPE couchdb_erlang_context_switches_total counter
couchdb_erlang_context_switches_total 280269

# HELP couchdb_erlang_reductions_total total number of reductions
# TYPE couchdb_erlang_reductions_total counter
couchdb_erlang_reductions_total 56869462

# HELP couchdb_erlang_processes the number of Erlang processes
# TYPE couchdb_erlang_processes gauge
couchdb_erlang_processes 1181

# HELP couchdb_erlang_process_limit the maximum number of simultaneously existing Erlang processes
# TYPE couchdb_erlang_process_limit gauge
couchdb_erlang_process_limit 262144

# HELP couchdb_erlang_ets_table number of ETS tables
# TYPE couchdb_erlang_ets_table gauge
couchdb_erlang_ets_table 229

# HELP couchdb_internal_replication_jobs count of internal replication changes to process
# TYPE couchdb_internal_replication_jobs gauge
couchdb_internal_replication_jobs 0

# HELP couchdb_membership count of nodes in the cluster
# TYPE couchdb_membership gauge
couchdb_membership{nodes="cluster_nodes"} 1
couchdb_membership{nodes="all_nodes"} 1

[vikrantwiz02]

Hey @mrjones-plip, I wanna take on this.

New endpoint: GET /api/v1/couchdb_prometheus

Thin proxy, CHT API fetches /_node/_local/_prometheus internally with admin creds, returns the raw text as-is. No auth required on the public side.

What changes:

• api/src/services/couchdb-prometheus.js —> makes the authenticated CouchDB call
• api/src/controllers/couchdb-prometheus.js —> returns text/plain response
• Two lines in routing.js to register it alongside the other no-auth monitoring routes

Going with _local for now as you suggested, multi-node can be a follow-up if it ever comes up.

[dianabarsan]

On a quick scan, I didn't find any PII, but we should check thoroughly.
Also, it's just a lot of data and I'm unsure which of it is useful.

Maybe we can select some sections that are relevant and only return those metrics - like number of errors, number of non-success response codes, time spent doing something and etc.

[mrjones-plip]

Thanks @vikrantwiz02 - we're not quite ready to ship this yet. Stay tuned!

@dianabarsan - I'm torn. On the one hand, more data, more better when it comes to trouble shooting. On the other hand, more data, more storage, more CPU etc. for watchdog to chew on. I also haven't heard back from Jan or Robert on the slack post. They may yet think it's a terrible idea.

To move us along, there's an intimidating 1148 lines total:

• 187 Blanks lines
• 376 Comments
• 584 Metrics which all start with couchdb_

but let's let's remove "couchdb_" on just the 584 metrics. Then if we take the next two values separated by an underscore (so search for ^([a-z]+)(_[a-z]+)_.* and replace with \1\2. This leaves with 88 "groups" of sorts- this feels more digestible to review in this ticket. See below.

Also cc @witash who I know has been all up in Couch & it's erlang shell and all! Thoughts?

214	erlang_message
40	dreyfus_index
36	couch_replicator
35	query_server
32	dreyfus_rpc
25	httpd_status
10	couch_log
9	mango_query
9	httpd_bulk
8	request_time
8	nouveau_update
8	nouveau_search
8	httpd_partition
8	fsync_time
8	dreyfus_httpd
8	dbinfo_seconds
8	db_open
8	collect_results
7	httpd_request
5	global_changes
3	rexi_streams
3	mem3_shard
3	fabric_doc
3	document_purges
3	ddoc_cache
2	pread_exceed
2	password_hashing
2	membership
2	io_queue
2	httpd_view
2	fabric_read
2	erlang_io
2	coalesced_updates
2	auth_cache
1	uptime_seconds	54
1	rexi_dropped
1	rexi_down
1	rexi_buffered
1	open_os
1	open_databases
1	nouveau_connection
1	nouveau_active
1	mrview_map
1	mrview_emits
1	mango_unindexed
1	mango_too
1	mango_results
1	mango_quorum
1	mango_keys
1	mango_evaluate
1	mango_docs
1	local_document
1	legacy_checksums
1	internal_replication
1	httpd_temporary
1	httpd_requests
1	httpd_purge
1	httpd_find
1	httpd_explain
1	httpd_clients
1	httpd_all
1	httpd_aborted
1	httpd_abandoned
1	fsync_count
1	fabric_worker
1	fabric_open
1	erlang_scheduler
1	erlang_reductions
1	erlang_processes
1	erlang_process
1	erlang_gc
1	erlang_ets
1	erlang_dirty
1	erlang_context
1	document_writes
1	document_inserts
1	database_writes
1	database_reads
1	database_purges
1	couch_server
1	commits_total

[mrjones-plip]

Devs from slack advise against this:

exposing aggregated counters to the public is likely not a security concern, but it rubs me the wrong way to do this :)

and:

basic internet security/hygiene, you don't let anyone query your servers for anything without authorisation.

Given how much unauthenticated metrics we already expose in express and in core , I don't see how this is any more or less sensitive. Our precedent seems to be to expose the stats unauthenticated.

but I'm also the only one pushing for this, inside and outside of Medic 😅

[dianabarsan]

I think we should expose if we have a strong case of how it could be useful and have a documented way of troubleshooting based on dashboard data built around this. Without that, it's just noise.

[mrjones-plip]

also, from devs, there's no performance concerns:

the stats are captured internally on an interval, it doesn't matter how many times you query the endpoint really (it's not doing the stats gathering when you call it)

[mrjones-plip]

such a relevant point that we shouldn't add it unless we have a documented way of troubleshooting + dashboard. I'll do some experimentation to see what this looks like before we go any further.

@jkuester or @witash - any input here?

[jkuester]

I don't have any concerns from a security perspective from exposing this data (follows our existing patterns). I will defer to @mrjones-plip and @witash regarding if the metrics give any information that would be useful to track.... 🤷