Need help with metrics

[KoNEW]

Hi, need some help with understanding of Granian work logic

• Project deployed as granian --host 0.0.0.0 --workers 3 --interface asgi myproject.asgi:application --backlog 1024 --metrics --metrics-address 0.0.0.0
• Project uses both - simple HTTP requests and web-socket connections
• Metrics collected in prometheus
• I have built a graph based on requests_handled metric with PromQL query: rate(granian_requests_handled{instance=~"$instance"}[$__rate_interval]), graph from Grafana in attachments

I can't understand 2 things:

• nature of such changes on graph when some workers stop handling requests and rate become zero
• why distribution of requests between workers is not uniform

[gi0baro]

Is granian sitting behind a proxy/load balancer with keep-alive connections?

[KoNEW]

Yes, in front of traffic sitting farm of NGINX servers on independent nodes with keep-alive configured, this farm downstream traffic to working nodes at 8000 port

Configurations looks like:

http
{
	# main configs
	include /etc/nginx/mime.types;
	default_type application/octet-stream;

	sendfile                    on;
	tcp_nopush                  on;
	tcp_nodelay                 on;
	keepalive_timeout           600;
	types_hash_max_size         2048;
	server_tokens               off;
	charset                     utf-8;
	client_max_body_size        5060M;
	large_client_header_buffers 8 128k;
	...
}
...
map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}
upstream api {
    server 10.0.0.1:8000;
    server 10.0.0.2:8000;
    server 10.0.0.3:8000;
    server 10.0.0.4:8000;
}
server {
...
  location /api {
      proxy_request_buffering off;
  
      proxy_set_header    Host                $http_host;
      proxy_set_header    X-Real-IP           $remote_addr;
      proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
      proxy_set_header    X-Forwarded-Proto   $scheme;
      proxy_set_header    Upgrade             $http_upgrade;
      proxy_set_header    Connection          $connection_upgrade;
      proxy_redirect      off;
      proxy_http_version  1.1;
      proxy_pass          http://api;
  }
|

[gi0baro]

Then the non-uniform distribution of requests across workers is normal: for that number of RPS, the connection opened by nginx should be quite low, and if those connections are kept alive, then they stick with first worker that processed them. Using low backpressure values might help distributing connections more evenly, but if that number is too low then you risk to stop accepting connections completely. I guess you can tune the value depending on what you see from the connections_active metric, and also tune the keepalive settings in nginx.

As for the rate suddenly going to zero, I don't have a good explanation, but it might depends as well on nginx closing some of the keep-alive connections due to max lifetime.