For more context, look at this issue: grafana/loki#17100
Assume Loki is running with 3 units for each of the read, write, and backend roles and that Squid has been configured as the HTTP/HTTPS proxy.
Also assume that we have ingress available. Hence, we define the compactor's gRPC address like you see below. This is because we set the Compactor's gRPC address to the coordinator's external address as you can see here.
common:
compactor_grpc_address: http://REDACTED/REDACTED
The problem is that by default Squid cannot be a proxy to gRPC requests. Hence, inside the backend's pebble logs, you'll see errors such as:
caller=spanlogger.go:152 component=ruler evaluation_mode=local user=fake level=error msg="failed loading deletes for user" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to do connect handshake, response: \\\"HTTP/1.1 400 Bad Request\\\\r\\\\nConnection: close\\\\r\\\\nContent-Length: 3254\\\\r\\\\nContent-Language: en\\\\r\\\\nContent-Type: text/html;charset=utf-8\\\\r\\\\nDate: Wed, 24 Jun 2026 21:02:19 GMT\\\\r\\\\nMime-Version: 1.0\\\\r\\\\nServer: squid/5.9\\\\r\\\\nVary: Accept-Language\\\\r\\\\nVia: 1.1 REDACTED (squid/5.9)\\\\r\\\\nX-Cache: MISS fromREDACTED\\\\r\\\\nX-Cache-Lookup: NONE from REDACTED:3128\\\\r\\\\nX-Squid-Error: ERR_INVALID_URL 0\\\\r\\\\n\\\\r\\\\n<!DOCTYPE html PUBLIC \\\\\\\"-//W3C//DTD HTML 4.01//EN\\\\\\\" \\\\\\\"http://www.w3.org/TR/html4/strict.dtd\\\\\\\">\\\\n<html><head>\\\\n<meta type=\\\\\\\"copyright\\\\\\\" content=\\\\\\\"Copyright (C) 1996-2020 The Squid Software Foundation and contributors\\\\\\\">\\\\n<meta http-equiv=\\\\\\\"Content-Type\\\\\\\" content=\\\\\\\"text/html; charset=utf-8\\\\\\\">\\\\n<title>ERROR: The requested URL could not be retrieved</title>\\\\n<style type=\\\\\\\"text/css\\\\\\\"><!-- \\\\n /*\\\\n * Copyright (C) 1996-2023 The Squid Software Foundation and contributors\\\\n *\\\\n * Squid software is distributed under GPLv2+ license and includes\\\\n * contributions from numerous individuals and organizations.\\\\n * Please see the COPYING and CONTRIBUTORS files for details.\\\\n */\\\\n\\\\n/*\\\\n Stylesheet for Squid Error pages\\\\n Adapted from design by Free CSS Templates\\\\n http://www.freecsstemplates.org\\\\n Released for free under a Creative Commons Attribution 2.5 License\\\\n*/\\\\n\\\\n/* Page basics */\\\\n* {\\\\n\\\\tfont-family: verdana, sans-serif;\\\\n}\\\\n\\\\nhtml body {\\\\n\\\\tmargin: 0;\\\\n\\\\tpadding: 0;\\\\n\\\\tbackground: #efefef;\\\\n\\\\tfont-size: 12px;\\\\n\\\\tcolor: #1e1e1e;\\\\n}\\\\n\\\\n/* Page displayed title area */\\\\n#titles {\\\\n\\\\tmargin-left: 15px;\\\\n\\\\tpadding: 10px;\\\\n\\\\tpadding-left: 100px;\\\\n\\\\tbackground: url('/squid-internal-static/icons/SN.png') no-repeat left;\\\\n}\\\\n\\\\n/* initial title */\\\\n#titles h1 {\\\\n\\\\tcolor: #000000;\\\\n}\\\\n#titles h2 {\\\\n\\\\tcolor: #000000;\\\\n}\\\\n\\\\n/* special event: FTP success page titles */\\\\n#titles ftpsuccess {\\\\n\\\\tbackground-color:#00ff00;\\\\n\\\\twidth:100%;\\\\n}\\\\n\\\\n/* Page displayed body content area */\\\\n#content {\\\\n\\\\tpadding: 10px;\\\\n\\\\tbackground: #ffffff;\\\\n}\\\\n\\\\n/* General text */\\\\np {\\\\n}\\\\n\\\\n/* error brief description */\\\\n#error p {\\\\n}\\\\n\\\\n/* some data which may have caused the problem */\\\\n#data {\\\\n}\\\\n\\\\n/* the error message received from the system or other software */\\\\n#sysmsg {\\\\n}\\\\n\\\\npre {\\\\n}\\\\n\\\\n/* special event: FTP / Gopher directory listing */\\\\n#dirmsg {\\\\n font-family: courier, monospace;\\\\n color: black;\\\\n font-size: 10pt;\\\\n}\\\\n#dirlisting {\\\\n margin-left: 2%;\\\\n margin-right: 2%;\\\\n}\\\\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\\\\n border-bottom: groove;\\\\n}\\\\n#dirlisting td.size {\\\\n width: 50px;\\\\n text-align: right;\\\\n padding-right: 5px;\\\\n}\\\\n\\\\n/* horizontal lines */\\\\nhr {\\\\n\\\\tmargin: 0;\\\\n}\\\\n\\\\n/* page displayed footer area */\\\\n#footer {\\\\n\\\\tfont-size: 9px;\\\\n\\\\tpadding-left: 10px;\\\\n}\\\\n\\\\n\\\\nbody\\\\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\\\\n:lang(he) { direction: rtl; }\\\\n --></style>\\\\n</head><body id=ERR_INVALID_URL>\\\\n<div id=\\\\\\\"titles\\\\\\\">\\\\n<h1>ERROR</h1>\\\\n<h2>The requested URL could not be retrieved</h2>\\\\n</div>\\\\n<hr>\\\\n\\\\n<div id=\\\\\\\"content\\\\\\\">\\\\n<p>The following error was en
2026-06-24T21:02:22.863Z [loki] countered while trying to retrieve the URL: <a href=\\\\\\\":9095\\\\\\\">:9095</a></p>\\\\n\\\\n<blockquote id=\\\\\\\"error\\\\\\\">\\\\n<p><b>Invalid URL</b></p>\\\\n</blockquote>\\\\n\\\\n<p>Some aspect of the requested URL is incorrect.</p>\\\\n\\\\n<p>Some possible problems are:</p>\\\\n<ul>\\\\n<li><p>Missing or incorrect access protocol (should be <q>http://</q> or similar)</p></li>\\\\n<li><p>Missing hostname</p></li>\\\\n<li><p>Illegal double-escape in the URL-Path</p></li>\\\\n<li><p>Illegal character in hostname; underscores are not allowed.</p></li>\\\\n</ul>\\\\n\\\\n<p>Your cache administrator is <a href=\\\\\\\"mailto:webmaster\\\\\\\">webmaster</a>.</p>\\\\n<br>\\\\n</div>\\\\n\\\\n<hr>\\\\n<div id=\\\\\\\"footer\\\\\\\">\\\\n<p>Generated Wed, 24 Jun 2026 21:02:19 GMT by REDACTED-14 (squid/5.9)</p>\\\\n<!-- ERR_INVALID_URL -->\\\\n</div>\\\\n</body></html>\\\\n\\\"\""
I think the issue specifically is that we're trying to route a gRPC request through an HTTP only Squid proxy. The way I see it, there are two options to fix this.
Not routing through ingress at all
.svc is always part of the NO_PROXY entries, so we could bypass the proxy by NOT setting the compactor's gRPC address to the coordinator's extneral URL (which is always a path based HTTP ingressed address) and instead rely on the app service to route gRPC traffic. A caveat is that we'd need to expose 9095 as well, in addition to 3100. We can add .svc.cluster.local to NO_PROXY list. The compactor will never be called from outside the cluster anyway, so I'm not sure why we even put an ingressed address for it in the config anyway.
Using HTTP
The workload config supports a compactor_address argument, which is the HTTP equivalent of compactor_grpc_address. This option too expects the value provided to be in format host:port, so we could just rely on the FQDN:port for this. This way we don't have to go through ingress (which, due to Traefik, is path based).
For more context, look at this issue: grafana/loki#17100
Assume Loki is running with 3 units for each of the read, write, and backend roles and that Squid has been configured as the HTTP/HTTPS proxy.
Also assume that we have ingress available. Hence, we define the compactor's gRPC address like you see below. This is because we set the Compactor's gRPC address to the coordinator's external address as you can see here.
The problem is that by default Squid cannot be a proxy to gRPC requests. Hence, inside the backend's pebble logs, you'll see errors such as:
caller=spanlogger.go:152 component=ruler evaluation_mode=local user=fake level=error msg="failed loading deletes for user" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: failed to do connect handshake, response: \\\"HTTP/1.1 400 Bad Request\\\\r\\\\nConnection: close\\\\r\\\\nContent-Length: 3254\\\\r\\\\nContent-Language: en\\\\r\\\\nContent-Type: text/html;charset=utf-8\\\\r\\\\nDate: Wed, 24 Jun 2026 21:02:19 GMT\\\\r\\\\nMime-Version: 1.0\\\\r\\\\nServer: squid/5.9\\\\r\\\\nVary: Accept-Language\\\\r\\\\nVia: 1.1 REDACTED (squid/5.9)\\\\r\\\\nX-Cache: MISS fromREDACTED\\\\r\\\\nX-Cache-Lookup: NONE from REDACTED:3128\\\\r\\\\nX-Squid-Error: ERR_INVALID_URL 0\\\\r\\\\n\\\\r\\\\n<!DOCTYPE html PUBLIC \\\\\\\"-//W3C//DTD HTML 4.01//EN\\\\\\\" \\\\\\\"http://www.w3.org/TR/html4/strict.dtd\\\\\\\">\\\\n<html><head>\\\\n<meta type=\\\\\\\"copyright\\\\\\\" content=\\\\\\\"Copyright (C) 1996-2020 The Squid Software Foundation and contributors\\\\\\\">\\\\n<meta http-equiv=\\\\\\\"Content-Type\\\\\\\" content=\\\\\\\"text/html; charset=utf-8\\\\\\\">\\\\n<title>ERROR: The requested URL could not be retrieved</title>\\\\n<style type=\\\\\\\"text/css\\\\\\\"><!-- \\\\n /*\\\\n * Copyright (C) 1996-2023 The Squid Software Foundation and contributors\\\\n *\\\\n * Squid software is distributed under GPLv2+ license and includes\\\\n * contributions from numerous individuals and organizations.\\\\n * Please see the COPYING and CONTRIBUTORS files for details.\\\\n */\\\\n\\\\n/*\\\\n Stylesheet for Squid Error pages\\\\n Adapted from design by Free CSS Templates\\\\n http://www.freecsstemplates.org\\\\n Released for free under a Creative Commons Attribution 2.5 License\\\\n*/\\\\n\\\\n/* Page basics */\\\\n* {\\\\n\\\\tfont-family: verdana, sans-serif;\\\\n}\\\\n\\\\nhtml body {\\\\n\\\\tmargin: 0;\\\\n\\\\tpadding: 0;\\\\n\\\\tbackground: #efefef;\\\\n\\\\tfont-size: 12px;\\\\n\\\\tcolor: #1e1e1e;\\\\n}\\\\n\\\\n/* Page displayed title area */\\\\n#titles {\\\\n\\\\tmargin-left: 15px;\\\\n\\\\tpadding: 10px;\\\\n\\\\tpadding-left: 100px;\\\\n\\\\tbackground: url('/squid-internal-static/icons/SN.png') no-repeat left;\\\\n}\\\\n\\\\n/* initial title */\\\\n#titles h1 {\\\\n\\\\tcolor: #000000;\\\\n}\\\\n#titles h2 {\\\\n\\\\tcolor: #000000;\\\\n}\\\\n\\\\n/* special event: FTP success page titles */\\\\n#titles ftpsuccess {\\\\n\\\\tbackground-color:#00ff00;\\\\n\\\\twidth:100%;\\\\n}\\\\n\\\\n/* Page displayed body content area */\\\\n#content {\\\\n\\\\tpadding: 10px;\\\\n\\\\tbackground: #ffffff;\\\\n}\\\\n\\\\n/* General text */\\\\np {\\\\n}\\\\n\\\\n/* error brief description */\\\\n#error p {\\\\n}\\\\n\\\\n/* some data which may have caused the problem */\\\\n#data {\\\\n}\\\\n\\\\n/* the error message received from the system or other software */\\\\n#sysmsg {\\\\n}\\\\n\\\\npre {\\\\n}\\\\n\\\\n/* special event: FTP / Gopher directory listing */\\\\n#dirmsg {\\\\n font-family: courier, monospace;\\\\n color: black;\\\\n font-size: 10pt;\\\\n}\\\\n#dirlisting {\\\\n margin-left: 2%;\\\\n margin-right: 2%;\\\\n}\\\\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\\\\n border-bottom: groove;\\\\n}\\\\n#dirlisting td.size {\\\\n width: 50px;\\\\n text-align: right;\\\\n padding-right: 5px;\\\\n}\\\\n\\\\n/* horizontal lines */\\\\nhr {\\\\n\\\\tmargin: 0;\\\\n}\\\\n\\\\n/* page displayed footer area */\\\\n#footer {\\\\n\\\\tfont-size: 9px;\\\\n\\\\tpadding-left: 10px;\\\\n}\\\\n\\\\n\\\\nbody\\\\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\\\\n:lang(he) { direction: rtl; }\\\\n --></style>\\\\n</head><body id=ERR_INVALID_URL>\\\\n<div id=\\\\\\\"titles\\\\\\\">\\\\n<h1>ERROR</h1>\\\\n<h2>The requested URL could not be retrieved</h2>\\\\n</div>\\\\n<hr>\\\\n\\\\n<div id=\\\\\\\"content\\\\\\\">\\\\n<p>The following error was en 2026-06-24T21:02:22.863Z [loki] countered while trying to retrieve the URL: <a href=\\\\\\\":9095\\\\\\\">:9095</a></p>\\\\n\\\\n<blockquote id=\\\\\\\"error\\\\\\\">\\\\n<p><b>Invalid URL</b></p>\\\\n</blockquote>\\\\n\\\\n<p>Some aspect of the requested URL is incorrect.</p>\\\\n\\\\n<p>Some possible problems are:</p>\\\\n<ul>\\\\n<li><p>Missing or incorrect access protocol (should be <q>http://</q> or similar)</p></li>\\\\n<li><p>Missing hostname</p></li>\\\\n<li><p>Illegal double-escape in the URL-Path</p></li>\\\\n<li><p>Illegal character in hostname; underscores are not allowed.</p></li>\\\\n</ul>\\\\n\\\\n<p>Your cache administrator is <a href=\\\\\\\"mailto:webmaster\\\\\\\">webmaster</a>.</p>\\\\n<br>\\\\n</div>\\\\n\\\\n<hr>\\\\n<div id=\\\\\\\"footer\\\\\\\">\\\\n<p>Generated Wed, 24 Jun 2026 21:02:19 GMT by REDACTED-14 (squid/5.9)</p>\\\\n<!-- ERR_INVALID_URL -->\\\\n</div>\\\\n</body></html>\\\\n\\\"\""I think the issue specifically is that we're trying to route a gRPC request through an HTTP only Squid proxy. The way I see it, there are two options to fix this.
Not routing through ingress at all
.svcis always part of theNO_PROXYentries, so we could bypass the proxy by NOT setting the compactor's gRPC address to the coordinator's extneral URL (which is always a path based HTTP ingressed address) and instead rely on the app service to route gRPC traffic. A caveat is that we'd need to expose 9095 as well, in addition to 3100. We can add.svc.cluster.localtoNO_PROXYlist. The compactor will never be called from outside the cluster anyway, so I'm not sure why we even put an ingressed address for it in the config anyway.Using HTTP
The workload config supports a
compactor_addressargument, which is the HTTP equivalent ofcompactor_grpc_address. This option too expects the value provided to be in formathost:port, so we could just rely on the FQDN:port for this. This way we don't have to go through ingress (which, due to Traefik, is path based).