deny_paths config enhancements

[alexdlukens]

Enhancement Proposal

Feature requests for enhancement to the "deny_paths" configuration on the haproxy-ddos-protection-configurator charm

1. Per-site deny_paths support. Currently deny_paths is applied to all hosts haproxy serves traffic for
2. Configurable deny behavior akin to limit-policy-http. In cases where we want traffic to only be allowed from specific subnets, we will set ip-allow-list accordingly and want to silent-drop or deny-503 traffic from other subnets.

Impact

High (The feature has short-term technical value)

Impact Rationale

HAProxy is used in the PS7 ingress solution.

Here, deployments serve backends for many domains. In these situations it is difficult to use deny_paths config as this applies to all served domains. It would be preferable to set deny_paths per-domain, so that this configuration is not used on some sites, and enabled on differing paths per-site.

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/ISD-5204.

This message was autogenerated

[jsimpso]

1. Per-site deny_paths support. Currently deny_paths is applied to all hosts haproxy serves traffic for

If the scope of haproxy-ddos-protection-configurator is to apply settings to the HAProxy instance as a whole, perhaps ingress-configurator could be enhanced to provide per-service deny_paths and ip-allow-list settings for more fine-grained control?

[seb4stien]

Could you please provide the use case for this feature?

The current model is that DDOS is a global configuration to protect the infrastructure, and that the equivalent of "deny_paths" and "ip-allow-list" features for a given service would/should be implemented by the backend application. Is there anything preventing the backends to do their own access control?

[alexdlukens]

As @jsimpso mentions, if HAProxy ACLs can be exposed in ingress-configurator on the haproxy-route{-tcp} interfaces, I believe this would be preferable and more flexible than my initial proposal

Could you please provide the use case for this feature?

We want to be able to control ACLs and deny behavior per-backend on HAProxy deployments. Specifically we want to filter/restrict traffic sent to a backend based on source IP address. The current implementation in the ddos-configurator charm lets us do this, but not at a per-backend level.

E.g. we want to:

backend defined_restricted_service
    acl allowed_ips_restricted_service src -f /var/lib/haproxy/ip_allow_list_restricted_service.lst
    http-request allow if allowed_ips_restricted_service

    acl denied_paths path_beg -f /var/lib/haproxy/deny_paths_restricted_service.lst
    http-request deny if denied_paths_restricted_service


backend second_restricted_service
    # different ip allow list for this backend
    acl allowed_ips_second_restricted_service src -f /var/lib/haproxy/ip_allow_list_second_restricted_service.lst
    http-request allow if allowed_ips_second_restricted_service

    # different deny paths for this backend
    acl denied_paths path_beg -f /var/lib/haproxy/deny_paths_second_restricted_service.lst
    http-request deny if denied_paths_second_restricted_service

The current model is that DDOS is a global configuration to protect the infrastructure, and that the equivalent of "deny_paths" and "ip-allow-list" features for a given service would/should be implemented by the backend application.

In the case where a HAProxy instance serves multiple sites, how is it intended to use the DDOS configurator's deny_paths in the current configuration then? This would need to be setup with full knowledge of all served paths by all services on the HAProxy instance to avoid unintentionally blocking access to a path. This could lead to confusion when services are added in the future, as the paths would be blocked there as well.

Or if an application is served by HAProxy -> traefik-k8s, then the path for api endpoints could be https://<hostname>/<model-name>-<app-name>/api and not https://<hostname>/api, which would be hard to pre-emptively deny traffic to using the current deny_paths configuration. This situation makes more sense if ACLs/deny_paths configuration is exposed in ingress-configurator, where it could be configured per-service.

Is there anything preventing the backends to do their own access control?

The ingress interface does not expose access to HAProxy ACLs, so it is not possible to configure this from the backend application without implementing allowlists/blocking traffic some other way. Perhaps using iptables or in traefik-k8s.

In this case, it makes the most sense to me to design this feature once in HAProxy/ingress-configurator instead of for each application that has an ingress interface. Implementing in HAProxy/ingress-configurator would let us control this as administrators without further implementation from client applications.

It is also preferable to implement this in HAProxy, as we use hardware-offloaded networking for the PS7 ingress solution using HAProxy. Restricting traffic here is desireable, as backing applications would not be able to serve similar volumes of traffic.

[hk21702]

If my understanding is correct, I think adding a per-service option via ingress-configurator could be a good alternative to the initial proposal with enhancing haproxy-ddos-protection-configurator, especially since we then get control of this from the consuming model.

To add on an even more specific use case specific to our infra, PS7 presently doesn't have a viable solution to expose deployments to an allowlist set of IPs, for instance, to allow access to internal deployments via only a VPN connection. This has been a pretty commonly requested setup. For myself as well as others in my team, it's highly desirable to have internal-only staging deployments of otherwise public production deployments.

We could maybe have a dedicated type of ingress on PS7 with haproxy-ddos-protection-configurator configured to achieve this, but then we'd need another ingress per team to achieve this, at least based on my current understanding of how ingresses are done on PS7.

Allowing this to be set per route instead would be easier since we can use the same ingress, and it's easier on the individual application development side.

On the mention of traefik-k8s, I don't think the charm itself right now has the ability to restrict IPs via just configuration. You could achieve it on the application side with a Traefik route interface, but that means that we're touching charm code for a deployment issue, and it also just complicates the application development, restricting us to using Traefik-route. Considering how common this use case seems to be, it would be easier to just be able to do this by configuring HAProxy in some way.

[seb4stien]

Hi. Some answers:

• "how is it intended to use the DDOS configurator's deny_path" <- the DDOS configurator audience is not the teams behind HAProxy, it's a tool for the cloud provider to take emergency measures if something turns bad. It's a way to block traffic from a attacker IP or a worm scanning a given path for all the apps hosted behind HAProxy. Same for the rate limiting, this is to be used in case of emergency if we are under attack.

• Regarding "ip-allow-list". We discussed about it with IS leadership and this is not a feature we plan to implement on ingresses for now as the main use case seems to be to restrict access to VPN IPs, but VPN are supposed to disappear. IS recommended to have a discussion with them first to see how they can best support your use case, and if something needs to be added to HAProxy we can of course support afterwards.

[alexdlukens-canonical]

It has been a while since this issue was opened. An update from IS team: @jsimpso has used the existing functionality from the DDOS configurator charm to implement an internal-only ingress solution for PS7 and microcloud.

The specific use case driving the internal-only ingress solution was the need to restrict IPs for handling ingress for COS / opentelemetry collector data forwarding. This is a separate usecase from the VPN IP restriction.

I still believe the features discussed here would be beneficial, but if they are not planned this issue can be closed.