Cherry pick stable/2026-1.m3 commits - #123
Open
sven-rosenzweig wants to merge 77 commits into
Open
sven-rosenzweig wants to merge 77 commits into
sven-rosenzweig wants to merge 77 commits into
Conversation
In the ``neutron-ovn-db-sync-util``, the ``AgentCache`` singleton is created but never populated because the OVSDB monitor events that fill it in the Neutron API (``ChassisAgentWriteEvent``, ``ChassisOVNAgentWriteEvent``) are not registered; the db-sync ``OVNMechanismDriver`` overrides ``post_fork_initialize`` as a no-op. This results in an empty ``AgentCache`` when ``OvnSbSynchronizer`` methods like ``_get_hosts_mapped`` query it, making the sync unable to correctly determine which hosts have OVN controller agents. This patch adds ``AgentCache.populate()`` which reads all ``Chassis_Private`` rows from the SB database and creates the corresponding agent entries, replicating the logic of the monitor events. Call it in ``OvnSbSynchronizer.__init__`` so any sync method sees a fully populated cache. Conflicts: neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py Closes-Bug: #2150109 Related-Bug: #2136864 Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Change-Id: If7bec416a9c9f5ed6cd8a04775606092dde2390a (cherry picked from commit 606ef2a)
Neutron's Designate external DNS driver does not pass edit_managed=True when constructing the admin designateclient. When a floating IP has associated managed DNS records in Designate, deleting the floating IP fails with HTTP 500 because Designate rejects the deletion with "Managed records may not be deleted". Pass edit_managed=True only to the admin_client, which has admin credentials and passes Designate's edit_managed_records policy check. The regular user client must not set this flag because Designate requires SYSTEM_ADMIN role for it. For forward DNS deletion of managed records, fall back to admin_client when the user client gets a BadRequest from Designate. Closes-Bug: #2149807 Assisted-By: Claude Code 4.6 Opus Change-Id: Iedf0f26708560fe063a71925d5d39542ca181154 Signed-off-by: Omer <oschwart@redhat.com> (cherry picked from commit 44d061a)
The TaggingController.create() and update() methods enforce policy action names using the plural collection key (e.g. create_networks:tags) instead of the singular member name (e.g. create_network:tags). Since the registered policy rules use the singular form, the unmatched plural names fall through to oslo.policy's default rule, allowing project readers to mutate tags on same-project resources. Fix the delete_floatingips:tags policy rule name (should be singular delete_floatingip:tags) and add a unit test that validates _get_policy_action produces the correct singular form for all supported resources and actions, and that each generated name matches an actually registered policy rule. Closes-Bug: #2150132 Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Change-Id: I783510565e4fc4191b5494eb9a6dc0bdd3ace3fc (cherry picked from commit 7401244)
The ``OVNL3RouterPlugin`` class registers a set of OVN events in the ``_post_fork_initialize`` method. The event processing should be done only by Neutron API workers (aka, ``WorkerService`` instances). Conflicts: neutron/services/ovn_l3/plugin.py Closes-Bug: #2152562 Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Change-Id: I206cd34b1d5559d52c8e841006cf4d6e1eb361ef (cherry picked from commit 131d985)
Verify the port-forwarding floating IP id matches the child floating IP id so we satisfy the policy requirements for GET and PUT operations. Closes-bug: #2150121 Change-Id: Ie68e57da6222965e79d015a5568190d8bc29b9e8 Signed-off-by: Brian Haley <haleyb.dev@gmail.com> Assisted-by: Claude Sonnet 4.6
The _is_vlan_project_network method in DNSExtensionDriverML2 was only checking the static network_vlan_ranges configuration from ml2_conf.ini, ignoring the dynamic ranges provided by the network-segment-range service plugin. This caused a KeyError when creating subnets or VMs on VLAN networks with segment ranges managed via the API. This fix updates the method to use VlanTypeDriver.get_network_segment_ranges(), which properly handles both static configuration and DB-based dynamic ranges when the network-segment-range plugin is enabled. Closes-Bug: #2140291 Signed-off-by: Kyuyeong Lee <kyu0.lee@samsung.com> Change-Id: I8f3a2b1c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a (cherry picked from commit cd8f88e)
Several default port policies that require network ownership incorrectly included PROJECT_MANAGER. That rule checks the port project_id, not network ownership, so any project manager could perform those actions on shared/RBAC networks where they do not own the network. Remove PROJECT_MANAGER from the affected create/update port policies and rely on NET_OWNER_MEMBER or ADMIN_OR_NET_OWNER_MEMBER instead. Project managers who own the network remain authorized through the default Keystone role implication chain (manager implies member). Closes-Bug: #2152115 Assisted-By: Claude Composer 2.5 Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Change-Id: I4e258d28cdf72adcc13fc9d03749256c65881c45 (cherry picked from commit d0f61db)
The ``OVNL3RouterPlugin`` registers OVN OVSDB monitor events during ``_post_fork_initialize``. That callback is also invoked from neutron-periodic-workers and other non-API processes, where the OVN IDL connections are not initialized. Checking OVN readiness before filtering by worker type raised ``MechanismDriverOVNNotReady`` and blocked worker startup. Return early when the trigger is not a Neutron API ``WorkerService``, and only then validate OVN readiness and register monitor events. Conflicts: neutron/services/ovn_l3/plugin.py Related-Bug: #2154192 Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Change-Id: I9e43fdb06ce7868e5fa699657d8da9a4aa4d02fb (cherry picked from commit b6d260e)
The ``_is_tunnel_project_network`` method in ``DNSExtensionDriverML2`` was only checking the static tunnel ranges from ml2_conf.ini, ignoring the dynamic ranges provided by the ``network-segment-range service`` plugin. When segment ranges are managed via the API, external DNS eligibility for tenant tunnel networks could be wrong. Update the method to use the ML2 tunnel type driver's ``get_network_segment_ranges()``, which handles both static configuration and DB-based ranges when the network-segment-range plugin is enabled. Closes-Bug: #2154266 Related-Bug: #2140291 Assisted-By: Cursor Composer 2.5 Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Change-Id: I5d96efa626d18b7589cf2563fdac3a8d399a69a8 (cherry picked from commit 6913303)
The openvswitch firewall driver has got OF rules to mark some connections as invalid using mark CT_MARK_INVALID in conntrack. Later packets which belongs to such invalid connection are dropped by different OF rules in br-int. The problem is that such conntrack entries were cleaned for each of the ports for which filters were updated, even with deferred apply on. In such case new OpenFlow rules are applied in deferred way after all ports are processed by the agent. That could lead to the situation where conntrack entries to mark some kind of connection as invalid was deleted by the fw driver, but new OF rules were not yet applied in the br-int. If during that time the same packets were still flowing to/from the port, new conntrack entry to mark such packets would be added back and that would not be cleaned after OF rules were actually applied on the `br-int` bridge. This patch fixes that issue be deferring cleanup of the conntrack entries for all ports to be done after OF rules are actually updated in `br-int`. Assisted-by: claude-opus-4.6 Closes-bug: #2154561 Change-Id: I8655da7f7a0cf033ce4a7145f5345195881238e5 Signed-off-by: Slawek Kaplonski <skaplons@redhat.com> (cherry picked from commit 8fde7d4)
Ports with mac=['unknown'] (e.g. network:dhcp ports without port security) cause the metadata agent to crash with a ValueError when iterating chassis ports during datapath provisioning. Add an explicit check for mac=['unknown'] alongside the existing empty-MAC guard to skip these ports gracefully. Closes-Bug: #2150434 Assisted-By: Claude Opus 4.6 Change-Id: I966c7a22eca589770793a0ef30c3bbc591c41e41 Signed-off-by: Eduardo Olivares <eolivare@redhat.com> (cherry picked from commit fecac2c)
Singleton conntrack helper API operations (GET/PUT/DELETE on
/routers/{router_id}/conntrack_helpers/{id}) authorized the request
against the URL ``router_id`` but loaded the helper by child ID only.
A project member with access to one router could therefore operate on
another tenant's helper by reusing its UUID in the path.
Validate that the loaded helper's ``router_id`` matches the URL
``router_id`` before returning, updating, or deleting it, following
the same pattern used for floating IP port forwarding sub-resources.
Closes-Bug: #2152109
Assisted-By: Claude Opus 4.6
Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com>
Change-Id: Icb4a7d2e8f1c35609d8a4b7e2f6c9d0a1b3e5f78
(cherry picked from commit bc8a554)
The operation for these actions is PUT, not POST. Closes-Bug: #2156054 Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Change-Id: Iec4b8ddf3717ddc781acfb46ada81839f853bdea (cherry picked from commit 8ccf3f9)
When a VIF port is picked up by the OVS agent ``rpc_loop`` before OVS has assigned it a valid ofport (the underlying TAP device may not yet exist), ``port_alive()`` and ``port_dead()`` pass the invalid value (``[]`` or ``-1``) through to ``uninstall_flows()`` / ``drop_port()``. The resulting OpenFlow FlowMod with ``in_port=None`` causes os-ken's ``send_msg`` to hang until the ``of_request_timeout`` (300 s) fires, blocking the ``rpc_loop`` for the entire duration and preventing all subsequent port processing. Guard ``port_alive()``, ``port_dead()`` and ``treat_vif_port()`` so they return early when the ofport is unassigned or invalid. ``treat_vif_port()`` returns False so the port is not marked as bound; the OVSDB monitor will detect the ofport change on a later iteration and re-trigger processing. Closes-Bug: #2155883 Assisted-By: Claude Opus 4.6 Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Change-Id: Ic137f8a2862794c3c7ac670e643ca532873e474b (cherry picked from commit 8a4a4b1)
When a port is inserted into OVS, the kernel may transiently assign ``ofport=-1`` (``INVALID_OFPORT``) before settling on a valid value. Previously, ``_process_port`` only deferred ports with ``UNASSIGNED_OFPORT`` (``[]``), so a port arriving with ``ofport=-1`` was immediately added to the processing pipeline. Patch [1] then correctly skipped OF operations in ``treat_vif_port`` for such ports but never re-queued them, which meant the ``network-vif-plugged`` event was never sent to Nova, causing a 300 s VIF-plug timeout. Extend the ``_process_port`` check to also defer ``INVALID_OFPORT`` ports via the existing ``ports_not_ready_yet`` mechanism. On the next ``rpc_loop`` iteration the port attributes are re-read from OVS; if the ofport is now valid the port is processed normally, otherwise it is deferred again. [1]https://review.opendev.org/c/openstack/neutron/+/992423 Closes-Bug: #2155883 Assisted-By: Claude Opus 4.6 Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Change-Id: I9fbc8f75f8084461901a004dfcb07e42f76db62b (cherry picked from commit 7ebcce0)
When a service plugin package (e.g. neutron-fwaas, neutron-vpnaas,
networking-sfc) is installed but its service plugin is not configured
in ``service_plugins``, the quota details API endpoint
(GET /v2.0/quotas/{project_id}/details) returns a 500 Server Error.
The installed package registers quota resources (e.g. firewall_group,
firewall_policy, firewall_rule) at import time via
``resource_helper.build_resource_info(register_quota=True)``. When the
quota details endpoint iterates over all registered resources to
count usage, it calls ``_count_resource()`` which looks for a plugin
that provides ``get_<collection>_count`` or ``get_<collection>``.
Since the service plugin is not loaded, no plugin supports counting
those resources, and a ``NotImplementedError`` is raised.
Catch the ``NotImplementedError`` in
``DbQuotaDriver.get_detailed_project_quotas()`` and skip the resource
instead of letting the exception propagate as a 500 error. Also
guard the project-specific limit update loop against skipped
resources.
Closes-Bug: #2155846
Assisted-By: Claude Opus 4.6
Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com>
Change-Id: I923e90279edf3de3fa85c83fd46e1b5dec0468de
With tunneling disabled (enable_tunneling=False), there is no br-tun and self.tun_br is None. _restore_local_vlan_map(), called from __init__(), calls self.tun_br.get_flood_to_tun_ofports() unconditionally for every integration-bridge port that carries a net_uuid, so the agent crashes: AttributeError: 'NoneType' object has no attribute 'get_flood_to_tun_ofports' The branch is only reached once the agent has populated net_uuid on the ports, so the crash hits on agent restart and recurs on every restart, preventing the agent from restoring dataplane flows until manual recovery. Skip the flood-port restore when tunneling is disabled, leaving tun_ofports as an empty set as for ports with no tunnel flood entries. Closes-Bug: #2156566 Change-Id: I4741480f8a7fbce51f521579e78d5687773c29b8 Signed-off-by: Seyeong Kim <seyeong.kim@canonical.com> (cherry picked from commit 92181b3)
Increased ``allowed_network_downtime`` to 7.0 in the ``neutron-ovs-tempest-slow`` CI job. That will improve the stability of test ``test_server_connectivity_live_migration``. Closes-Bug: #2157882 Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Change-Id: Idad087b2a0abd76d1297882011bc7deb6924c8ca (cherry picked from commit 4aabb3e)
Python's `requests` lib is always asking for 'gzip,deflate' encoding by itself, and decoding those by itself too, but does not remove the content-encoding header from response, so don't rely on this header alone to decide if content should be decoded or not. Also, content-length and transfer-encoding are left untouched, even if they correspond to originally gzip-ed content. This all interferes when user data is itself gzip-ed and there's a proper web server like apache2 that itself generates proper gzip-ed response. Clean-up erroneous headers, and let webob recalculate them as needed. Related-Bug: #2120723 Closes-Bug: #2156587 Change-Id: Ia5d0cbdc5c715462438e38135e441ce9f5ea98c1 Signed-off-by: Pavlo Shchelokovskyy <shchelokovskyy@gmail.com> (cherry picked from commit d70bdc9)
Seeing random failures for MacAddressInUse in unit tests, extending the range to lower such conflicts. Closes-Bug: #2158374 Signed-off-by: Yatin Karel <ykarel@redhat.com> Change-Id: I24f429adfcdbf7411663459998c36479aef92350 (cherry picked from commit 2683e23)
When a subnet is deleted before its segment, ``_delete_nova_inventory`` is called twice for the same resource provider. The second call now catches ``PlacementResourceProviderNotFound`` at DEBUG level instead of logging a misleading INFO message. Closes-Bug: #2158118 Assisted-By: Claude Opus 4.6 Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Change-Id: Icf643cc9dff81ce65e26fe0bc704942d4772400d (cherry picked from commit 2856ae5)
When multiple Neutron API workers concurrently create or update external ports on the same network, they race to create the same ``HA_Chassis_Group``. The ``ha_chassis_group_add`` command uses ``may_exist=True``, but this check relies on the local IDL cache which can be stale, causing duplicate inserts and OVSDB constraint violations. The previous code called ``sync_ha_chassis_group_network()`` passing the parent transaction. Because ovsdbapp merges nested transactions on the same thread (``nested=True`` by default), the HA Chassis Group creation ended up in the same transaction as the port creation, making it impossible to retry independently. Fix this by: 1. Passing ``txn=None`` to ``sync_ha_chassis_group_network()`` in ``create_port()`` and ``update_port()`` so that ``ovn_context`` manages its own transaction. 2. Using ``nested=False`` in ``_run_with_retry()`` to force a truly independent OVSDB transaction that is not merged with the outer port transaction. 3. Retrying on ``RuntimeError`` (constraint violations) with ``tenacity``, giving the IDL cache time to catch up between attempts. Closes-Bug: #2158358 Assisted-By: Claude Opus 4.6 Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Change-Id: I9426e6bc508556a2bfe82e5c384ec07701fd18fb (cherry picked from commit 70394b3)
We switched these jobs to python3.12 long back but recently it was noticed the job was still running with python3.9 during tox venv prep. The job failed after more-itertools update[1] as the newer version is not compatible with python3.9. This patch switches venv prep to also run with python3.12. [1] https://review.opendev.org/c/openstack/requirements/+/983340 Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/984404 Signed-off-by: Yatin Karel <ykarel@redhat.com> Change-Id: Iecdf6b45c070711a3448e7bade3577c340c291cc (cherry picked from commit a5420f5)
When searching for an LRP to find its MAC when implementing openflow rules for a BGP bridge, we need to consider only local chassis. Closes-Bug: #2146077 Change-Id: If070b7c9fcf54db14b93f422f5cbd9a08a4ea696 Signed-off-by: Jakub Libosvar <libosvar@redhat.com> (cherry picked from commit b197e46)
RPC workers do not need to process OVN database notification events such as port status updates or chassis agent events. These events are only relevant to the API workers that handle the OVSDB monitoring loop. When an ``RpcWorker`` creates its OVN IDL connections via ``from_worker()``, use ``BaseOvnIdl`` and ``BaseOvnSbIdl`` instead of the monitoring IDL classes (``OvnNbIdl``, ``OvnSbIdl``). This avoids registering unnecessary event watchers in RPC workers, reducing overhead and preventing potential event handling errors. Closes-Bug: #2158611 Assisted-By: Claude Opus 4.6 Signed-off-by: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> Change-Id: I99e4f116678b7da35ba8ab19059af342ca4858e9 (cherry picked from commit e6a0f5a)
Migration plugins registered under the ``neutron.ovn.db_migration`` entry point are plain callables (functions), not subclasses of ``BaseOvnDbSynchronizer``. The shared ``_load_drivers`` helper applies an ``issubclass`` guard that is only valid for class-based sync plugins, causing a ``TypeError`` when loading function-based migration plugins and ultimately a ``NoMatches`` exception that aborts the OVS-to-OVN database migration step. Fix this by giving ``load_db_migration_drivers`` its own ``EnabledExtensionManager`` construction that skips the ``issubclass`` check and simply filters by driver name. Closes-Bug: #2160210 Assisted-By: Claude Sonnet 4.6 Change-Id: I5c9b689558a9553ff6968868b43bf3d9239c92ad Signed-off-by: Michal Nasiadka <mnasiadka@gmail.com> (cherry picked from commit c615e18)
Out of convenience for running the tests locally, we should by default pick our own "upper-constraints.txt" and not upstream's.
Upstream Neutron has the assumption that all subnets of a network are in the same subnet pool (and therefore same address scope), while our brand of Neutron requires all subnets in a network to be either in the same address scope or in no address scope at all. Neutron adds the address scope to each port in the l3 router sync data, but looks at all subnets present in the network. We now change this behavior to only add the address scope when the router port is actually in a subnet with this address scope. With this DAPNets (Directly Accessible Private Networks) and non-DAPNets can live in peace together. add missing key device_owner in test test__populate_ports_for_subnets_mixed_address_scopes For populating subnets for ports, it is checked whether the port is owned by a router. Runing the test with neutron yoga requires the key device_owner otherwise the test will fail with a KeyError.
Upgrading from openstack version Caracal to Gazpacho, the location, where the python flag is being honored, changes. This is due to the upstream eventlet removal. rewrite import due to pylinter error - C0415: Import outside toplevel
… by SIGWINCH Upgrading openstack from version ussuri to yoga, the _get_application_ was removed. In order to enable guru mediation reporting, triggered by SIGWINCH, the setup moves to _eventlet_api_server. The removal goes back to [upstream commit](609508b). With [commit](a7c4426) guru mediation reporting was introduced in ussuri.
We want to get a better impression of what our users do with our
metadata service and for this we want to improve the logging. The
current logging logs part of the request (part by eventlet.wsgi, part by
Neutron itself) or the complete ports request ("which ports do belong to
the requester) as a multiline string representation of the request
object.
To make debugging this easier for us we now log an extra single line for
each request, containing many informations, as the current OpenStack
network, client ip, user-agent, request path etc., which can then be
easily consumed by some log parser (fluentd + grok in our case).
When sending an OPTIONS request to the metadata agent this request gets proxied to Nova, which then returns an empty HTTP response with content-length: 0 and no content-type key. This results in a KeyError, when we try to access this attribute. Using get() solves this and results in sending no content-type header to the client who requested this. In our environment this can be reproduced with: curl http://169.254.169.254/lol -X OPTIONS
When the user of our metadata service provides an X-Forwarded-For header in the request it will be forwarded to the neutron metadata service by haproxy, alongside with haproxy's own X-Forwarded-For header. On Neutron side this is then processed by eventlet.wsgi, which merges all values of headers that appear multiple times (separated by ','). The result of the merged addresses is then used by neutron in _get_instance_and_tenant_id(), where it is converted to an netaddr.IPAddress. As this is not a valid value, the AddrFormatError exception is raised. We do not want this exception to be raised, but even before that we don't want the user of a metadata service to be able to manipulate the X-Forwarded-For header in any way. Therefore we are now deleting this header in the haproxy frontend.
The OwnerCheck makes it possible to define rules in the policy that allow the owner of a network to see all ports in the network - even if they belong to another project that has the network shared into it. To be able to do this policy check, OwnerCheck queries out the related Network object e.g. when listing ports and caches the result for 5s by default. Since the project_id/tenant_id of an object - or at least of a Network - does not change during its lifetime, expiring the cache this often will lead to more load on the DB without much benefit - we only get at small amount of RAM back. Therefore, we allow administrators to increase the expiration time with the config option `owner_check_cache_expiration_time`. Administrators should check if their policy only uses static attributes and if they do not have a huge amount of those objects when increasing this setting.
When multiple subnets are configured in the same network, the DHCP agent will choose the first suitable subnet with a gateway ip set as default gateway, where the subnets are in a generally stable, but arbitrary order. In some cases only some of the subnets in a network actually have a router in it, meaning a DHCP agent might loose Internet access (used for example for DNS). Now we sort the subnets before selecting a default gateway. Subnets are sorted first by if their gateway ip is present on a port in the network and then by their created_at time, so we always have a stable order. Also, using the first created subnet seems like a good decision, as it will probably be the one with the least fluctuations. We won't filter for a router port, as the user could also provide a custom router via a VM in their network. In case no subnet is present on a port we will still use one of them as default route, just to preserve the old behavior.
External subnets might have an external gateway IP outside of the subnets allocation pool. This IP is not allocated by the OpenStack IPAM, but is used as default gateway. A user might create a floating IP for this specific IP, which results in double usage of the IP. To prevent this we disallow allocating gateway IPs of external subnets as floating IPs. This commit only checks that on FIP creation the IP is not a gateway IP of any subnet it is allocated inside. We do not check this condition if the subnet's gateway IP is modified. This bug has been discussed with upstream, but was found to be invalid by them, because in the default configuration only admins can create a FIP with a predetermined IP address. For us this is different, but additionally I think we also shouldn't allow admins to do this, as in our infra this brings the network down. Discussion with upstream can be found here: https://bugs.launchpad.net/neutron/+bug/1959699
1. Motivation and goals The motivation is to assist the upstream DNS resolvers in identifying the clients using the neutron dnsmasq resolvers. The actual goal is monitoring of the DNS traffic for suspicious activity and uncovering compromised systems. 2. Implementation This is done by passing each client's IP address and network id to the upstream resolvers by adding that information as eDNS payload to the DNS query going out. If this boolean option is toggled then the following two command line options will be appended to dnsmasq: --add-cpe-id=<network_id> --umbrella The former will add the network id to eDNS option 65074. The latter will add the client IP to eDNS option 20292. 3. Caveat The `--umbrella` option was introduced in dnsmasq 2.86. A dnsmasq version check is in place to make sure the option does not get activated if the currently installed version is older than that. In this case a warning will be logged and the fingerprinting will be reduced to the network id only.
When we have a network with stateful DHCPv6 we still need router advertisements to tell the hosts the prefix length of the current network (else all hosts will just have /128 routes). In cases where a network does not have a router (or don't use an l3 plugin with ra support) we do not have something in the network that sends out ras (radvd itself is launched on the l3 agent with Neutron's default implementation). Therefore we now let dnsmasq take over this task. With --enable-ra we enable dnsmasq's ra implementation. We use --ra-param=$iface,0,0 to deactivate that dnsmasq is advertising itself as default gateway. Params are (interface, use default ra-interval, router lifetime).
Currently it is possible to delete a active binding via the port binding API extension. Deleting a active port binding leaves the port in a state of not being editable anymore. Each attempt editing the port results in a PortNotFound error. During a nova live migration a port entered the state described above. For some reasons nova was not able to set the binding of the new target host to active, thus the live migration rollback process started. During the rollback, the newly created binding got deleted (apparently already set to active) leaving the port with no active binding. Thus setting old binding information was not possible anymore. Fixing such a zombie requires to set the status of a binding to active (either via db directly or port binding extension)
It can happen, that we clean up a network segment while there's already another port binding in progress. That port binding can then find the segment in the first steps, but is later unable to commit the binding to DB as a foreign key constraint fails. We catch the foreign key constraint failing (DBReferenceError) and raise a retryable RetryRequest instance instead, which the retry_db_errors() decorator around _bind_port_if_needed() should handle.
Using the 'regular' /etc/resolv.conf inside the network namespaces prevents proper DNS resolution, as these nameservers usually are unreachable from within the namespaces. This change enables creation of a resolv.conf in each namespace. The new setting netns_resolvconf enables this feature, by default it is disabled. Default nameservers are 127.0.0.1 and ::1 if IPv6 is enabled, for search domains the networks dns_domain is used if set. These defaults can be overwritten via settings file. To skip setting nameservers, search domains or options in the generated file and not use the dynamic defaults, one can add the settings key in the configuration file but leave out any value, e.g.: netns_resolvconf = yes netns_resolvconf_search = netns_resolvconf_options = netns_resolvconf_nameservers = This would create an empty resolv.conf.
This adds support for configuring custom upstream DNS servers and to disable the edns logging for neutron networks, based on the name of the OpenStack domain or, for development, the project id. By changing the upstream DNS servers used by dnsmasq, local hostnames are still resolvable, but queries for external names will be sent to the custom set of resolvers from the configuration. This is in contrast to the existing dns server setting for networks, which changes the nameservers announced via DHCP. The configuration is done in neutron server. When a dhcp-agent first configures a network, it does an rpc request to retrieve all necessary configuration data. On the server side we add the custom DNS servers and a flag for the edns logging to the returned data. The agent code then uses this data to configure the dnsmasq instances accordingly. When that data is absent the local configuration is used as default, like before. To decide if the network is to be configured with a custom set of nameservers, the neutron rpc server will query keystone for the domain of the network and match the domain name against its configuration file. To help development and debugging, support to directly match the id of a project was also added.
After updating to Caracal and using openstacksdk 3.0.0 the sdk refused initializing the identity client correctly to talk to keystone. This apparently happens when a full config is given to the Connection, and should not have worked before. Also now calling identity directly without indirection, which also should be more efficient because it uses a direct get for the id.
To customize behavior in Nova’s server creation process, an effective approach is to send custom events that Nova can wait on. In our case, we want to pause the server creation until the ML2 port binding (handled by the NSX-T agent) is completed. This helps avoid a race condition.
Modify the general default for port security in the rpc server. The previous commit changed only the defaults for the linuxbridge agent. However, the rpc server also needs to be updated to ensure that the default for port security is consistent. fixes: 34819de
This allows to disable the configuration of the anti-spoofing rules using a feature flag. This is currently required since we do not yet have port security enabled and using the Linux bridge driver will still configure the anti-spoofing rules by default. For the spoofing case this configures DHCP to be allowed by default like in the anti-spoofing case. In addition, this adds default allow rules towards the metadata server for the anti spoofing case as well.
When we migrate a vm between two kvm hypervisors that are using the linuxbridge driver, we need to make sure that the network is present when the vm comes up. This is important so the VMs/HVs rARP packets are send out and the network fabric behind the HV knows where to find the VM after a migration. With linuxbridge and Nova it is not clear who actually creates the bridge and adds the tap interface to it. Generally it seems to be that Nova does this first and Neutron only discovers this in one of its agent's loops. But here we have a problem: The bond is only created and put into the bridge once the port binding is active. It seems like OVS had a similar problem, which was patched here[0]. The code looks very similar to our problematic piece of code, so we adapt the idea: The port is not only bound if the host of the port matches, but also if the migrating_to field from the binding profile matches the host. This helps us getting the bond faster into the bridge. Additionally, we can - via config - increase the polling timeout of the linuxbridge agent to make sure we do our deed as fast as possible. Our patch looks a bit different from [0]: We don't fetch the active bindings, as all the information we need is already present in the port dict that we have. In some cases (at least in the tests) the profile might be a string and not already a json-decoded data structure, so we need to do the decoding ourselves in some cases (same as [0]). As a midterm solution it would probably make sense to get this done in a quicker, more reactive fashion (i.e. not wait for a loop, but directly react on the event). Another option would be to migrate away from linuxbridge, but one step after the other. Note that this is only happening on "cold" hosts, which have never seen the OpenStack network so far. Linuxbridge does not seem to clean up the bridge or bond interface (which might also be something which we need to look into in the future). [0] openstack@f8a22c7
Admin-shutting an agent leads to the agent being completely vacated. However we have cases in which we want an agent to stay online, but prevent any new workloads to be scheduled on it. This can be a capacity consideration or because it is meant to be decomissioned. In addition this allows us in the dev environment to replace agents with experimental versions, without causing disruption. It is still possible to manually schedule a network on this dhcp-agent for testing. A similar feature was implemented for asr1k-neutron-l3 here: sapcc/asr1k-neutron-l3@425cc81 Note: We are not filtering in `AutoScheduler.auto_schedule_networks`. This method gets called with the host as parameter, so the scheduling decision is already made.
…g and DNS
We already allow configuration of custom upstream DNS servers for
dnsmasq, based on the OpenStack domain or -- for testing -- based on
the project-id.
We currently can configure only two sets of DNS upstream servers:
One for SAP OpenStack domains and one for external OpenStack domains.
For external domains we also always disable the edns and umbrella logging
options in dnsmasq.
Now the requirement arose to configure DNS servers individually on a
per-domain basis and also to be able to enable logging individually.
The requirements now are:
- use default DNS servers and enable logging when no config is set
- enable logging and custom upstreams for specific domains individually
- allow fallback for domains matching a certain prefix to use a
different default set of resolvers and setting for logging.
This is achieved via a new configuration setting 'config_file' in the
customdns section of neutron.conf that points to a yaml file.
This file allows setting logging and upstream dns settings as follows:
```yaml
matches:
- domain_name_prefixes:
- external-abc
project_ids:
- 5dc81c6355ff478188f8fda11a971c41
upstream_dns_servers:
- 192.0.2.10
- 192.0.2.20
ednslogging: True
- domain_name_prefixes:
- external-abcd
- external-
upstream_dns_servers:
- 192.0.2.30
- 192.0.2.40
ednslogging: False
```
related:
f39dcce
f186f86
c4bdb29
When importing dhcp_rpc without first registering the DNSSETTINGS_OPTS config options, the import will fail with a oslo_config.cfg.NoSuchOptError exception, as we were instantiating CustomNetworkConfigurator() as a class variable. This results in the code doing the import (in our case the nsx-t agent from a custom driver) crashing on startup. To fix this, we now create this object only once when creating the first instance of DhcpRpcCallback.
Enable container readiness checks by writing the DHCP agent's status to a local JSON file. The status includes sync state, timestamp, and network namespace information, allowing containers to determine readiness without querying the Neutron database. Previously, we relied on the agent's status in the Neutron DB for readiness checks. This caused database overload during mass restarts, especially under high customer request volumes, as readiness checks added unnecessary pressure to the DB. The status file tracks whether all active networks have corresponding network namespaces, reporting "synced", "unsynced", or "error" states. This eliminates external dependencies and reduces database load during container lifecycle events.
The customdns configuration is now done using a yaml config file pointed to in the neutron config file. This commit removes the now unused options from the configuration, removes the obsolete code and the tests and cleans up the code. With this change the yaml configuration file is now mandatory when the customdns feature is enabled. We also renamed some of the tests to better describe their purpose and renamed the "doubled" tests for the functionality back to the original names by removing the _yaml postfix. Additional tests ensure the configuration file is required when the customdns feature is enabled and exceptions are handled accordingly.
Problem: The sync_loop runs only when triggered by specific events (e.g., network create/delete operations). This means the agent status check does not run regularly as previously assumed. In rapid create/delete scenarios, the status can become unsynced: the network exists in the database but the network namespace is never created. From Neutron's perspective everything appears fine, but the agent status shows unsynced and never updates. Solution: Use the cache as the source of truth for agent readiness checks. The cache is kept up-to-date by the agent on every operation (create/ delete network/port/subnet). Run regular checks against the cache to determine if all networks are synced and mark the agent ready accordingly. For detecting when the cache is out of sync, we rely on other measures such as the custom exporter for network namespaces and MariaDB exports.
This writes the list of synced networks into a file, that can be shared to a k8s sidecar container. Moved the file to /run/dhcp-agent to enable shared mount between containers. This enables a prometheus exporter, running in a sidecar, to export the synced networks in an agent. Because a restart of the dhcp-agent container will remove all network namespaces from the kernel but not the files in /run/netns, sharing this as mount is not an option. This will prevent the dhcp-agent from starting up again, because the namespace files are in place, but no namespaces.
Similar to the recently introduced feature of setting custom DNS upstream servers for dnsmasq based on the OpenStack Domain or project of a network, this adds support for configuring the NTP servers distributed via DHCP option 42 to the clients. Note: To take effect, the option must not be set in the dnsmasq config file, as commandline options are overwritten when they are only allowed to be given once.
We extended the DHCP agent to write the synced networks to a status file. With the new release Neutron moved away from Eventlet, introducing changes, breaking our custom agent tests. Explicitly call init_host (which internally also calls sync_state) to initialise the network cache. In the safe_get_network_info method we fetch fresh network information. As this would return a plain Mock object, not containing the network_id and subnet information, the test would fail without mocking it. With the drop of eventlet, the network cache spawns a thread fetching updates. We need to stop this thread to stop the tests from hanging.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.