[BUG] Adding Keyvault-based Spring SSL bundle breaks mTLS

[jkuipers]

I have a Spring Boot 3.5.8 app running on Java 25 on Azure App Service. It depends on Spring Cloud Azure, using the 6.1.0 BOM.
In the app we have an Apache Components HTTP Clients configured to use mTLS to talk to a SOAP service. When we configure the client to use a key and certificate from an SSL Bundle backed by a local .p12 keystore, this works as expected.
However, as soon as we use a KeyVault-based bundle, this breaks: when it's the bundle to use for the mTLS connection, but also even when the KeyVault-based bundle is simply added next to the JKS bundle without being used.
This typically breaks by throwing a java.net.SocketException: Connection reset by peer or java.net.SocketException: Broken pipe.

To Reproduce

I wrote a standalone Java app as a single source file next to our web app for quick testing on the Azure box from an SSH shell. It looks like this:

package nl;

import nl.aon.libs.schedulerlock.config.ShedlockAutoConfiguration;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManagerBuilder;
import org.apache.hc.client5.http.ssl.ClientTlsStrategyBuilder;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.SpringBootConfiguration;
import org.springframework.boot.WebApplicationType;
import org.springframework.boot.actuate.autoconfigure.metrics.MetricsAutoConfiguration;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
import org.springframework.boot.ssl.SslBundle;
import org.springframework.boot.ssl.SslBundles;
import org.springframework.context.annotation.Bean;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.ws.client.core.WebServiceTemplate;
import org.springframework.ws.transport.http.ClientHttpRequestMessageSender;
import org.springframework.xml.transform.StringResult;
import org.springframework.xml.transform.StringSource;

import java.security.Provider;
import java.security.Security;
import java.util.List;

@SpringBootConfiguration
@EnableAutoConfiguration(exclude = { DataSourceAutoConfiguration.class, ShedlockAutoConfiguration.class, MetricsAutoConfiguration.class})
public class Client {
    public static void main(String[] args) throws Exception {
        SpringApplication springApplication = new SpringApplication(Client.class);
        springApplication.setWebApplicationType(WebApplicationType.NONE);
        springApplication.run(args);
    }

    @Bean
    WebServiceTemplate wsTemplate(SslBundles sslBundles) {
        SslBundle audaScan = sslBundles.getBundle("audascanBundle");
        var tlsStrategy = ClientTlsStrategyBuilder.create()
                .setSslContext(audaScan.createSslContext())
                .buildClassic();
        var connectionManager = PoolingHttpClientConnectionManagerBuilder.create()
                .setTlsSocketStrategy(tlsStrategy)
                .build();
        var httpClient = HttpClients.custom()
                .setConnectionManager(connectionManager)
                .build();
        var webServiceMessageSender = new ClientHttpRequestMessageSender(new HttpComponentsClientHttpRequestFactory(httpClient));
        WebServiceTemplate webServiceTemplate = new WebServiceTemplate();
        webServiceTemplate.setMessageSender(webServiceMessageSender);
        return webServiceTemplate;
    }

    @Bean
    CommandLineRunner call(WebServiceTemplate webServiceTemplate) {
        return args -> {
            Provider[] providers = Security.getProviders();
            System.out.println("providers = " + List.of(providers));

            System.out.println("Calling WS");
            var result = new StringResult();
            webServiceTemplate.sendSourceAndReceiveToResult("https://isa-ws.dem.solera.nl/isa-ws/services/isa3", new StringSource("""
                            <ns2:AUDASCAN-360-V2 xmlns:ns2="http://www.abz.nl/schema/audascan" xmlns:ns3="http://www.abz.nl/schema/isa" xmlns:ns4="http://www.abz.nl/schema/foto" xmlns:ns5="urn:isa3"><ns3:VT><ns3:VT_KENTEKEN>GHT69D</ns3:VT_KENTEKEN><ns3:VT_MELDCODE>1010</ns3:VT_MELDCODE></ns3:VT></ns2:AUDASCAN-360-V2>
                            """),
                    result);
            System.out.println("result = " + result);
        };
    }
}

and I've added the following application.yml config file:

spring:
  cloud:
    azure:
      keyvault:
        jca:
          vaults:
            keyvault1:
              endpoint: https://kv-[redacted].vault.azure.net/
              credential:
                client-id: [redacted]
                managed-identity-enabled: true
---
spring:
  config.activate.on-profile: p12-solo
  ssl:
    bundle:
      jks:
        audascanBundle:
          keystore:
            location: classpath:audascan.p12
            password: changeit
---
spring:
  config.activate.on-profile: p12-duo
  ssl:
    bundle:
      jks:
        audascanBundle:
          keystore:
            location: classpath:audascan.p12
            password: changeit
      keyvault:
        audascanBundleKV:
          for-client-auth: true
          keystore:
            keyvault-ref: keyvault1
          key:
            alias: audascan
---
spring:
  config.activate.on-profile: keyvault
  ssl:
    bundle:
      keyvault:
        audascanBundle:
          for-client-auth: true
          keystore:
            keyvault-ref: keyvault1
          key:
            alias: audascan

On the local filesystem we have the audascan.p12 file, which is the same keystore that's also present in the Azure KeyVault under the alias audascan.

I can now run the app in different way. If I start it like this, it works as expected:

java -cp "/usr/local/tomcat/webapps/ROOT/WEB-INF/lib/*" -Dspring.profiles.active=p12-solo nl/Client.java

When run like this, this is what it prints for the security providers:

providers = [SUN version 25, SunRsaSign version 25, SunEC version 25, SunJSSE version 25, SunJCE version 25, SunJGSS version 25, SunSASL version 25, XMLDSig version 25, SunPCSC version 25, JdkLDAP version 25, JdkSASL version 25, SunPKCS11 version 25]

Now, I can also run it like this which will add a KeyVault-based SSL Bundle to the configured list of bundles, without actually using it:

java -cp "/usr/local/tomcat/webapps/ROOT/WEB-INF/lib/*" -Dspring.profiles.active=p12-duo nl/Client.java

This breaks the application. The output for the providers now shows this (which I think is relevant, therefore the code prints this):

providers = [AzureKeyVault version 1.0, SUN version 25, SunRsaSign version 25, SunEC version 25, SunJSSE version 25, SunJCE version 25, SunJGSS version 25, SunSASL version 25, XMLDSig version 25, SunPCSC version 25, JdkLDAP version 25, JdkSASL version 25, SunPKCS11 version 25]

I've read somewhere that some versions of the JCA integration do not support mTLS: however, a) I'm not clear on whether this applies to my version and b) even if it does, I would not expect that simply configuring an additional KeyVault-backed SSL bundle completely breaks my application that doesn't even use that bundle for its mTLS config.

If I run the app by passing the keyvault profile, which enables only the KeyVault-backed SSL bundle, I get the same behavior as with the p12-duo profile.

Is this indeed a bug? Or is it still expected that simply adding a KeyVault-backed SSL bundle will completely break any form of mTLS in my application, even when I configure my own keystores for that? I also tried to rewrite my code to not even use a Spring SSL Bundle but to read the .p12 from the filesystem as a Keystore myself and create the SSL context from that, but it behaves in the same way. Therefore I suspect that this is related to the Azure security provider that gets registered as soon as you add a KeyVault bundle.

[github-actions]

@moarychan @Netyyyy @rujche @saragluna

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.