Keyless mTLS in Java: Using the OpenSSL Provider & Remote Signing for keyless handshake

Mutual TLS (mTLS) requires the client to prove possession of the private key associated with its certificate. The catch? A lot of security policies today don’t want private keys anywhere near application code or memory.

That’s where keyless mTLS comes in — the TLS handshake still happens as usual, but the private-key signature step is delegated to a remote signing service.

In this article, we’re going to implement this in Java + Spring Boot.

Netty’s tcnative boringssl makes this possible by using the OpenSSL TLS provider along with your own implementation of OpenSslPrivateKeyMethod.

In short: the certificate chain sits locally, the private key stays external, and TLS just….works.

Getting Started

Here’s how to wire this up:

1. First, we add the required dependencies in pom.xml

..
<!-- Spring WebClient auto-configuration (uses Netty under the hood) -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-webflux</artifactId>
    </dependency>
<!-- OpenSSL / BoringSSL native TLS provider (keyless-capable) -->
    <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty-tcnative-boringssl-static</artifactId>
    </dependency>
..

2. Next, we setup the SSLContext for keyless mTLS

SslContext sslContext = SslContextBuilder.forClient()
        .sslProvider(SslProvider.OPENSSL) // Important!
        .trustManager(trustChain)
        .keyManager(OpenSslX509KeyManagerFactory.newKeyless(certChain)) // cert chain for mTLS
        .protocols("TLSv1.3")
        .option(OpenSslContextOption.PRIVATE_KEY_METHOD, new CustomPrivateKeyMethod())
        .build();

3. And this is what the private key hook for our custom remote signing service looks like.

Note: the signatureAlgorithm in sign(..) is an integer which must be mapped to the remote signing service’s algorithm suite before making the call.

public class CustomPrivateKeyMethod extends OpenSslPrivateKeyMethod {

    @Override
    public byte[] sign(
            SSLEngine engine,
            int signatureAlgorithm,
            byte[] digest
    ) throws Exception {

        // Map OpenSSL signature algorithm → remote signing service algorithm
        String alg = mapSignatureAlgorithm(signatureAlgorithm);

        // Delegate signing to remote signing service
        return SigningClient.sign(digest, alg);
    }

    @Override
    public byte[] decrypt(
            SSLEngine engine,
            byte[] input
    ) throws Exception {
        // Only required if external key supports RSA decryption (not common)
        throw new UnsupportedOperationException("decrypt not supported");
    }

    // Mapping OpenSSL signature algorithm codes to signing-service identifiers
    // These values originate from TLS 1.3 signature scheme registry:
    // https://datatracker.ietf.org/doc/html/rfc8446#section-4.2.3
    private static final Map<Integer, String> SIG_ALG_MAP = Map.of(
            // RSA-PSS
            2052, "PS256",
            2053, "PS384",
            2054, "PS512",
            // RSA PKCS#1 v1.5
            1025, "RS256",
            1281, "RS384",
            1537, "RS512"
    );

    private static String mapSignatureAlgorithm(int opensslAlg) {
        return SIG_ALG_MAP.get(opensslAlg);
    }
}

4. Here’s a sample SigningClient that calls the remote signing API.

public class SigningClient {

private static final WebClient client = WebClient.builder()
            .baseUrl("https://signing.example.com")  // remote signing service endpoint
            .build();
     /**
     * @param digest base hash of handshake payload (OpenSSL provides this)
     * @param alg    algorithm string (PS256, RS256, etc) mapped by CustomPrivateKeyMethod
     */
    public static byte[] sign(byte[] digest, String alg) {
        SignRequest body = new SignRequest(
                "key_identifier", //identifier of the key to use
                alg,
                Base64.getEncoder().encodeToString(digest)
        );
        SignResponse resp = client.post()
                .uri("/sign")
                .bodyValue(body)
                .retrieve()
                .bodyToMono(SignResponse.class)
                .block(); 
        return Base64.getDecoder().decode(resp.signatureB64());
    }
    // --- request/response DTOs ---
    public record SignRequest(String keyId, String alg, String payloadB64) {}
    public record SignResponse(String signatureB64) {}
}

5. Finally, we setup the SslContext in the WebClient.

HttpClient httpClient = HttpClient.create()
        .secure(ssl -> ssl.sslContext(sslContext));

WebClient mtlsClient = WebClient.builder()
        .clientConnector(new ReactorClientHttpConnector(httpClient))
        .build();

6. Now the mTLS server is ready to be called using the mtlsClient we configured in the previous step.

var result = mtlsClient.post()
        .uri("https://actual-server/do-something")
        .bodyValue(payload)
        .retrieve()
        .bodyToMono(String.class)
        .block();

That’s it! The keyless mTLS will now function like so :

Considerations

1. The OpenSSL / BoringSSL provider relies on platform-specific native libraries, distributed via netty-tcnative-boringssl-static. Implications:

• Container base image choice matters (Alpine, Amazon Linux, Ubuntu behave differently)
• Missing dependencies → hard failures at runtime (UnsatisfiedLinkError)

JDK TLS works “anywhere Java runs”, OpenSSL TLS needs the right native lib on the right platform.

2. Signature-algorithm handling

OpenSSL provides an integer, not a Java enum. Applications need to maintain a mapping for the signature algorithms.

Summary

This is one of those edge-case TLS setups that most teams will (hopefully?) never run into. But on the off chance you do need a handshake without loading the private key into the JVM, this is one way to pull it off without losing your sanity. 😄