TLS 1.3 mTLS with BouncyCastle: client cert missing

Most likely the problem is either (a) your BouncyCastle TLS client never actually emits the TLS 1.3 Certificate/CertificateVerify records, or (b) it does send them but the server rejects the CertificateVerify signature because of a key/encoding or signature-algorithm mismatch. Check key encoding (PKCS#8 vs PKCS#1), force/choose the right SignatureAndHashAlgorithm (prefer rsa_pss_* for TLS 1.3 if the server requests it), enable BC/nginx debug and capture a packet trace (filter for tls.handshake.type == 11 and 15) to see which of those it is.

Contents

• Quick checklist: confirm whether the client sends certs (tls 1.3 / bouncycastle)
• Capture & decode the TLS 1.3 Certificate / CertificateVerify messages (Wireshark + nginx)
• PrivateKey encoding and PrivateKeyFactory.createKey — PKCS#8 vs PKCS#1
• Signature algorithms in TLS 1.3 — prefer RSA‑PSS; how to pick/force sigAlg
• Known BouncyCastle TLS 1.3 pitfalls and recommended versions / workarounds
• Custom socket and lifecycle checks (TlsClientProtocol wrapper)
• Targeted diagnostics and minimal reproducer to isolate A/B/C causes
• Practical example: getClientCredentials pattern and debug hooks (snippet)
• Sources
• Conclusion

Quick checklist: confirm whether the client sends certs (tls 1.3 / bouncycastle)

Start simple, rule things out in order:

• Does nginx actually request a client certificate? Check your nginx config (ssl_verify_client on; and what CA list you provide). If disabling verify makes the request succeed, the server definitely received something different from what it wanted—or nothing at all. See nginx docs for how nginx logs handshake failures and how to enable debug logging: https://nginx.org/en/docs/http/ngx_http_ssl_module.html

• Is getClientCredentials() being invoked early enough and returning a valid TlsCredentialedSigner? Your logs show “Successfully obtained client credentials for mTLS.” — that’s a good sign, but it doesn’t prove the client sent the certificate or produced a valid signature.

• Next: do a packet capture and check for Certificate (handshake type 11) and CertificateVerify (type 15) frames (exact filter below). If the client never sends those two messages the server will reject the handshake.

If you want a one-line priority: 1) capture the handshake, 2) enable BC and nginx debug, 3) check private-key encoding and signature algorithm selection.

Capture & decode the TLS 1.3 Certificate / CertificateVerify messages (Wireshark + nginx)

What to capture

• On the client or gateway run:
• sudo tcpdump -i any host 192.168.178.88 and port 443 -w mtls.pcap
• Open mtls.pcap in Wireshark and apply these filters:
• tls.handshake.type == 11 (Certificate)
• tls.handshake.type == 15 (CertificateVerify)

Important TLS 1.3 note

• Certificate and CertificateVerify are sent encrypted after the ServerHello in TLS 1.3. To see their contents you must decrypt the session. The Wireshark TLS wiki summarizes filters and decryption: https://wiki.wireshark.org/TLS

How to get decrypt keys / logs

• If you can run a JSSE-based client or BCJSSE with key logging, write secrets to an SSLKEYLOGFILE and configure Wireshark to use it (Wireshark → Preferences → Protocols → TLS → (Pre)-Master-Secret log filename). For BC JSSE enable provider debug as described in the BC docs/issue thread: https://github.com/bcgit/bc-java/issues/1062 and the BC user guide: https://downloads.bouncycastle.org/fips-java/docs/BC-FJA-(D)TLSUserGuide-1.0.13.pdf

If you cannot easily produce key logs

• You can still detect presence/absence: even when encrypted you will see an Encrypted Handshake/Encrypted Application Data record after ServerHello. If the client never sends an encrypted handshake record at that point it likely didn’t send Certificate/CertificateVerify. If you see the records but can’t decrypt them, proceed to BC debug and server-side verify logs.

Server-side logging

• Enable nginx debug logging (error_log … debug) and inspect the exact SSL error. Nginx will log SSL_do_handshake() failures and the TLS alert number. Alert 80 means internal_error — often used when signature verification fails or when OpenSSL returns a verification/interoperability error (see nginx module docs): https://nginx.org/en/docs/http/ngx_http_ssl_module.html

PrivateKey encoding and PrivateKeyFactory.createKey — PKCS#8 vs PKCS#1

Why this matters

• The BC lightweight conversion PrivateKeyFactory.createKey(byte[] keyBytes) expects standard PKCS#8 DER-encoded private key bytes. If your PrivateKey is in PKCS#1 (traditional “BEGIN RSA PRIVATE KEY”) or the key bytes are not PKCS#8, signature creation can silently fail or produce an invalid signature that the server rejects. StackOverflow and BC examples call this out: https://stackoverflow.com/questions/18065170/how-do-i-do-tls-with-bouncycastle

Quick checks

• In Java: System.out.println(privateKey.getFormat()); // expected: “PKCS#8”
• If privateKey.getEncoded() is null or getFormat() != “PKCS#8”, you must convert to PKCS#8.

How to fix

• If you exported keys using OpenSSL you can convert PKCS#1 -> PKCS#8:
• openssl pkcs8 -topk8 -nocrypt -in key-pkcs1.pem -out key-pkcs8.pem
• Or in Java/BouncyCastle parse PEM (PEMParser/JcaPEMKeyConverter) and re-encode as PKCS#8 before calling PrivateKeyFactory.createKey(byte[]).
• After conversion use:
• AsymmetricKeyParameter bcKeyParam = PrivateKeyFactory.createKey(pkcs8Bytes);

If encoding was wrong it explains why the client appears to “get credentials” but nginx rejects the signature with an internal_error.

References: StackOverflow example and diagnosis: https://stackoverflow.com/questions/18065170/how-do-i-do-tls-with-bouncycastle

Signature algorithms in TLS 1.3 — prefer RSA‑PSS; how to pick/force sigAlg

TLS 1.3 specifics

• Servers advertise supported signature algorithms in the CertificateRequest. TLS 1.3 prefers RSA‑PSS variants for RSA keys (rsa_pss_rsae_sha256 etc). If the client signs with PKCS#1 v1.5 (rsa_pkcs1_sha256) while the server expects PSS, the server may reject the CertificateVerify. That can manifest as alert internal_error(80) in nginx logs.

What to do in your getClientCredentials()

• Inspect certificateRequest.getSupportedSignatureAlgorithms() and pick the best match. Try to prefer RSA‑PSS entries if present.
• If you need a quick test, force the signer to use an RSA‑PSS algorithm (the StackOverflow thread that matched your symptoms used forcing rsa_pss_rsae_sha256 to solve a similar nginx rejection): https://stackoverflow.com/questions/79868563/not-able-to-send-client-certificate-to-nginx-server-bouncycastle

Pseudocode example (concept):

Vector<SignatureAndHashAlgorithm> sigAlgs = certificateRequest.getSupportedSignatureAlgorithms();
SignatureAndHashAlgorithm chosen = pickPreferPSS(sigAlgs);
if (chosen == null) chosen = fallbackSha256Pkcs1();
return new BcDefaultTlsCredentialedSigner(cryptoParams, crypto, bcKeyParam, bcTlsCertificate, chosen);

If the server and client disagree on signature format, the server will see an invalid CertificateVerify even though the client “sent” something.

Known BouncyCastle TLS 1.3 pitfalls and recommended versions / workarounds

History and concrete guidance

• There are known issues in older BC TLS implementations where TLS 1.3 client auth could be mishandled (client credentials returned but messages not sent or signed incorrectly). The BC user guide and GitHub issue threads discuss TLS 1.3 client-auth edge cases and fixes; users recommend upgrading to a BC release that contains TLS 1.3 client-auth fixes (versions after the pre-1.71 problem range). See the BC TLS user guide and the project issue thread for reproduction/diagnostics: https://downloads.bouncycastle.org/fips-java/docs/BC-FJA-(D)TLSUserGuide-1.0.13.pdf and https://github.com/bcgit/bc-java/issues/1062

Recommended actions

• Upgrade BouncyCastle to a release that includes TLS 1.3 client-auth fixes.
• If you don’t need a custom lightweight client, try the BCJSSE provider path; it aligns more closely with JSSE behavior and provides better debug flags: https://github.com/bcgit/bc-java/issues/1062

Custom socket and lifecycle checks (TlsClientProtocol wrapper)

Common mistakes when wrapping TlsClientProtocol

• Reusing a TlsClientProtocol instance across multiple sockets.
• Returning from createSocket early (not giving the handshake time to run) or closing streams prematurely.
• Not calling tlsClientProtocol.connect(…) in the expected thread or with full streams available.

What to check in your code

• Ensure createSocket returns a fresh socket object whose startHandshake() blocks until tlsClientProtocol.connect(…) completes.
• Verify you do not wrap InputStream/OutputStream with layers that buffer and delay the bytes needed by the handshake.
• Make sure you don’t reuse the same TlsClientProtocol over multiple TCP connections; it is per-connection.

A simple test

• Replace the custom socket wrapper with a direct small main() that creates a Socket, calls new TlsClientProtocol(in, out).connect(new MyDefaultTlsClient(…)) and prints logs — if that works, the problem is in your wrapper lifecycle.

Targeted diagnostics and minimal reproducer to isolate A / B / C causes

Step-by-step prioritized diagnostics

1. Packet capture: capture and open in Wireshark; filter for tls.handshake.type == 11 || tls.handshake.type == 15. If you see Certificate/CertificateVerify packets (after decryption) the client is sending them — go to step 3. (See Wireshark: https://wiki.wireshark.org/TLS)

2. If you don’t see them:

• Add verbose BC debug for JSSE path: start JVM with -Dorg.bouncycastle.jsse.provider.debug=true and/or enable any lightweight TLS debugging available. See https://github.com/bcgit/bc-java/issues/1062 and the BC user guide.
• Add explicit logging in getClientCredentials() and wrap the signer: log before and after signature generation, log any exceptions.

3. If client sends packets but server rejects:

• Check private key encoding and ensure it’s PKCS#8 (StackOverflow: https://stackoverflow.com/questions/18065170/how-do-i-do-tls-with-bouncycastle).
• Force/choose RSA‑PSS if server advertised it; test both PSS and PKCS#1 variations and compare nginx debug output.

4. Server-side: set nginx error log level to debug and temporarily set ssl_verify_client optional to avoid immediate rejection and to collect $ssl_client_* nginx variables to see the raw cert and verification result (nginx docs: https://nginx.org/en/docs/http/ngx_http_ssl_module.html).

5. Quick repro:

• Try the same keystore with a known-working client (Firefox works per your notes). Try a SunJSSE-based Java client or BCJSSE to see if the same keystore + key performs client auth. If yes, that isolates the problem to your lightweight BC path.

6. Fallback test:

• If possible, create a tiny test server (OpenSSL s_server) and try the BC client against it while logging both sides; OpenSSL will show why signature verification failed, often with clearer output than nginx.

Practical example: getClientCredentials pattern and debug hooks (snippet)

Below is a compact pattern to log/choose sigAlg and produce a BcDefaultTlsCredentialedSigner (conceptual; adapt imports):

@Override
public TlsCredentials getClientCredentials(CertificateRequest certificateRequest) {
 System.out.println("certificateRequest = " + certificateRequest);
 String alias = "CN-ClientCertificate";
 java.security.cert.Certificate[] certChain = keyStore.getCertificateChain(alias);
 PrivateKey privateKey = (PrivateKey) keyStore.getKey(alias, password);

 // BC TLS certificates
 TlsCertificate[] bcChain = new TlsCertificate[certChain.length];
 for (int i = 0; i < certChain.length; i++) {
 bcChain[i] = crypto.createCertificate(certChain[i].getEncoded());
 System.out.println("chain cert subject: " + ((X509Certificate)certChain[i]).getSubjectDN());
 }
 org.bouncycastle.tls.Certificate bcTlsCertificate = new org.bouncycastle.tls.Certificate(bcChain);

 // Ensure PKCS#8 bytes
 byte[] keyBytes = privateKey.getEncoded();
 if (keyBytes == null) throw new RuntimeException("privateKey.getEncoded() == null; wrong format");
 AsymmetricKeyParameter bcKeyParam = PrivateKeyFactory.createKey(keyBytes);

 // Select signature algorithm from certificateRequest
 SignatureAndHashAlgorithm sigAlg = null;
 Vector sigAlgs = certificateRequest.getSupportedSignatureAlgorithms();
 if (sigAlgs != null) {
 for (Object o : sigAlgs) {
 SignatureAndHashAlgorithm a = (SignatureAndHashAlgorithm)o;
 System.out.println("server supports: " + a);
 // prefer PSS if present (pseudo check)
 if (/* a indicates rsa_pss_rsae_sha256 */) {
 sigAlg = a; break;
 }
 }
 }
 if (sigAlg == null) sigAlg = new SignatureAndHashAlgorithm(HashAlgorithm.sha256, SignatureAlgorithm.rsa);

 System.out.println("Using sigAlg: " + sigAlg);
 return new BcDefaultTlsCredentialedSigner(cryptoParams, crypto, bcKeyParam, bcTlsCertificate, sigAlg);
}

If you want guaranteed observability, implement a custom TlsCredentialedSigner that delegates to BcDefaultTlsCredentialedSigner and logs the signature byte[] returned — if signature creation throws or yields unexpected length, you’ll see it.

References for patterns and lightweight TLS client examples: https://7thzero.com/blog/how-to-use-bouncy-castle-lightweight-api-s-tlsclient and the StackOverflow TLS guide: https://stackoverflow.com/questions/18065170/how-do-i-do-tls-with-bouncycastle

Sources

• BouncyCastle GitHub issue discussing TLS 1.3 client-auth failures and debugging: https://github.com/bcgit/bc-java/issues/1062
• StackOverflow: How do I do TLS with BouncyCastle? (PKCS#8 vs PKCS#1 / signer details): https://stackoverflow.com/questions/18065170/how-do-i-do-tls-with-bouncycastle
• BouncyCastle TLS/JSSE user guide (TLS 1.3 client auth notes and version notes): https://downloads.bouncycastle.org/fips-java/docs/BC-FJA-(D)TLSUserGuide-1.0.13.pdf
• Wireshark TLS wiki — filters and TLS 1.3 notes: https://wiki.wireshark.org/TLS
• nginx SSL module docs (ssl_verify_client, debug logging): https://nginx.org/en/docs/http/ngx_http_ssl_module.html
• StackOverflow thread matching your symptom (client cert not sent/rejected with nginx + BouncyCastle): https://stackoverflow.com/questions/79868563/not-able-to-send-client-certificate-to-nginx-server-bouncycastle
• Example blog for BouncyCastle lightweight TLSClient usage: https://7thzero.com/blog/how-to-use-bouncy-castle-lightweight-api-s-tlsclient

Conclusion

Start with packet capture + nginx debug to answer the single key question: did the client send Certificate and CertificateVerify? If it didn’t, the problem is handshake flow or wrapper lifecycle; if it did, the most common root causes are private-key encoding (ensure PKCS#8) or a signature-algorithm mismatch (pick/force rsa_pss_* when the server requests it). Upgrade BouncyCastle to a release that includes TLS 1.3 client-auth fixes or test with the BCJSSE path for a quick comparison. In practice the fastest path is: capture the trace (tls.handshake.type == 11 or 15), enable BC debug and log signature generation inside your signer, and verify privateKey.getFormat() == “PKCS#8” — those three checks resolve almost all bouncycastle + tls 1.3 mtls nginx failures.