Fix MSCAPISignatureToken compatibility with RSASSA-PSS (SunMSCAPI delegation)

[gustavoam-asdf]

Problem Description

When using MSCAPISignatureToken with certificates stored in the Windows Certificate Store (MY) with high level of security in Crypto API, signing operations fail with a DSSException when attempting to use RSASSA-PSS algorithms (e.g., SHA256withRSAandMGF1), even if the certificate itself is a standard SHA256withRSA.

Observation

• Signing with the same algorithm as the certificate (e.g., SHA256withRSA) works correctly because the default provider selection might favor a compatible engine or a simple RSA implementation.
• Signing with RSASSA-PSS triggers the error. This happens because DSS (often with Bouncy Castle registered as a provider) selects the BC engine for PSS. Bouncy Castle expects a software-based RSAPrivateKey instance, but the Windows Certificate Store provides native keys (CAPI/KSP) which are opaque handles.

Error Stack Trace

-------------- CERTIFICATE INFO --------------
Serial Number           : 45be9789de7d8206b4c63c2ed9e0afc4cae225aa
Signature algorithm     : SHA256withRSA
----------------------------------------------
...
[main] INFO eu.europa.esig.dss.token.AbstractSignatureTokenConnection - Signature algorithm : SHA256withRSAandMGF1
Exception in thread "main" eu.europa.esig.dss.model.DSSException: Unable to sign : Supplied key is not a RSAPrivateKey instance
        at eu.europa.esig.dss.token.AbstractSignatureTokenConnection.sign(AbstractSignatureTokenConnection.java:79)
Caused by: java.security.InvalidKeyException: Supplied key is not a RSAPrivateKey instance
        at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineInitSign(Unknown Source)
        at java.base/java.security.Signature$Delegate.engineInitSign(Signature.java:1357)
        at java.base/java.security.Signature.initSign(Signature.java:636)

Proposed Solution

The MSCAPISignatureToken is inherently tied to the Windows environment. To ensure maximum compatibility with native keys, it should prioritize the SunMSCAPI provider for all signing operations.

Key Changes

1. Provider Priority: Overrides getSignatureInstance to explicitly request the SunMSCAPI provider. This ensures that the native Windows engine, which "owns" the key handle, is the one performing the cryptographic operation.
2. Algorithm Normalization: Maps Bouncy Castle/DSS style PSS algorithm names (containing MGF1 or PSS) to the standard RSASSA-PSS name recognized by SunMSCAPI (available since Java 11).
3. Graceful Fallback: If SunMSCAPI is not available or doesn't support the requested algorithm (e.g., on very old Java versions), it falls back to the default implementation in AbstractSignatureTokenConnection.

Implementation

@Override
protected Signature getSignatureInstance(final String javaSignatureAlgorithm) throws NoSuchAlgorithmException {
  try {
    // ? Check if this is the best way to map the algorithm.
    if (javaSignatureAlgorithm.contains("RSAandMGF1") || javaSignatureAlgorithm.contains("PSS")) {
      return Signature.getInstance("RSASSA-PSS", SUN_MSCAPI);
    }

    return Signature.getInstance(javaSignatureAlgorithm, SUN_MSCAPI);
  } catch (NoSuchAlgorithmException | NoSuchProviderException e) {
    // Fallback to default behavior (e.g. if algorithm not supported by SunMSCAPI or
    // provider not available)
    return super.getSignatureInstance(javaSignatureAlgorithm);
  }
}

Benefits

• Enables RSASSA-PSS signatures for Windows-stored certificates.
• Prevents provider-mismatch errors when Bouncy Castle is present in the classpath.
• Maintains backward compatibility for environments where SunMSCAPI might be restricted or outdated.

[bsanchezb]

Hello,

Thank you for the ticket and identifying the issue. I indeed was able to reproduce it locally.

It looks like a limitation in BouncyCastle, not explicitly supporting the MS-CAPI keystore. I have created a ticket #2280 addressing the issue.

I stay reluctant on hardcoding the security provider and corresponding logic directly in DSS, and would prefer first to wait for a response from BouncyCastle.

KR,
Aleksandr

[axelabinet]

Hello @gustavoam-asdf,

Thank you again for identifying the issue.

Following the response from BouncyCastle, the solution is indeed to explicitly use the SunMSCAPI provider for signing with RSASSA-PSS.

We have reviewed your pull request and included it in the develop branch with a few changes:

1. We only check if the signature algorithm contains RSAandMGF1.
2. We only explicitly use the SunMSCAPI provider for a RSASSA_PSS algorithm, for all the other cases we use the default behavior of getSignatureInstance.
3. We noticed that the digest signing with RSASSA-PSS and SunMSCAPI provider was not working, because the provider will always recompute the digest. We added a check to handle this case.

This change will be included in the release 6.6 of DSS.

Kind regards,
Axel ABINET