-
Title
AppServer login fails with mixed RSA/ECDSA session token certificate -
Description
Selected a custom certificate for the AppServer session token certificate. With that certificate active, login failed and the browser returned:
Http failure response for https://<host>/AppServer/auth/apphost: 400 OK
The certificate appeared selectable only when Show invalid certificates too was enabled in Web Installer.
The original installation-created certificate (CN=Application Server) continued to work.AppServer logs showed authentication succeeded, then token creation failed immediately:
INFO : User viadmin ... authenticated.
ERROR : [System.ArgumentNullException] Value cannot be null. (Parameter 'padding')
at System.Security.Cryptography.RSACng.SignHash(Byte[] hash, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding)
at VI.DB.Auth.AuthToken.Create(X509Certificate2 certificate, ClaimsIdentity identity, DateTime validToUtc)
at VI.DB.Entities.TokenCreatorImpl.CreateTokenAsync(...)
at QBM.AppServer.Base.SessionHandler.TryOpenSessionAsync(...) -
Cause
Certificate had this algorithm combination:
- Public key algorithm: RSA (4096-bit)
- Certificate signature algorithm: sha256ECDSA
A certificate includes two distinct algorithm properties:
- Key algorithm used by the application for signing/decryption
- Certificate signature algorithm used by the issuing CA to sign the certificate
This mixed combination is valid in PKI, but the AppServer AuthToken.Create path does not handle it correctly.
It uses the RSA private key path and calls RSACng.SignHash, but no RSASignaturePadding is selected. RSA signing requires padding.
Lab tests confirmed:
- RSA key and ECDSA certificate signature: fails (reproduces error)
- RSA key and RSA certificate signature: works
- ECDSA key and ECDSA certificate signature: works (with correct private-key ACLs)
Product documentation (10.0 LTS Installation Guide, Set session token certificate) supports pure RSA and pure ECDSA certificates with SHA1/SHA256/SHA384/SHA512 hashing.
It does not describe support for mixed RSA-key/ECDSA-signature certificates. -
Resolution
Replace the AppServer session token certificate with a consistent supported format:
- Option 1: RSA key with SHA256RSA or SHA512RSA certificate signature
- Option 2: ECDSA key with ECDSA certificate signature