Risks with OpenSSL verifying a signature with un-trusted PEM encoded public key

Risks with OpenSSL verifying a signature with un-trusted PEM encoded public key

If a website user wants to use WebAuthn, they will start by creating a credential, where their authentication device provides a public key.

This key is encoded, and sent back to the server to store against their account.

Later, when the user needs to be verified, the website provides a challenge, their authentication device signs it, and that's sent back to the server.

Assuming PEM encoding of the public key (originally sent to the server from a potentially hostile user), are there any risks with this?

• Is it possible to cause a Denial of Service?
• Could it cause OpenSSL/PHP/Apache to crash?
• Use up too much memory, or take a long time to process?
• Provide a key format that OpenSSL does not understand, or get confused by, and return an unexpected result?

Some example questions I'm unsure about:

• If it's an Elliptic Curve (e.g. prime256v1), could it include excessively large x/y values?
• Is there a problem if a DSA key was provided?
• Is there a problem with an RSA key using PKCS1v1.5 padding?
• While RSA is normally 2048-bits, what happens if it's a 65,536 bit key?
• What about invalid DER encoding (e.g. wrong field lengths)?
• How about an invalid DER Object Identifier?
• Could invalid base64 encoding of the PEM data cause issues for OpenSSL?

I know a normal user would not do any of these things, and I accept that anyone who does provide a flawed public key would be affecting their own account, but could it cause other problems?

---

This is a basic implementation in PHP:

<?php

// PEM encoded public key, from hostile user
$key = '-----BEGIN PUBLIC KEY----- [...] -----END PUBLIC KEY-----';

// Other checks

$verify  = base64_decode($response['authenticatorData']);
$verify .= hash('sha256', base64_decode($response['clientDataJSON']), true);

$signature = base64_decode($response['signature']);

if (openssl_verify($verify, $signature, $key, OPENSSL_ALGO_SHA256) === 1) {
    // Success
}

?>

Note how openssl_verify() takes 3 values that came from the user.

And I could use openssl_pkey_get_details() to check the type, curve_name/oid, and x/y values.

---

In short, should the server be doing any additional checks on the public key?