Fix buffer overflow in RSA-PSS signature verification when the hash is
too large for the key size. Found by Seth Terashima, Qualcomm.

Added a non-regression test and a positive test with the smallest
permitted key size for a SHA-512 hash.