GitLab

calculation of RR.

pkcs11_sign() reuses *sig to store the header and hash, but those might
be larger than the actual sig, causing a buffer overflow.

An overflow can occur when using raw sigs with hashlen > siglen, or when
the RSA key is less than 664 bits long (or less when using hashes
shorter than SHA512)

As siglen is always within the 'low realm' < 32k, an overflow of asnlen
+ hashlen is not possible.

Introduced in 5246ee5c