Fix signature buffer size for RSA keys
When using the pyOpenSSL crypto module to sign data using a large key,
e.g. 8192 bit, a memory allocation error occurs. A test case to show
this, which comes from OpenStack Glance, is:
```
$ openssl genrsa -out server.key 8192
$ ...
$ cat test.py
from OpenSSL import crypto
import uuid
key_file = 'server.key'
with open(key_file, 'r') as keyfile:
key_str = keyfile.read()
key = crypto.load_privatekey(crypto.FILETYPE_PEM, key_str)
data = str(uuid.uuid4())
digest = 'sha256'
crypto.sign(key, data, digest)
$ python test.py
*** Error in `python': free(): invalid next size (normal): 0x0000000002879050 ***
Aborted
```
Other errors that may appear to the user are:
```
Segmentation Fault
```
```
*** Error in `python': double free or corruption (!prev): 0x0000000001245300 ***
Aborted
```
```
*** Error in `python': munmap_chunk(): invalid pointer: 0x0000000001fde540 ***
Aborted
```
The reason this happens is that the sign function of the crypto module
hard-codes the size of the signature buffer to 512 bytes (4096 bits).
An RSA key generates a signature that can be up to the size of the
private key modulus, so for an 8192 bit key, a buffer for a 4096 bit
signature is too short and causes a memory allocation error.
Technically the maximum size key this code should be able to handle is
4096 bits, but due to memory allocation alignment the problem only
becomes apparent for keys of at least 4161 bits.
This patch does two things. First, it determines the correct size of
the signature buffer, in bytes, based on the real size of the private
key, and passes that the buffer allocation instead of the static number
512. Second, it no longer passes in a signature length. This is because
the OpenSSL EVP_SignFinal function uses this argument as an output and
completely ignores it as an input[1], so there is no need for us to set
it.
This is only a problem for RSA keys, and this patch only affects RSA
keys. For DSA keys, the key size is restricted to 1024 bits (128
bytes), and the signature a DSA key will generate will be about 46
bytes, so this buffer will still be big enough for DSA signatures.
[1] https://github.com/openssl/openssl/blob/349807608f31b20af01a342d0072bb92e0b036e2/crypto/evp/p_sign.c#L74
diff --git a/src/OpenSSL/crypto.py b/src/OpenSSL/crypto.py
index 6d78bd7..9a3a05a 100644
--- a/src/OpenSSL/crypto.py
+++ b/src/OpenSSL/crypto.py
@@ -2583,9 +2583,9 @@
_lib.EVP_SignInit(md_ctx, digest_obj)
_lib.EVP_SignUpdate(md_ctx, data, len(data))
- signature_buffer = _ffi.new("unsigned char[]", 512)
+ pkey_length = (PKey.bits(pkey) + 7) // 8
+ signature_buffer = _ffi.new("unsigned char[]", pkey_length)
signature_length = _ffi.new("unsigned int*")
- signature_length[0] = len(signature_buffer)
final_result = _lib.EVP_SignFinal(
md_ctx, signature_buffer, signature_length, pkey._pkey)