Fix PKey.check for some broken keys (#897)

* fix PKey.check for some broken keys

RSA_check_key is documented to return 1 for valid keys.
It (currently) returns 0 or -1 for invalid ones.
The previous code accepted invalid keys if RSA_check_key returns -1!

* add test
2 files changed