openssl.encrypt and openssl.decrypt with short keys and ivs

[David Fifield <david () bamsoftware com>]

I discovered that the openssl.encrypt and openssl.decrypt was reading
past allocated memory when given a key or iv that was too short:

openssl.encrypt("DES", "key", "iv", "datadatadatadata")

==5691== Invalid read of size 1
==5691==    at 0x412F07D: DES_set_key_unchecked (in /usr/lib/libcrypto.so.0.9.8)
==5691==  Address 0x4bcc415 is 2 bytes after a block of size 19 alloc'd
==5691==    at 0x402598A: malloc (vg_replace_malloc.c:195)
==5691==    by 0x4025A16: realloc (vg_replace_malloc.c:476)
==5691==    by 0x80ED502: luaM_realloc_ (in /home/david/nmap/nmap)
==5691==    by 0x80F134B: luaS_newlstr (in /home/david/nmap/nmap)
==5691==    by 0x80F85FA: luaX_newstring (in /home/david/nmap/nmap)
==5691==    by 0x80F8FF6: llex (in /home/david/nmap/nmap)
==5691==    by 0x80F9861: luaX_next (in /home/david/nmap/nmap)
==5691==    by 0x80EDCAF: testnext (in /home/david/nmap/nmap)
==5691==    by 0x80EF718: explist1 (in /home/david/nmap/nmap)
==5691==    by 0x80EF7C0: funcargs (in /home/david/nmap/nmap)
==5691==    by 0x80EFA1C: primaryexp (in /home/david/nmap/nmap)
==5691==    by 0x80EEE16: subexpr (in /home/david/nmap/nmap)

I added checks to prevent this in r17002. The key size is checked by
calling EVP_CIPHER_CTX_set_key_length and looking for an OpenSSL error.
The iv length is checked against EVP_CIPHER_CTX_iv_length().

I would appreciate authors of scripts and libraries that use these
function giving them a quick test.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/