VNC authentication weakness

[jepler () unpythonic net]

VNC authentication weakness
---------------------------

VNC uses a DES-encrypted challenge-response system to avoid passing passwords
over the wire in plaintext.

However, it seems that a weakness in the way the challenge is generated by
some servers would make this useless.

The following program attempts to repeatedly connect to a vnc server and
prints the challenge string.

Against tightvnc-1.2.1_unixsrc, you'll see output like
$ python pvc.py somehost:1
4b24fbab355452b55729d630fcf73d43
b3acdf3fab422b7aa49b8d786f93def3
b3acdf3fab422b7aa49b8d786f93def3
b3acdf3fab422b7aa49b8d786f93def3
b3acdf3fab422b7aa49b8d786f93def3
88e37f1677c4e4f56eb2fa00a2804ded
88e37f1677c4e4f56eb2fa00a2804ded
88e37f1677c4e4f56eb2fa00a2804ded
88e37f1677c4e4f56eb2fa00a2804ded
[...]
each time the same string is printed twice in a row the server has
repeated a challenge.

WinVNC version 3.3.3R9 will display output more like
$ python pvc.py otherhost:0
Server declined connection
Server declined connection
91ff701f7dce8c6eebbc6062ffebcc6a
Server declined connection
Server declined connection
[...]
It appears that connects are rate-limited, even if the connects come
from two distinct machines.  This appears to foil the below attack on
VNC authentication.  (Whether this means there is a good DoS opportunity
against WinVNC is a separate question)

If your server will give the same challenge repeatedly, and you can
sniff somebody else's challenge and response, it appears that you could
authenticate without knowing the password simply by connecting within
the 1-second window to get the same challenge, and then send the same
response as the legitimate client.

Another weakness in the challenge is that it uses 'random()%256'.  Many 
implementations of random() have highly predictable low bits.  It's not
clear that this leads to as easy a compromise as the repeated challenge
problem, but it's something that warrants consideration..

On systems with /dev/urandom, the following function will give challenge
strings which should be immune to the problems discussed:

void
vncRandomBytes(unsigned char *bytes)
{
    int f;
    
    f = open("/dev/urandom", O_RDONLY);
    while(read(f, bytes, 16) != 16) ;
    close(f);
}

#------------------------------------------------------------------------
#   pvc.py -- check for weak vnc challenges
#------------------------------------------------------------------------
import socket, sys, time


def print_vnc_challenge(host, port):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect((host, port))
    f = s.makefile("r+")
    banner = f.readline()
    f.write("RFB 003.003\n")
    response = f.read(20)
    if response[:4] != "\0\0\0\2":
        print "Server declined connection"
        return
    challenge = response[4:]
    print "".join(map(lambda x: "%02x" % ord(x), challenge))

if len(sys.argv) > 1:
    host_port = sys.argv[1]
    if ":" in host_port:
        host, port = host_port.split(":")
        port = int(port) + 5900
    else:
        host, port = host_port, 5900
else:
    host, port = "", 5900

for x in range(20):
    print_vnc_challenge(host, port)