Fix for mysterious Zenmap crash under OS X 10.5: "could not open display"

[David Fifield <david () bamsoftware com>]

Hi,

I just (r10959) committed a workaround for a crash that appeared a few
times on the SourceForge bug tracker. It looked like

GtkWarning: could not open display

This long message is mainly aimed at debugging enthusiasts, but the
end-user summary is this: If you're running OS X 10.5 and you have
definitions of the DISPLAY environment variable in any of your shell
startup files (.profile, .bash_profile), remove them. This is no longer
necessary for Zenmap but it will make your other X11 applications work
better.

I finally managed to reproduce the crash, and the process is this: Under
OS X 10.5, edit ~/.profile and add the line "export DISPLAY=:0".
Double-click the Zenmap icon. Zenmap icon appears in the dock. X11 icon
appears in the dock. Zenmap crash reporter dialog opens with "could not
open display."

But wait, you ask, if Zenmap couldn't open the display, how did it open
the display to show the crash dialog? That's why I call this crash
mysterious. I suspect it's because of a bug or misdesign on Apple's
part.

The workaround is small, in zenmapGUI/App.py it just changes

    import gtk

to

    try:
        import gtk
    except Exception:
        import gtk

I'm the first to admit that's highly weird, so I included this big
comment:

    # On Mac OS X 10.5, X11 is supposed to be automatically launched on demand.
    # It works by setting the DISPLAY environment variable to something like
    # "/tmp/launch-XXXXXX/:0" and intercepting traffic on that socket; see
    # http://homepage.mac.com/sao1/X11/#four. However this breaks in a strange
    # way if DISPLAY is set in one of the shell startup files like .profile.
    # Those files only have an effect on the shell, not the graphical
    # environment, so X11 starts up as expected, but in doing so it reads the
    # startup scripts, and for some reason the first connection (the one that
    # caused the launch) is rejected. But somehow subsequent connections work
    # fine! So if the import fails, try one more time.

(You should know that "import gtk" implictly connects to the X server.)

This problem isn't limited to Zenmap. You can simulate it with xterm as
follows: Stop X11 if it is running. Remove DISPLAY from ~/.profile. Open
a terminal ("echo $DISPLAY" will give "/tmp/launch-XXXXXX/:0"). Add
"export DISPLAY=:0" to ~/.profile ($DISPLAY in the running shell is
unchanged). Run "xterm; xterm". The first xterm starts X11 but quits
with

    Xlib: connection to "/tmp/launch-XXXXXX/:0" refused by server
    Xlib: No protocol specified

    xterm Xt error: Can't open display: /tmp/launch-XXXXXX/:0

But the second xterm appears on the screen! Removing DISPLAY from
.profile and running the command works like you would expect: X11
starts, then two xterms, one after the other.

The following messages would appear in system.log (viewable in
Console.app) after the crash:

org.x.startx[62341]: AUDIT: Fri Nov  7 14:20:50 2008: 62408 X: client 2 rejected from local host (uid 501)
[0x0-0x17e17e].org.insecure.Zenmap[62337]: Xlib: connection to "/tmp/launch-HCRGPD/:0" refused by server
[0x0-0x17e17e].org.insecure.Zenmap[62337]: Xlib: No protocol specified

And in fact they will still appear in your logs, if you have DISPLAY
improperly set. It's just that the second X11 connection Zenmap makes
will make it work anyway.

It's worth noting that the shell startup files don't have an effect on
the Aqua graphical environment; DISPLAY will always be
"/tmp/launch-XXXXXX/:0" no matter what it is in ~/.profile. That's why
the auto-launcher still works, even if it rejects the first connection.
That's why it's baffling that it should affect the X server at all. My
best guess is that /System/Library/LaunchAgents/org.x.startx.plist, the
code behind the auto-launcher, starts X11 like
    /bin/bash --login -c /usr/X11/bin/startx
which would make it read the startup files. But why that would cause
this strange behavior is anyone's guess.

And note that none of this applies under OS X 10.4. X11 works completely
differently in that release.

David Fifield

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org