Re: OS scanning causes send_ip_packet: sendto() EAGAIN errors (Linux raw socket issue?)

[ithilgore <ithilgore.ryu.l () gmail com>]

Brandon Enright wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hey all, especially the raw socket experts,
>
> I've been running into an issue where Nmap calls sendto() on a raw
> socket and receives EAGAIN (Resource temporarily unavailable).  I can
> get this to trigger any time I do a OS scan (-O) against a large group
> of hosts.
>
> For example, during a printer discovery scan with this command:
>
> $ sudo nmap -O -sV -v -d -T5 -PN -p 23,80,443,515,631,8000,8080,9100 --min-hostgroup 256 -n -iL cse.txt -oA 
> cse_printers.txt
>
> I got a whole bunch of:
>
> Initiating OS detection (try #1) against 256 hosts
> sendto in send_ip_packet: sendto(9, packet, 178, 0, x.y.10.182, 16) => Resource temporarily unavailable
> Offending packet: ICMP x.y.1.115 > x.y.10.182 echo request (type=8/code=0) ttl=50 id=19072 iplen=178 
> Sleeping 15 seconds then retrying
> sendto in send_ip_packet: sendto(9, packet, 328, 0, x.y.10.93, 16) => Resource temporarily unavailable
> Offending packet: UDP x.y.1.115:44903 > x.y.10.93:38709 ttl=52 id=4162 iplen=328 
> Sleeping 15 seconds then retrying
> sendto in send_ip_packet: sendto(9, packet, 328, 0, x.y.10.207, 16) => Resource temporarily unavailable
> Offending packet: UDP x.y.1.115:44903 > x.y.10.207:41664 ttl=52 id=4162 iplen=328 
> Sleeping 15 seconds then retrying
>
>
> A few weeks ago David and I looked into this and he pointed out that
> the "error" is EAGAIN which just means try again.  David suggested
> using --send-eth to bypass the raw socket and that, indeed, takes care
> of the problem.
>
> My best guess is that raw sockets have a relatively small send buffer
> and Nmap is filling the buffer before the OS can service it.
>
> I've tried finding documentation on raw socket buffers but have come up
> empty handed.  I'm also not seeing a way to tweak them in /proc or via
> sysctl.  I haven't looked into whether or not setsockopt(...,
> SO_SNDBUF, ...) works but I think it's worth a try.
>
> Can anyone shed more light on this issue?  Does anyone know how to
> tweak/see/test raw sockets and reproduce this error in a controlled way?
>
> Brandon
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.9 (GNU/Linux)
>
> iEYEARECAAYFAkmweI0ACgkQqaGPzAsl94LCcgCeN7+0LB/jqEBFTGxPLTxsyq6y
> JWEAn0F/MBemuvaZdPtssjkXg+op9NiY
> =b8Mh
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Sent through the nmap-dev mailing list
> http://cgi.insecure.org/mailman/listinfo/nmap-dev
> Archived at http://SecLists.Org


Brandon and I had a private talk over the issue, but I think it's better to
have the rest of the developers here know the findings so far. I might need
more people to post some test cases to track down the problem. What I have
found so far is the following:

The issue obviously has something to do with one of the kernel handlers that is involved with output.
Beginning with the system call send/sendto at net/socket.c at SYSCALL_DEFINE6(sendto ...) we
can see that sock_sendmsg() is called. sock_sendmsg() calls __sock_sendmsg() which in turn
calls sock->ops->sendmsg(). Now this needs a bit of explanation to see which function will be 
called next. So far, note that no error condition returns EAGAIN however.

Exploring the socket initialization procedure we delve into inet_create() at net/ipv4/af_inet.c
Assuming that a socket has been created with type = SOCK_RAW and protocol = IPPROTO_RAW, as happens
with the sockets from osscan2.cc and any raw socket that Nmap creates, we can see that at line 355
(2.6.28.7 version) sock->ops = answer->ops. answer is just a struct that defines the protocol, and in
our case it is inet_sockraw_ops defined in the same file (af_inet.c). Notice that this assigns:
        .sendmsg           = inet_sendmsg,

So we move on to inet_sendmsg() defined in the same file. Now, here is a strange point. This is the only
place that I have found so far that might return EAGAIN but according to my opinion this shouldn't normally
happen. Take a look:

int inet_sendmsg(struct kiocb *iocb, struct socket *sock, struct msghdr *msg,
                 size_t size)
{
        struct sock *sk = sock->sk;

        /* We may need to bind the socket. */
        if (!inet_sk(sk)->num && inet_autobind(sk))
                return -EAGAIN;

        return sk->sk_prot->sendmsg(iocb, sk, msg, size);
}

Normally, the 'if' check should fail before even calling inet_autobind(), since inet_sk(sk)->num
is assigned the protocol number if we deal with a raw socket, according to inet_create():

        if (SOCK_RAW == sock->type) {
                inet->num = protocol;
                if (IPPROTO_RAW == protocol)
                        inet->hdrincl = 1;
        }

Leaving that behind, we can see that sk->sk_prot->sendmsg() is actually the function raw_sendmsg()
from net/ipv4/raw.c for our raw socket case. This goes according to this: (defined in raw.c)

struct proto raw_prot = {
        .name              = "RAW",
        .owner             = THIS_MODULE,
        .close             = raw_close,
        .destroy           = raw_destroy,
        .connect           = ip4_datagram_connect,
        .disconnect        = udp_disconnect,
        .ioctl             = raw_ioctl,
        .init              = raw_init,
        .setsockopt        = raw_setsockopt,
        .getsockopt        = raw_getsockopt,
        .sendmsg           = raw_sendmsg,
        .recvmsg           = raw_recvmsg,
        .bind              = raw_bind,
        .backlog_rcv       = raw_rcv_skb,
        .hash              = raw_hash_sk,
        .unhash            = raw_unhash_sk,
        .obj_size          = sizeof(struct raw_sock),
        .h.raw_hash        = &raw_v4_hashinfo,
#ifdef CONFIG_COMPAT
        .compat_setsockopt = compat_raw_setsockopt,
        .compat_getsockopt = compat_raw_getsockopt,
#endif
};


Take a look at raw_sendmsg(). No EAGAIN there. raw_sendmsg() will later call raw_send_hdrinc()
since we prepend our own IP header (not letting the kernel mess with it) which is the last raw
socket function to be called. No EAGAIN there either. raw_send_hdrinc() calls dst_output() which
is the neighbour handler which means that we move to L2 leaving behind the upper level network functions.
I don't think dst_output() can cause a EAGAIN either, though I haven't checked that yet.

Another thing is that according to man 7 socket, EAGAIN can be the same as EWOULDBLOCK, so a search for 
it should be done too. However, no EWOULDBLOCK error has been found so far either.

I conducted some very simple printk-debugging in a 2.6.28 vanilla kernel and deduced that
indeed the inet_sendmsg doesn't return a EAGAIN. I am referring to the lines:

        if (!inet_sk(sk)->num && inet_autobind(sk))
                return -EAGAIN;

Indeed, num = 255 for IPPROTO_RAW and thus the check fails and inet_autobind() is never
called since C uses short-circuited checks.

Now for the most strange thing I encountered: I managed to reproduce the EAGAIN error *only* in a 2.6.28 (64 bit) 
kernel of the Arch Linux distribution (it's not a vanilla kernel) and *only* for Nmap version 4.76 (stable).

I conducted the same test on a Debian with a vanilla 2.6.28 kernel and neither Nmap 4.76 nor 4.85BETA3 produced
the error.

Nmap 4.85BETA3 on Arch Linux 2.6.28 did not produce the error. Neither the latest development version from svn.

Judging from the above, it might be an obscure end-case for some partially modified kernels or a combination
of values from /proc/sys/net. The strange thing is that on the same kernel only Nmap 4.76 produced the error.
I don't remember, however any serious changes being made at osscan2.cc from 4.76 to 4.85

It would be very helpful if some people posted the results from their own system, mentioning the kernel version, whether
it comes from a distribution or it is a vanilla, as well as their Nmap version. That way we may limit certain 
possibilities
and track down the problem more easily.

Cheers,
 ithilgore











_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org