Nmap Development mailing list archives
Re: OS scanning causes send_ip_packet: sendto() EAGAIN errors (Linux raw socket issue?)
From: ithilgore <ithilgore.ryu.l () gmail com>
Date: Fri, 13 Mar 2009 01:12:50 +0200
Brandon Enright wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hey all, especially the raw socket experts, I've been running into an issue where Nmap calls sendto() on a raw socket and receives EAGAIN (Resource temporarily unavailable). I can get this to trigger any time I do a OS scan (-O) against a large group of hosts. For example, during a printer discovery scan with this command: $ sudo nmap -O -sV -v -d -T5 -PN -p 23,80,443,515,631,8000,8080,9100 --min-hostgroup 256 -n -iL cse.txt -oA cse_printers.txt I got a whole bunch of: Initiating OS detection (try #1) against 256 hosts sendto in send_ip_packet: sendto(9, packet, 178, 0, x.y.10.182, 16) => Resource temporarily unavailable Offending packet: ICMP x.y.1.115 > x.y.10.182 echo request (type=8/code=0) ttl=50 id=19072 iplen=178 Sleeping 15 seconds then retrying sendto in send_ip_packet: sendto(9, packet, 328, 0, x.y.10.93, 16) => Resource temporarily unavailable Offending packet: UDP x.y.1.115:44903 > x.y.10.93:38709 ttl=52 id=4162 iplen=328 Sleeping 15 seconds then retrying sendto in send_ip_packet: sendto(9, packet, 328, 0, x.y.10.207, 16) => Resource temporarily unavailable Offending packet: UDP x.y.1.115:44903 > x.y.10.207:41664 ttl=52 id=4162 iplen=328 Sleeping 15 seconds then retrying A few weeks ago David and I looked into this and he pointed out that the "error" is EAGAIN which just means try again. David suggested using --send-eth to bypass the raw socket and that, indeed, takes care of the problem. My best guess is that raw sockets have a relatively small send buffer and Nmap is filling the buffer before the OS can service it. I've tried finding documentation on raw socket buffers but have come up empty handed. I'm also not seeing a way to tweak them in /proc or via sysctl. I haven't looked into whether or not setsockopt(..., SO_SNDBUF, ...) works but I think it's worth a try. Can anyone shed more light on this issue? Does anyone know how to tweak/see/test raw sockets and reproduce this error in a controlled way? Brandon -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkmweI0ACgkQqaGPzAsl94LCcgCeN7+0LB/jqEBFTGxPLTxsyq6y JWEAn0F/MBemuvaZdPtssjkXg+op9NiY =b8Mh -----END PGP SIGNATURE----- _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://SecLists.Org
Brandon and I had a private talk over the issue, but I think it's better to have the rest of the developers here know the findings so far. I might need more people to post some test cases to track down the problem. What I have found so far is the following: The issue obviously has something to do with one of the kernel handlers that is involved with output. Beginning with the system call send/sendto at net/socket.c at SYSCALL_DEFINE6(sendto ...) we can see that sock_sendmsg() is called. sock_sendmsg() calls __sock_sendmsg() which in turn calls sock->ops->sendmsg(). Now this needs a bit of explanation to see which function will be called next. So far, note that no error condition returns EAGAIN however. Exploring the socket initialization procedure we delve into inet_create() at net/ipv4/af_inet.c Assuming that a socket has been created with type = SOCK_RAW and protocol = IPPROTO_RAW, as happens with the sockets from osscan2.cc and any raw socket that Nmap creates, we can see that at line 355 (2.6.28.7 version) sock->ops = answer->ops. answer is just a struct that defines the protocol, and in our case it is inet_sockraw_ops defined in the same file (af_inet.c). Notice that this assigns: .sendmsg = inet_sendmsg, So we move on to inet_sendmsg() defined in the same file. Now, here is a strange point. This is the only place that I have found so far that might return EAGAIN but according to my opinion this shouldn't normally happen. Take a look: int inet_sendmsg(struct kiocb *iocb, struct socket *sock, struct msghdr *msg, size_t size) { struct sock *sk = sock->sk; /* We may need to bind the socket. */ if (!inet_sk(sk)->num && inet_autobind(sk)) return -EAGAIN; return sk->sk_prot->sendmsg(iocb, sk, msg, size); } Normally, the 'if' check should fail before even calling inet_autobind(), since inet_sk(sk)->num is assigned the protocol number if we deal with a raw socket, according to inet_create(): if (SOCK_RAW == sock->type) { inet->num = protocol; if (IPPROTO_RAW == protocol) inet->hdrincl = 1; } Leaving that behind, we can see that sk->sk_prot->sendmsg() is actually the function raw_sendmsg() from net/ipv4/raw.c for our raw socket case. This goes according to this: (defined in raw.c) struct proto raw_prot = { .name = "RAW", .owner = THIS_MODULE, .close = raw_close, .destroy = raw_destroy, .connect = ip4_datagram_connect, .disconnect = udp_disconnect, .ioctl = raw_ioctl, .init = raw_init, .setsockopt = raw_setsockopt, .getsockopt = raw_getsockopt, .sendmsg = raw_sendmsg, .recvmsg = raw_recvmsg, .bind = raw_bind, .backlog_rcv = raw_rcv_skb, .hash = raw_hash_sk, .unhash = raw_unhash_sk, .obj_size = sizeof(struct raw_sock), .h.raw_hash = &raw_v4_hashinfo, #ifdef CONFIG_COMPAT .compat_setsockopt = compat_raw_setsockopt, .compat_getsockopt = compat_raw_getsockopt, #endif }; Take a look at raw_sendmsg(). No EAGAIN there. raw_sendmsg() will later call raw_send_hdrinc() since we prepend our own IP header (not letting the kernel mess with it) which is the last raw socket function to be called. No EAGAIN there either. raw_send_hdrinc() calls dst_output() which is the neighbour handler which means that we move to L2 leaving behind the upper level network functions. I don't think dst_output() can cause a EAGAIN either, though I haven't checked that yet. Another thing is that according to man 7 socket, EAGAIN can be the same as EWOULDBLOCK, so a search for it should be done too. However, no EWOULDBLOCK error has been found so far either. I conducted some very simple printk-debugging in a 2.6.28 vanilla kernel and deduced that indeed the inet_sendmsg doesn't return a EAGAIN. I am referring to the lines: if (!inet_sk(sk)->num && inet_autobind(sk)) return -EAGAIN; Indeed, num = 255 for IPPROTO_RAW and thus the check fails and inet_autobind() is never called since C uses short-circuited checks. Now for the most strange thing I encountered: I managed to reproduce the EAGAIN error *only* in a 2.6.28 (64 bit) kernel of the Arch Linux distribution (it's not a vanilla kernel) and *only* for Nmap version 4.76 (stable). I conducted the same test on a Debian with a vanilla 2.6.28 kernel and neither Nmap 4.76 nor 4.85BETA3 produced the error. Nmap 4.85BETA3 on Arch Linux 2.6.28 did not produce the error. Neither the latest development version from svn. Judging from the above, it might be an obscure end-case for some partially modified kernels or a combination of values from /proc/sys/net. The strange thing is that on the same kernel only Nmap 4.76 produced the error. I don't remember, however any serious changes being made at osscan2.cc from 4.76 to 4.85 It would be very helpful if some people posted the results from their own system, mentioning the kernel version, whether it comes from a distribution or it is a vanilla, as well as their Nmap version. That way we may limit certain possibilities and track down the problem more easily. Cheers, ithilgore _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://SecLists.Org
Current thread:
- OS scanning causes send_ip_packet: sendto() EAGAIN errors (Linux raw socket issue?) Brandon Enright (Mar 05)
- Re: OS scanning causes send_ip_packet: sendto() EAGAIN errors (Linux raw socket issue?) ithilgore (Mar 12)