Re: [Nsock] Scalable event handling

[David Fifield <david () bamsoftware com>]

On Sat, Jun 15, 2013 at 09:13:46AM -0400, Henri Doreau wrote:
> Longer version:
> I'm working on cleaning and optimizing nsock, for the fun of it and
> because of the plans we have to add features (like proper ssl proxy
> support) and to eventually use nsock for connect scan. I'd eventually
> like to introduce a hooking framework into nsock, so that both proxy
> handlers and SSL code could transparently intercept and overload
> connect/read/write operations. Overloading would consist in adding
> events internally.
>
> The problem is that adding many events is currently quite expensive.
> You can easily observe it by stressing nsock a bit, via nping[1] for
> instance. The reason why it's that slow is because each iteration of
> the runloop goes through _all_ events and check for completion (I/O or
> timeout).
>
> This big patch (it'd be complicated to split it...) I'm proposing here
> makes nsock iterate over the active events only. IOW only IODs that
> either have I/Os to handle or expired events are processed at each
> iteration. I had to change the internal data structures for that:
>
>   * Introduced a gh_heap data structure, to efficiently make priority
> queues. All "expirables" (timers and timeouts) are now handled via
> such a tree. So that we don't need to iterate over _all_ elements at
> each nsock loop, only the active I/O events and the expired timers and
> I/O events are processed. Needless to say that the difference is
> noticeable when you have tens of thousands of events queued!
>
>   * Replaced gh_lists containers by embedded objects (intrusive data
> structure). All operations are inline, and adding an item to a list
> doesn't allocate extra memory. It's needed so that we can manipulate
> the events lists simply from events that we extract from the gh_heap.

Thanks Henri. This looks like a high-quality, worthwhile change as
always. I tried it and the TCP flood against localhost goes faster as
you say. gprof shows that epoll_loop stops dominating the profile after
the change.

+ #if !defined(container_of)
+ #define container_of(ptr, type, member) \
+         ((type *)((char *)(ptr)-(char *)(&((type *)0)->member)))
+ #endif

Perhaps this should use offsetof? offsetof appears to already be used in
the code of a few dependencies.

+   gh_lnode_t nodeq;
+   gh_lnode_t nodeq2; /* used for pcap events */

Can you elaborate on how nodeq and nodeq2 work? From what I understand,
nodeq2 is used for pcap-related events and nodeq is used for other kinds
of events. Why do these need to be separate? I wonder if there is a more
orthogonal way to do it. first_ev_next taking a nodeq2 parameter to
choose which list to work on is a little strange.

run_tests.sh is failing for me:

started UDP listener on port 55234 (pid 5694)
started TCP listener on port 55235 (pid 5696)
started TCP SSL listener on port 55236 (pid 5697)
nsock pool user data                            [OK]
test timer operations                           [OK]
set standard log levels                         [OK]
check error log levels                          [OK]
simple tcp connection                           [FAILED] (Connection refused)

> Patch generated via git, let me know if you prefer another format or
> if you want me to open a dedicated branch on nmap-exp.

It looks like a "git show" patch; "git format-patch HEAD^" is better
because it's easier to apply with "git am".

David Fifield
_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/