Re: RFD: The Future of Memory Management in Wireshark

[Sébastien Tandel <sebastien.tandel () gmail com>]

On Wed, Oct 24, 2012 at 11:13 AM, Evan Huus <eapache () gmail com> wrote:

> On Wed, Oct 24, 2012 at 8:10 AM, Sébastien Tandel
> <sebastien.tandel () gmail com> wrote:
> > On Wed, Oct 24, 2012 at 1:10 AM, Guy Harris <guy () alum mit edu> wrote:
> > > On Oct 18, 2012, at 6:01 PM, Evan Huus <eapache () gmail com> wrote:
> > >
> > > > I have linked a tarball [2] containing the following files:
> > > > - wmem_allocator.h - the definition of the allocator interface
> > > > - wmem_allocator_glib.* - a simple implementation of the allocator
> > > > interface backed by g_malloc and a singly-linked list.
> > >
> > > Presumably an implementation of the allocator could, instead of calling
> a
> > > lower-level memory allocator (malloc(), g_malloc(), etc.) for each
> > > allocation call, allocate larger chunks and parcel out memory from the
> > > larger chunks (as the current emem allocator does), if that ends up
> saving
> > > enough CPU, by making fewer allocate and free calls to the underlying
> memory
> > > allocator, so as to make it worth whatever wasted memory we have at the
> ends
> > > of chunks?
> >
> > One step further, instead of mempools, I think wireshark could have great
> > interest in implementing slabs (slab allocator). Slabs had initially been
> > designed for kernel with several advantages over traditional allocators
> in
> > terms of resources needed to allocate (CPU), (external / internal)
> > fragmentation and also cache friendliness (most of the traditional
> > allocators don't care). I've attached some slides about a high-level
> > description of slab.
> >
> > Since then, another paper has been written showing some improvements and
> > what it took to write a slab for user-space (libumem). There is another
> > well-known exampel out there, called memcache, that implements its own
> > version (and could be a good intial point for wireshark implementation,
> who
> > knows? :))
>
> If I understand correctly, a slab allocator provides the most benefit
> when you have to alloc/free a large number of the same type of object,
you're right, that's where slab is the most efficient at. Although, the
second paper shows it can be efficient for general purpose allocation based
on size and not specific structure.

but I don't know if this is necessarily the case in Wireshark. There
> are probably places where it would be useful, but I can't think of any
> off the top of my head. TVBs maybe? I know emem is currently used all
> over the place for all sorts of different objects...
I guess the most obvious would be emem_tree (emem_tree_node) might be an
example used all over and over while dissecting. :)
There is indeed a bunch of different objects allocated with emem.  Also, it
might be used to allocate memory for some fragments.

Since your interface seems to allow it, we could create several slabs
types, one for each specific structures that are allocated very frequently
(emem_tree_node?), others for packets/fragments with some tuned slabs sizes
and another with some generic sizes.


You could certainly shoehorn a slab allocator into wmem's current
> architecture, but you'd have to abuse the wmem_allocator_t interface
> to do it. I suspect that it would make more sense to implement slabs
> separately anyways - since their goal is primarily performance you
> would want to cut out the function pointers that wmem currently uses.
>
> It's definitely worth thinking about though.
>
> Thanks,
> Evan
> ___________________________________________________________________________
> Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
> Archives:    http://www.wireshark.org/lists/wireshark-dev
> Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
>              mailto:wireshark-dev-request () wireshark org
> ?subject=unsubscribe
___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
Archives:    http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request () wireshark org?subject=unsubscribe