Re: RFD: The Future of Memory Management in Wireshark

[Evan Huus <eapache () gmail com>]

On Wed, Oct 24, 2012 at 8:10 AM, Sébastien Tandel
<sebastien.tandel () gmail com> wrote:
> On Wed, Oct 24, 2012 at 1:10 AM, Guy Harris <guy () alum mit edu> wrote:
> > On Oct 18, 2012, at 6:01 PM, Evan Huus <eapache () gmail com> wrote:
> >
> > > I have linked a tarball [2] containing the following files:
> > > - wmem_allocator.h - the definition of the allocator interface
> > > - wmem_allocator_glib.* - a simple implementation of the allocator
> > > interface backed by g_malloc and a singly-linked list.
> >
> > Presumably an implementation of the allocator could, instead of calling a
> > lower-level memory allocator (malloc(), g_malloc(), etc.) for each
> > allocation call, allocate larger chunks and parcel out memory from the
> > larger chunks (as the current emem allocator does), if that ends up saving
> > enough CPU, by making fewer allocate and free calls to the underlying memory
> > allocator, so as to make it worth whatever wasted memory we have at the ends
> > of chunks?
>
> One step further, instead of mempools, I think wireshark could have great
> interest in implementing slabs (slab allocator). Slabs had initially been
> designed for kernel with several advantages over traditional allocators in
> terms of resources needed to allocate (CPU), (external / internal)
> fragmentation and also cache friendliness (most of the traditional
> allocators don't care). I've attached some slides about a high-level
> description of slab.
>
> Since then, another paper has been written showing some improvements and
> what it took to write a slab for user-space (libumem). There is another
> well-known exampel out there, called memcache, that implements its own
> version (and could be a good intial point for wireshark implementation, who
> knows? :))

If I understand correctly, a slab allocator provides the most benefit
when you have to alloc/free a large number of the same type of object,
but I don't know if this is necessarily the case in Wireshark. There
are probably places where it would be useful, but I can't think of any
off the top of my head. TVBs maybe? I know emem is currently used all
over the place for all sorts of different objects...

You could certainly shoehorn a slab allocator into wmem's current
architecture, but you'd have to abuse the wmem_allocator_t interface
to do it. I suspect that it would make more sense to implement slabs
separately anyways - since their goal is primarily performance you
would want to cut out the function pointers that wmem currently uses.

It's definitely worth thinking about though.

Thanks,
Evan
___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
Archives:    http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request () wireshark org?subject=unsubscribe