Re: tvb allocator (was: Re: [Wireshark-commits] master b6d20a2: Optimize reseting epan_dissect_t when filtering.)

[Bálint Réczey <balint () balintreczey hu>]

2014-07-12 0:07 GMT+02:00 Anders Broman <a.broman58 () gmail com>:
> Den 11 jul 2014 23:13 skrev "Bálint Réczey" <balint () balintreczey hu>:
>
>
> > Hi All,
> >
> > Please provide the input data for letting others reproduce the results
> > or perform the performance tests on pcap files already available to
> > the public
>
> Ok I'll see if we can use something from the wiki instead.
>
> > I'm not a fan of implementing custom memory management methods because
> > partly because I highly doubt we can beat jemalloc easily on
> > performance and custom allocation methods can also have nasty bugs
> > like the one observed in OpenSSL:
> > http://www.tedunangst.com/flak/post/analysis-of-openssl-freelist-reuse
>
> We have gone through a set of memory allocation schemes already to try to
> improve performance (g_slice, emen and now wmem) are you saying that you are
> opposed to that?
No, IMO there is need for wmem for being able to keep memory allocated
during scopes,
but I would prefer seeing it only tracking and one-by-one freeing the
memory areas instead of doing bulk alloc()-s for optimizing for speed.

I tried to detail this in my answer to Evan's email.

> As wmem seems to be the faster scheme for packet scope memory allocation
> /free, why not use it in all possible places where  the scope is "packet"?
>
> > Please don't sacrifice protection for 2% speedup. Please keep wmem
> > usage for cases where it is used for garbage collecting (free() after
> > end of frame/capture file) not when the allocation and deallocation
> > are already done properly.
>
> ? A slow scheme might be working well but that in it self does not warrant
> to not replace it with a faster one. With this reasoning we shouldn't have
> introduced ep memory in the first place.
>
> What percentage if improvement do you think makes a change worthwhile?
>
> The set of improvements Jacub and I have done lately has given a reduction
> of 40-50 percent compared to 1.10 measuring with the sample file. The
> problem is that each improvement only yeald a percent or 2. Agreed that we
> shouldn't complicate the code for a small speed gain.
40-50% reduction is great and congratulations for such a speedup!
I hope memory allocation handling is responsible for only small
fraction of it because I would like to keep the possibility of
detecting memory allocation related errors and I would prefer using
tools implemented outside of Wireshark.
For example I would avoid bulk allocations to make us able to use ASAN
and Valgrind, even if we would implement our canaries.

> In your blog you say that people would accept double the execution time with
> increased security - I'm not so sure. If say the reformatting of a video
> takes one hour instead of 30 minutes.
In Debian you would be able to pick a slow and secure video player for
streaming from the untrusted Internet and a fast and less secure video
encoder.
I expect people to install Wireshark from the hardened-amd64
repository if they want to monitor a hostile network, while others
pick the fast Wireshark for using it in their safe labs.
So it depends and there will be good options to choose from.

Cheers,
Balint

> Just my 2 cents
> Anders
>
> > Thanks,
> > Balint
> >
> > 2014-07-11 8:58 GMT+02:00 Jakub Zawadzki <darkjames-ws () darkjames pl>:
> > > Hi,
> > >
> > > On Sat, Jun 21, 2014 at 10:12:48PM +0200, Jakub Zawadzki wrote:
> > > > If we're in topic of optimizing 'slower' [de]allocations in common
> > > > functions:
> > > >
> > > > - tvb allocation/deallocation (2.5%, or 3.4% when no filtering)
> > > >
> > > >    243,931,671  *  ???:tvb_new
> > > > [/tmp/wireshark/epan/.libs/libwireshark.so.0.0.0]
> > > >    202,052,290  >   ???:g_slice_alloc (2463493x)
> > > > [/usr/lib64/libglib-2.0.so.0.3600.4]
> > > >
> > > >    291,765,126  *  ???:tvb_free_chain
> > > > [/tmp/wireshark/epan/.libs/libwireshark.so.0.0.0]
> > > >    256,390,635  >   ???:g_slice_free1 (2435843x)
> > > > [/usr/lib64/libglib-2.0.so.0.3600.4]
> > >
> > > > This, or next week I'll try to do tvb.
> > >
> > > ... or maybe this week:
> > >
> > > ver0 | 18,055,719,820 (-----------) | Base version
> > > 96f0585268f1cc4e820767c4038c10ed4915c12a
> > > ver1 | 18,185,185,838 (0.6% slower) | Change tvb allocator g_slice_* to
> > > wmem with file scope
> > > ver2 | 17,809,433,204 (1.4% faster) | Change tvb allocator g_slice_* to
> > > wmem with file/packet scope
> > > ver3 | 17,812,128,887 (1.3% faster) | Change tvb allocator g_slice_* to
> > > simple object allocator with epan scope
> > > ver4 | 17,704,132,561 (2.0% faster) | Change tvb allocator g_slice_* to
> > > simple object allocator with file scope
> > >
> > > I have uploaded patches & profiler outputs to:
> > > http://www.wireshark.org/~darkjames/tvb-opt-allocator/
> > >
> > > Please review, and check what version is OK to be applied.
> > >
> > >
> > > P.S: I'll might find some time to do ver5 with slab allocator, but it'll
> > > look like object allocator API with epan scope.
> > >
> > > Cheers,
> > > Jakub.
> > >
> > > ___________________________________________________________________________
> > > Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
> > > Archives:    http://www.wireshark.org/lists/wireshark-dev
> > > Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
> > >
> > > mailto:wireshark-dev-request () wireshark org?subject=unsubscribe
> >
> > ___________________________________________________________________________
> > Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
> > Archives:    http://www.wireshark.org/lists/wireshark-dev
> > Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
> >
> > mailto:wireshark-dev-request () wireshark org?subject=unsubscribe
___________________________________________________________________________
Sent via:    Wireshark-dev mailing list <wireshark-dev () wireshark org>
Archives:    http://www.wireshark.org/lists/wireshark-dev
Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev
             mailto:wireshark-dev-request () wireshark org?subject=unsubscribe