Wireshark mailing list archives
Re: Memory consumption in tshark
From: Evan Huus <eapache () gmail com>
Date: Wed, 28 Aug 2013 07:00:47 -0400
On 2013-08-28, at 1:22 AM, Anders Broman <a.broman () bredband net> wrote:
Evan Huus skrev 2013-08-28 00:17:On Tue, Aug 27, 2013 at 6:02 PM, Anders Broman <a.broman () bredband net> wrote:Joerg Mayer skrev 2013-08-27 23:24:On Tue, Aug 27, 2013 at 05:09:19PM -0400, Evan Huus wrote:IIRC, two-pass allows for most/all of the reassembly/request-response stuff, which we want to do sometimes. Any ideas why we have to keep some information indefinitely?Two-pass requires us to keep *all* the state around through the first pass so that it is available during the second pass (at which point it can be discarded). Even in single-pass mode, there is some state that we can't always immediately discard. If I see a fragment of a TCP message then it doesn't make sense to discard that until the other fragments have arrived and been reassembled. If I see a request, I probably need to keep state from that request until the response (which may never show up). We already do reassembly and a lot of other stateful work in single-pass mode. The only thing two-pass mode provides is the ability to "see the future" (for example, saying: this request has a response 5 packets later).So (assuming we really free everything we could already) could add a possibly configurable foresight horizon of 10000 packets. If a packet number is older than 10000 packets, forget it?I haven't really looked but we do keep the reassembled fragments around even in tshark as there is no mechanism to discard them selectively if run by tshark as opposed to wireshark and that's the really big memory eater I would think.This is the primary problem. Any state saved with p_add_proto_data, or the reassembly API, or the conversation API, or (god-forbid) globals is not being freed right now. Any memory allocated with seasonal/file scope is not being freed right now. We do free frame data, column data and misc. other bits and pieces, but the majority of the state we create is not freed until the end of the file. As Anders says, this is because we have no way right now to selectively discard it: much of the data is stored in a way that we can only get rid of all of it, or none. Part of that is because there was no se_free call (for which wmem_free is the convenient solution) and part of that is just because nobody's ever added it to, for example, the reassembly API. I'm sure there are some significant improvements we could make if somebody figures out how, but you also have to beware of bugs like 9027 [1] which is occurring precisely because we were trying to free old reassembly data. It turns out there are certain cases where that data, at least, could still be in use. The logic to selectively free state is not simple, and is often dissector-dependent. Cheers, Evan [1] https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=9027 P.S. Given the above it would actually be fairly easy to create a fully state-less tshark: make file-scope memory redirect to packet-scope, and make our stateful APIs (p_add_proto_data, reassembly, conversations) no-ops. I'm sure this will break some dissectors that use globals, but they should be fixed anyways :)make our stateful APIs (p_add_proto_data, reassembly, conversations) no-opsI'm not sure I get this, p_add_proto_data might work as a no-op in tshark's case but isn't conversation data reassembly, needed to fully dissect a "future" frame?
That would be the other downside of state-less tshark. You would get no reassembly, no request-response matching, no conversations etc. On the other hand, memory usage would never go much above baseline :)
A way to fix the reassembly case might be to store the fully reassembled data with p_add_proto_data rather than in a hash_table and in tshark's case discard it when finished with the frame this might have other benefits as well. Unfortunately it means changing all dissectors using reassembly but if the most frequently used ones where done it might fix most use cases ( TCP, HTTP...?).___________________________________________________________________________ Sent via: Wireshark-dev mailing list <wireshark-dev () wireshark org> Archives: http://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-request () wireshark org?subject=unsubscribe___________________________________________________________________________ Sent via: Wireshark-dev mailing list <wireshark-dev () wireshark org> Archives: http://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-request () wireshark org?subject=unsubscribe
___________________________________________________________________________ Sent via: Wireshark-dev mailing list <wireshark-dev () wireshark org> Archives: http://www.wireshark.org/lists/wireshark-dev Unsubscribe: https://wireshark.org/mailman/options/wireshark-dev mailto:wireshark-dev-request () wireshark org?subject=unsubscribe
Current thread:
- Re: Memory consumption in tshark, (continued)
- Re: Memory consumption in tshark Evan Huus (Aug 27)
- Re: Memory consumption in tshark Jakub Zawadzki (Aug 27)
- Re: Memory consumption in tshark Evan Huus (Aug 28)
- Re: Memory consumption in tshark Evan Huus (Aug 27)
- Re: Memory consumption in tshark Joerg Mayer (Aug 27)
- Re: Memory consumption in tshark Evan Huus (Aug 27)
- Re: Memory consumption in tshark Joerg Mayer (Aug 27)
- Re: Memory consumption in tshark Anders Broman (Aug 27)
- Re: Memory consumption in tshark Evan Huus (Aug 27)
- Re: Memory consumption in tshark Anders Broman (Aug 27)
- Re: Memory consumption in tshark Evan Huus (Aug 28)
- Re: Memory consumption in tshark Jakub Zawadzki (Aug 27)
- Re: Memory consumption in tshark Evan Huus (Aug 28)
- Re: Memory consumption in tshark Jakub Zawadzki (Aug 27)
- Re: Memory consumption in tshark Dario Lombardo (Aug 28)
- Re: Memory consumption in tshark Evan Huus (Aug 28)
- Re: Memory consumption in tshark Dario Lombardo (Aug 28)
- Re: Memory consumption in tshark Evan Huus (Aug 28)
- Re: Memory consumption in tshark Evan Huus (Aug 28)
- Re: Memory consumption in tshark Dario Lombardo (Aug 29)
- Re: Memory consumption in tshark Evan Huus (Aug 29)