Re: Snort Timestamps Out of Sequence

[Russ via Snort-devel <snort-devel () lists snort org>]

Hi Alan,

This sounds like a flow stalled and was not flushed until it was 
pruned.  That can happen much later depending on the cache sizes and 
timeouts configured as well as packet arrival rate.  Are you seeing any 
prune counts?  Can you capture an offending session so we can look at it?

Also, are you just trying to understand what is going on or is there a 
problem resulting from the alert timing?

Russ

On 5/23/19 7:04 AM, ROTNEMER, ALAN H via Snort-devel wrote:
> Joel,
>
> Our customer was running into the 7-hour delay. We have a daemon 
> process that launches Snort and, via a Fifo pipe, sends packets to 
> Snort that were read in by another process that reads packets off the 
> Interface card.
>
> We get the alerts back using:
>
>        output alert_unified2: filename $OUT_FILE, vlan_event_types
>
> The alerts eventually make their way to either a MySQL or Hadoop 
> database. When we attempt the Hadoop ingestion, we spot an alert whose 
> timestamp is 7 hours prior to the previous one, and we log an snmp trap.
>
> When this problem was reported,  I saw the way to create the ALERT_CSV 
> file, and I saw that while most of the output came in timestamp order, 
> there were numerous instances where there were the “late” ones.
>
> I took our product out of the picture by running a separate instance 
> of Snort that got its input directly from the Interface card. The 
> delays weren’t quite as long – up to 30 minutes, but they happened.  I 
> ran this instance for about 2 hours. (The customer reported the 6-hour 
> delay after running Snort for a couple of days).
>
> Let me know if you need anything else to help diagnose this.
>
> Alan
>
> *From:* Joel Esler (jesler) <jesler () cisco com>
> *Sent:* Wednesday, May 22, 2019 4:11 PM
> *To:* ROTNEMER, ALAN H <ar435f () att com>
> *Cc:* snort-devel () lists snort org
> *Subject:* Re: [Snort-devel] Snort Timestamps Out of Sequence
>
> Alan,
>
> 7 /hours /certainly seems incorrect.  What is the output method?  
> Syslog?  Are you doing syslog output directly from Snort, or from 
> barnyard2?
>
> *From: *"ROTNEMER, ALAN H" <ar435f () att com <mailto:ar435f () att com>>
> *Date: *Wednesday, May 22, 2019 at 3:14 PM
> *To: *"Joel Esler (jesler)" <jesler () cisco com <mailto:jesler () cisco com>>
> *Cc: *"snort-devel () lists snort org 
> <mailto:snort-devel () lists snort org>" <snort-devel () lists snort org 
> <mailto:snort-devel () lists snort org>>
> *Subject: *RE: [Snort-devel] Snort Timestamps Out of Sequence
>
> Hey Joel,
>
> After looking at our packets and discussing this with my development 
> and analyst groups, I guess there is just one thing I need to know:
>
> From what you have said in your replies, there appears to be 
> “conditions” where Snort will receive a packet, and, for whatever 
> reasons, delay sending the alert. It could be something about the 
> packet, or something about the rule. My customer has had instances of 
> delays of up to 7 hours.  This ends up causing a bit of a problem on 
> our backend.
>
> Our product that processes the alerts coming from Snort does not 
> expect these delays. In fact, until this issue appeared, we always 
> thought that the alerts would arrive in the (timestamp) order they 
> were fed into Snort.
>
> Can you confirm, then, that Snort COULD delay alerts for some packets, 
> and, thus, it is possible that alerts will not be returned in 
> timestamp order?
>
> If yes, are there known circumstances where this could occur, and 
> would you be able to document them for us? Or describe the processing 
> that occurs within Snort that could lead to this situation?
>
> We can make some adjustments to our backend, but I want to be able to 
> explain this to my development group and our customer.
>
> Many thanks,
>
> Alan
>
> P.S. If you know of someone else I need to contact, could you let me know?
>
> *From:* Joel Esler (jesler) <jesler () cisco com <mailto:jesler () cisco com>>
> *Sent:* Monday, May 6, 2019 9:44 AM
> *To:* ROTNEMER, ALAN H <ar435f () att com <mailto:ar435f () att com>>
> *Cc:* snort-devel () lists snort org <mailto:snort-devel () lists snort org>
> *Subject:* Re: [Snort-devel] Snort Timestamps Out of Sequence
>
> Hey Alan,
>
> My "off the cuff" theory, without looking at your Snort configuration 
> and requesting a full traffic reassembly is that something was holding 
> the connection open (for 7 minutes) (keep-alive?) and Snort is 
> reassembling the HTTP session in the background into what we call a 
> "pseudo" packet.  A large reassembled stream.  That's what your rule 
> alerted on, and should have logged it to disk.
>
> --
>
> Joel Esler
>
> Manager, Communities Division
>
> Cisco Talos Intelligence Group
>
> http://www.talosintelligence.com 
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.talosintelligence.com&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=SF14kolGOfb9ES7hY9eg3w&m=cN5KAl1AsQe0jtiNGpL0uP5-1hkgqFQNgRED18eFkm4&s=29AEhc-8fQJZvQEt2b0d4cmzF8_G5pWBHKOucV-PHaQ&e=>
>
>
>
>     On May 6, 2019, at 9:16 AM, ROTNEMER, ALAN H <ar435f () att com
>     <mailto:ar435f () att com>> wrote:
>
>     Is there some explanation as to why the alert took over 7 minutes
>     to publish? Could Snort be waiting on anything in order to
>     complete the alert?
>
>
> _______________________________________________
> Snort-devel mailing list
> Snort-devel () lists snort org
> https://lists.snort.org/mailman/listinfo/snort-devel
>
> Please visit http://blog.snort.org for the latest news about Snort!

_______________________________________________
Snort-devel mailing list
Snort-devel () lists snort org
https://lists.snort.org/mailman/listinfo/snort-devel

Please visit http://blog.snort.org for the latest news about Snort!