Snort mailing list archives
Re: Snort only partially alerting
From: Frank Calone <fc10011001 () gmail com>
Date: Fri, 19 Jul 2013 15:43:36 -0400
A thank you to all who helped to resolve this issue with partial alerting on events. It turns out that we had packet truncation (due to a large framing size) which was causing us to miss alerts. This was noticed when I ran snort in the foreground. The following message showed up: (snort_decoder) WARNING: IP dgm len > captured len We had an older Snort system that was alerting when our new one was not. The network tap is a vlan though I don't know if that contributed to the problem. Finding an external site where I was able to consistenly reproduce the problem helped because I was then able to do both tcpdump and snort binary captures. Comparing between the two showed clearly Snort was not processing everything. I think because there were so many truncated packets, the syslog supressed the alerts when I was running Snort as a Daemon. I actually had two issues to blame for partial alerting. The first fix was to set the packet size to the maximum with this setting -P 65535 The second fix was to tell Snort to ignore checksum errors -k none I checked twice this week to compare results and each time I had a 1 to 1 match between the two systems. Joel had advised of a better rule to use, so I am glad to say we are actually getting a few more hits than our old content rule was alerting on. Great stuff. Thanks for the help! Frank On Mon, Jul 15, 2013 at 1:32 PM, Joel Esler <jesler () sourcefire com> wrote:
Sounds good. Thanks. On Jul 15, 2013, at 10:56 AM, Frank Calone <fc10011001 () gmail com> wrote: Joel, Thanks much. I'm just so glad to have your tool finally working. Now hopefully I can get my boss the exe files he has been wanting for a long time. I will compare the alerts between the two systems this week and am hopeful everything matches. I will let you know how that goes. Thanks again!!! Frank On Fri, Jul 12, 2013 at 5:11 PM, Joel Esler <jesler () sourcefire com> wrote:If you remember, you were getting warnings: (snort_decoder) WARNING: IP dgm len > captured len The problem is that we had to figure out *why* we were getting those warnings. Turns out apparently that the device that is passing packets to your machine is going so in very large “jumbo” frames. While 99% of Snort users don’t have to deal with this, every once in awhile, we do find this issue. You just happened to be in that 1%. Annoying, I know. On Jul 12, 2013, at 4:46 PM, Frank Calone <fc10011001 () gmail com> wrote: Joel, Russ was able to determine the problem. I will forward you our correspondence if you want to see it (I left you off our correspondence as I am sure you have plenty just watching all the user group posts). Packet truncation was occuring. Using a setting of -P 65535 resolved the issue. Well, I hope it has fixed it anyways. I plan to compare the new server hits to the old one all of next week. If we match, I will be ecstatic! At that time I ought to post to the whole community the resolution (end of next week). I am troubled though by what I see as a weakness in your tool alerting me to this condition. I don't mean to be critical as there are a lot of complexities and this tool has tried to deal with all of them. It is not a simple thing to do. I think highly of what this tool does and all of your efforts. It does make a real difference in helping to protect networks which is an absolute must given the poor state of computing security today (it all starts with the sheer number of vulnerabilities each day/week/month. I am 30+ years of just IT security experience and I am disgusted with the state of IT security today. Just my personal opinion.). Even so, truncating packets is a serious problem and a error warning message should have shown up in bold letters somewhere and not just at shutdown of the snort process. For that matter, even an abort would have been better until the truncation was addressed in some fashion (say a forced override or parameter setting). What worries me is how many other Snort users may have packet truncation going on such that they too are only getting partial alerting and they don't even know it. Instead they think everything is being checked thoroughly and they are safe. Frank On Wed, Jun 26, 2013 at 3:42 PM, Joel Esler <jesler () sourcefire com> wrote:OH NO, HAIR! On Jun 26, 2013, at 3:41 PM, Frank Calone <fc10011001 () gmail com> wrote: I'd be glad to provide any other testing or results you might need to further diagnose what is happening, just let me know. In the end it will make your product better which is good for everyone. I have little hair left on my head at this time, hopefully it will grow back! Frank On Wed, Jun 26, 2013 at 3:39 PM, Joel Esler <jesler () sourcefire com> wrote::) I'll talk to development about it, and what we can do. -- *Joel Esler* Senior Research Engineer, VRT OpenSource Community Manager Sourcefire On Jun 26, 2013, at 3:36 PM, Frank Calone <fc10011001 () gmail com> wrote: The errors only showed up when I ran it in the foreground. Running it in the background (which is how I was running snort for the first few weeks) resulted in no warning messages in the syslog file. At the very least then you would think I would have gotten 1514 bytes, but that does not look to have happened either. Still seems like this should have aborted or alerted better. I guess I am thinking that after about 3-4 weeks of struggling with this, it came down to an issue of the capture length not being right. Why, I don't know. How that is handled by Snort I think is where there is room for improvement. Thanks again for all the help with this. I am relieved, well almost. Once I do some more testing and have the alerts show up, then I can celebrate. Frank On Wed, Jun 26, 2013 at 3:25 PM, Joel Esler <jesler () sourcefire com> wrote:Well, Snort won't know the size of the packets being presented to it. You *were* getting an error. The one about your IP dgm length being smaller than the packet received. but yeah, by default, Snort's snaplen is 1514. The packets you are presenting to Snort are apparently bigger than that. (Jumbo frames?) But yeah, you were receiving errors, we just had to figure out what was causing it. On Jun 26, 2013, at 3:20 PM, Frank Calone <fc10011001 () gmail com> wrote: Joel, So the million dollar question is what is going wrong such that the capture length must be hard coded in this way? Isn't there still a fundamental flaw somewhere? I see no real mention of this as an example of a command line option all users should be using to start Snort with. What is equally concerning is the lack of any error message in the syslog stating that the capture length is being truncated. If I hadn't had another snort server to compare my results against I would have thought everything is working fine. It did get occasional hits, at least initially. Just my opinion, but this kind of packet truncating ought to cause Snort to abort with a clear error message noting the problem. Otherwise, one might think they are doing monitoring when in reality most everything is going to a bit bucket someplace. Please don't take this the wrong way, I really like the Snort tool already and I've hardly begun using it. The out of the box startup though had the appearance of working when it was really failing misserably. I'll do some testing (downloads to make sure it is getting them). If it works, I'll post it to the entire group list. Frank On Wed, Jun 26, 2013 at 3:11 PM, Joel Esler <jesler () sourcefire com> wrote:Okay, that works. Make sure you start snort with that as well Snort -c /etc/snort.conf -P 9000 -l ./log -i p1p1 -D or whatever. On Jun 26, 2013, at 2:59 PM, Frank Calone <fc10011001 () gmail com> wrote: Joel, As requested. Looking better. Frank On Wed, Jun 26, 2013 at 2:48 PM, Joel Esler <jesler () sourcefire com> wrote:Try "snort -l ./logall -b -i p1p1 -P 9000 src 157.98.75.158 or dst 157.98.75.158" On Jun 26, 2013, at 2:37 PM, Frank Calone <fc10011001 () gmail com> wrote: Joel, I ran both captures at the same time and I downloaded putty.exe from the Internet. Here are the command lines I used and the respective output files. snort -l ./logall -b -i p1p1 src 157.98.75.158 or dst 157.98.75.158 tcpdump -i p1p1 -N -w tcpdump.jun26.v1.pcap src 157.98.75.158 or dst 157.98.75.158 Frank On Wed, Jun 26, 2013 at 2:27 PM, Joel Esler <jesler () sourcefire com> wrote:use the same bpf syntax you use for tcpdump. host 157.98.75.158 or whatever. On Jun 26, 2013, at 2:19 PM, Frank Calone <fc10011001 () gmail com> wrote: something is wrong with the command syntax- root@ehsseclp03 /var/log/snort $ snort -l ./logall -b -i p1p1 ip 157.98.75.158 Running in packet logging mode --== Initializing Snort ==-- Initializing Output Plugins! Snort BPF option: ip 157.98.75.158 Log directory = ./logall pcap DAQ configured to passive. Acquiring network traffic from "p1p1". ERROR: Can't set DAQ BPF filter to 'ip 157.98.75.158' (pcap_daq_set_filter: pcap_compile: syntax error)! Fatal Error, Quitting.. Frank On Wed, Jun 26, 2013 at 1:38 PM, Joel Esler <jesler () sourcefire com> wrote:You can filter to just an IP with Snort the same way you do with tcpdump. snort -l ./log -b ip 123.123.123.123 Can you send me the actual pcaps from both tcpdump and Snort on the same box, at the same time capturing the same thing? On Jun 26, 2013, at 1:35 PM, Frank Calone <fc10011001 () gmail com> wrote: Joel, Here are two samples of the snort -dvr playback of binary captures. The 2 MB capture represents the TCPDUMP pcap capture I made where I filtered for just my IP in either the src or dst, IP address. It shows the putty.exe download. The second playback is where I ran snort options to capture all packets into a binary file: ./snort -l ./log -b I wrote a quick PERL prgram to take the playback file and when it found my IP address, it took all output lines thereafter until it hit a delimiter (the =+=+=+=+ boundry lines). This way you can see what Snort is showing as the packets to be analyzed versus what TCPDUMP showed. The TCPDUMP program was included in the CENTOS OS that was loaded onto our Snort system. So the pcap capture was made on the same system monitoring the same interface (p1p1) as what Snort is running on. In looking at the Snort binary capture it becomes readily apparent there is a problem. Interesting that I do seem to get the initial http session request and server response. After that, virtually nothing other than headers. Frank On Wed, Jun 26, 2013 at 10:49 AM, Joel Esler < jesler () sourcefire com> wrote:BTW -- Here is an older presentation on Razorback: On Jun 25, 2013, at 6:04 PM, Frank Calone <fc10011001 () gmail com> wrote: I will look into what razorback offers. I do think though I will want to run most if not all of your rules at some time. Perhaps not to do individual captures on an alert, but rather just to be alerted so we can look at the web traffic and such to then determine if it is a real problem or not. At this time I'm just trying to get my boss what he wants, which is just exe captures. About what the networking guys are giving us, it sure looks like they are giving us all the traffic. The TCPDUMP testing I did sure looks just fine. the datagram sizes look normal and have payload. something is being translated (rather truncated) when Snort tries to look at it. I'm wondering if it is possible to have Snort do a full capture for my IP only. My boss might be okay with me sending that to you so you can see what I am seeing here. The size ought to be small enough to that I can just email. If this is an option, can you let me know what command to use. I can see how to do it with tcpdump, I'm not sure how to do it with Snort. Frank On Tue, Jun 25, 2013 at 5:54 PM, Joel Esler < jesler () sourcefire com> wrote:Sounds like the networking guys may be confused about what they are giving you. Or you aren't getting the full thing, or whatever. In any case it sounds like what you are trying to do is Razorback (We have a project for that!) http://labs.snort.org/razorback/ It sniffs all files off the wire (not only exes) and then judges them to be malicious or not through a variety of methods. -- *Joel Esler* Senior Research Engineer, VRT OpenSource Community Manager Sourcefire On Jun 25, 2013, at 5:50 PM, Frank Calone <fc10011001 () gmail com> wrote: What I am told is that the networking guys gave us a vlan tap. Our intent is to run the full Snort server on a standalone system rather than on the appliance. We were unable to get the logging to work on the appliance. We want to have session captures so when we get an alert, we can pull the exe file out of the session data. Then we can take the exe and run it on fireeye to see if it gets tagged as malicious. We can also check the md5 against known values that are malicious. So, having a log capture is really our end goal. I do think at some point we may turn up the other rules, just depending upon what that does to system load. My bosses main goal is to get the exe files first. We did have a nice java hit that I got from the emerging threats. There certainly is real value in using all the rules. We see a lot of exe downloads each day and the problem is knowing whether they are malicious or not. This is why he wants the session packet capture when an alert is triggered. I hope I answered your question. I'm the one that is supposed to be confused. I must be confusing you then. Frank On Tue, Jun 25, 2013 at 5:42 PM, Joel Esler < jesler () sourcefire com> wrote:And you are trying to capture it on the box you have sitting somewhere else? I'm confused. On Jun 25, 2013, at 5:37 PM, Frank Calone < fc10011001 () gmail com> wrote: It's an IBM Proventia 4000 or 5000 series systems. As far as the lan type being the same, I would say not the same. These devises are IPS units, so they are seeing the raw traffic in stream. Frank On Tue, Jun 25, 2013 at 5:31 PM, Joel Esler < jesler () sourcefire com> wrote:Yes, that's what that error means. (Packet says its bigger than it actually is). Is the box you are trying to sniff getting the same span as the appliance? What kind of appliance is it? On Jun 25, 2013, at 5:28 PM, Frank Calone < fc10011001 () gmail com> wrote: Turn snort on to do full packet captures, all sessions: ./snort -l ./log -b I've turned this on for like a minute or so, did my download, then did a playback of the capture and looked for just traffic tied to my PC's IP. I can see the initial get request and a reply, but after that it is almost nothing as far as a payload goes. If this is what snort is testing the rules against, it is no wonder I am getting no hits. The content string i'm testing for doesn't show up. So, when I say partial packets, I mean packets with no payload. I think this has something to do with the error message I saw when I did the snort testing in the foreground: (snort_decoder) WARNING: IP dgm len > captured len It simply looks like all of the payload is being truncated, which is what this error message seems to be telling me. Frank ps: Is the file-identify simply a category you use to put rules into, say much like "exploit-kit"? I really don't think the rules are the problem here. I've even tried using the exact same Snort rules my co-worker has to test for exe files. So, the rule works on our appliance, but not for me. Why has been this weeks old headache I have had. On Tue, Jun 25, 2013 at 5:11 PM, Joel Esler < jesler () sourcefire com> wrote:On Jun 25, 2013, at 5:10 PM, Frank Calone < fc10011001 () gmail com> wrote:Joel, What confuses me is if I turn on full packet capturemode, I ought to see complete traffic regardless of any rules. If what I am seeing in that mode is partial packets, then it sure seems any rules I may test for are going to be intermitent. I'm not sure how to set the file-identify rule category you note.What do you mean "turn on full packet capture mode" and what do you mean "partial packets"?<test.24.75.158.txt><testpcap.v3.jun20.txt><tcpdump.jun26.v1.pcap><snort.log.1372271318><snort.log.1372272837>
------------------------------------------------------------------------------ See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________ Snort-users mailing list Snort-users () lists sourceforge net Go to this URL to change user options or unsubscribe: https://lists.sourceforge.net/lists/listinfo/snort-users Snort-users list archive: http://sourceforge.net/mailarchive/forum.php?forum_name=snort-users Please visit http://blog.snort.org to stay current on all the latest Snort news!
Current thread:
- Re: Snort only partially alerting Frank Calone (Jul 19)
- Re: Snort only partially alerting waldo kitty (Jul 19)