Re: Barnyard2 fatal error duplicate references, but there are no duplicates

[beenph <beenph () gmail com>]

On Thu, Nov 1, 2012 at 7:57 AM,  <elof () sentor se> wrote:
> I just upgraded to barnyard2.1.10, and it complains...
>
>
> I dropped and created a brand new database so that there are no old
> garbage data.

> I pre-populated the reference-system with all the metadata like I've done
> for years.

Thats probably the reason why you are having issue.

> I ran barnyard2 in testmode, but it bails:
> barnyard2 in self-test mode:
> barnyard2 -T -v -c barnyard2.conf -d /log -f snort.unified2 --pid-path /var/run
> Found pid path directive (/var/run)
> Running in Test mode
>
>          --== Initializing Barnyard2 ==--
> Initializing Input Plugins!
> Initializing Output Plugins!
> Parsing config file "barnyard2.conf"
> Found pid path directive (/var/run)
> WARNING database: Defaulting Reconnect/Transaction Error limit to 10
> WARNING database: Defaulting Reconnect sleep time to 5 second
> Checking PID path...
> PID path stat checked out ok, PID path set to /var/run
> Writing PID "45577" to file "/var/run/barnyard2_mon0.pid"
> Chroot directory = /var/log/snort
> ERROR database: Query [SELECT ref_id FROM reference WHERE ref_system_id = '7' AND ref_tag =
> 'blog.fireeye.com/research/2010/10/feodosoff-a-new-botnet-on-the-rise.html#more';]
> returned more than one result
> [SystemCacheSynchronize()], Call to ReferencePopulateDatabase() failed
> [CacheSynchronize()]:, SystemCacheSyncronize() call failed.
> ERROR: database [DatabaseInitFinalize()]: CacheSynchronize() call failed
> ...
> Fatal Error, Quitting..
> Done. Cleaning up.
>
>
>
> So I guess the "returned more than one result" is the actual problem, and
> the following failed cache errors are just a result of it.
>
>
> Now, the strange thing is that there are no multiple results for this
> query!
> > From the same machine, I run 'psql', using the same configuration as
> barnyard2 use, and run the exact command as listed in the errorlog:
>
> psql -h 10.10.10.10 foo foo
> Password for user foo:
>
> foo=> SELECT ref_id FROM reference WHERE ref_system_id = '7' AND ref_tag =
> 'blog.fireeye.com/research/2010/10/feodosoff-a-new-botnet-on-the-rise.html#more';
>   ref_id
> --------
>     2872
> (1 row)
>
>
>
> There's just one result, so why do barnyard2 complain about "returned more
> than one result" ?

The database cache synchronize call appens in a transaction, thus if
something fail you will not have altered data
inserted in your database so this is normal.

> I guess the error message is related to some other query, not the one
> logged on screen.

No, its because of the behavior described above.

> So, apparently this specific url-based reference was _not_ duplicated.
>
> However, my old system that pre-populates the reference-system populates
> things just like they did a long time ago, before barnyard even existed
> and the database plugin was builtin in snort. I.e. it populates the same
> reference multiple times if several rules use the same reference:
> Example:
> The ref_id 7 = 'url'. This table unique.
> However, the following three rules would generate three rows in the
> reference table:
> alert udp $HOME_NET any -> any 53 (msg:"blah1"; ...
> reference:url,www.f-secure.com/weblog/archives/00002227.html; sid:2013481;
> rev:1;)
> alert udp $HOME_NET any -> any 53 (msg:"blah2"; ...
> reference:url,www.f-secure.com/weblog/archives/00002227.html; sid:2013482;
> rev:2;)
> alert udp $HOME_NET any -> any 53 (msg:"blah3"; ...
> reference:url,www.f-secure.com/weblog/archives/00002227.html; sid:2013483;
> rev:1;)
>
> ref_id   ref_system_id   ref_tag
> 17       7               www.f-secure.com/weblog/archives/00002227.html
> 18       7               www.f-secure.com/weblog/archives/00002227.html
> 19       7               www.f-secure.com/weblog/archives/00002227.html
>
> I'm not sure, but I can guess that the new barnyard2, in some other query
> that is not logged, complain about these kind of duplicates.

That shouldn't happen, do not pre-populate, let barnyard2 do it,
unless you can sort out duplicates
in your population script.

> 1)
> Should there only be only ONE 'www.f-secure.com/weblog/archives/00002227.html'
> in the database, and the signatures 2013481, 2013482 and 2013483 should
> all reuse it, not having their own ref_id instance?

Exact, there is no need for duplicates.

> 2)
> In the new barnyard2.1.10, you seem to have added some kind of
> pre-population of the reference system.
> Does this mean that there's no longer any need to pre-populate it using a
> separate system?
> If so that would be great - one system less to deal with.
> Also, if I'm correct in my assumptions above, the problem should not
> appear at all.

Yup it will populate all the required information like 2-1.9 did and
snort database output
plugin also did.

-elz

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Snort-devel mailing list
Snort-devel () lists sourceforge net
https://lists.sourceforge.net/lists/listinfo/snort-devel
Archive:
http://sourceforge.net/mailarchive/forum.php?forum_name=snort-devel

Please visit http://blog.snort.org for the latest news about Snort!