Re: Hit Throttling - Content Theft Prevention

[Peter Conrad <conrad () tivano de>]

Hi,

Am Mittwoch, 19. Oktober 2005 09:03 schrieb Nik Cubrilovic:
> When you have content of high value at stake, the
> 'other side' seems to get more sophisticated as opposed to your
> standard home user who has downloaded a website scraper from
> download.com.

I think this is the root of the problem. You're publishing valuable
content. The word "publish" already implies that your content is
publicly visible. This means that what you're trying to achieve is
actually a paradox: you want to "protect" content that is already
visible to the general public. This in turn means that no solution
to your problem exists.

> What your tips are leading towards are ways to 
> distinguish human visitors from bots, which with some attackers simply
> leads to a game of cat-and-mouse as opposed to a solution that can be
> handed to the client.

Yup, and that is about the best you can achieve. Since you're already
publishing your valuable content, the best you can do is make it more
expensive for the attacker to "steal" it. The downside is, (as you 
found out) that raising the cost for the attacker usually turns away 
some of your legitimate users as well.

An upper limit for the attacker's cost could be estimated as the cost
for paying a number of dumb users who actually surf around on your site
through a logging proxy server. I guess that kind of labour is available
for little money in some parts of the world. If your content is more
valuable than that, you're lost - you cannot win the race.

> I have contacted a number of appliance vendors to see if they offer a
> transparent application-layer firewall that could identify bad bots
> and drop them, but surprisingly not one had a solution to offer.

I don't find that surprising. If a company came up with a technical
solution to the problem, an attacker could produce a bot that evades the
specific protection provided by that solution. The more wide-spread
such a solution would be, the more effort could be invested by an attacker
(because the payoff would be higher). Again, this is a race that noone
can win.

Bye,
        Peter
-- 
Peter Conrad                        Tel: +49 6102 / 80 99 072
[ t]ivano Software GmbH             Fax: +49 6102 / 80 99 071
Bahnhofstr. 18                      http://www.tivano.de/
63263 Neu-Isenburg

Germany