Re: Combatting automated download of dynamic websites?

[Eoin Keary <eoinkeary () gmail com>]

Hi,
As mentioned before,

Keep the pages visited in session with a time stamp.

If pages are skipped or being requested too fast or violating "normal
user" speeds then kill the session.

Upon each HTTP request check session for authentication,
authorization, history and time of last request. If any are incorrect
kill session.

This shall take a little coding and needs to be in a central location
to service all HTTP requests for the resource being throttled.

Eoin (OWASP Ireland)




On 05/09/05, Paul M. <gpmidi () gmail com> wrote:
> On 8/30/05, Michael Boman <michael.boman () gmail com> wrote:
> > On 8/30/05, Matthijs R. Koot <matthijs () koot biz> wrote:
> > > Thanks for your reply zeno! But actually, referer-based anti leeching
> > > won't do it for me and mod_throttle isn't suitable for Apache 2. I'm in
> > > need of a throttling function based on something more advanced like a
> > > 'request history stack' to check the order in which pages were
> > > requested, probably within a certain time period, et cetera. Maybe it'd
> > > be better to move such security measures into the actual web application
> > > itself, but I'm still hoping someone knows of a service-based solution
> > > (i.e. like the beforementioned Apache module).
> > >
> > > Matthijs
> >
> > How about placing a hidden link (around a 1x1 transparent pixel), and
> > get anyone who "clicks" on it banned?
>
> Blocking
> I would try the above expect they have to follow three of the 1x1s to
> get banned. And then perhaps they could only access a page saying that
> they have been IDed as a bot.
>
> Blackhole-ing
> If you wanted to be evil/sly you could have 5-10 of the hidden links
> added to every page. The links would go to a black hole type page with
> some random value passed to it (or in the path if you use modrewrite).
> That page would generate 10 random links back to the black hole page.
> Basically any bot that crawls your site would end up getting stuck.
> You may also need to increase the number of links per page from 10 to
> 100 or 500 to be more effective. That keeps any bots that load all
> pages in levels from the start page. ie loop/list vs recursive/tree
> based.
>
> ~Paul
>
>
> > Best regards
> >  Michael Boman
> >
> > --
> > IT Security Researcher & Developer
> > http://proxy.11a.nu