Re: Combatting automated download of dynamic websites?

["Paul M." <gpmidi () gmail com>]

On 8/30/05, Michael Boman <michael.boman () gmail com> wrote:
> On 8/30/05, Matthijs R. Koot <matthijs () koot biz> wrote:
> > Thanks for your reply zeno! But actually, referer-based anti leeching
> > won't do it for me and mod_throttle isn't suitable for Apache 2. I'm in
> > need of a throttling function based on something more advanced like a
> > 'request history stack' to check the order in which pages were
> > requested, probably within a certain time period, et cetera. Maybe it'd
> > be better to move such security measures into the actual web application
> > itself, but I'm still hoping someone knows of a service-based solution
> > (i.e. like the beforementioned Apache module).
> >
> > Matthijs
>
> How about placing a hidden link (around a 1x1 transparent pixel), and
> get anyone who "clicks" on it banned?

Blocking
I would try the above expect they have to follow three of the 1x1s to
get banned. And then perhaps they could only access a page saying that
they have been IDed as a bot.

Blackhole-ing
If you wanted to be evil/sly you could have 5-10 of the hidden links
added to every page. The links would go to a black hole type page with
some random value passed to it (or in the path if you use modrewrite).
That page would generate 10 random links back to the black hole page.
Basically any bot that crawls your site would end up getting stuck.
You may also need to increase the number of links per page from 10 to
100 or 500 to be more effective. That keeps any bots that load all
pages in levels from the start page. ie loop/list vs recursive/tree
based.

~Paul


> Best regards
>  Michael Boman
>
> --
> IT Security Researcher & Developer
> http://proxy.11a.nu