[Previous] [Next] - [Index] [Thread Index] - [Previous in Thread] [Next in Thread]

Subject: Re: UKNM: Search Engines
From: Jamie Unwin
Date: Wed, 30 Sep 1998 17:40:21 +0100

>that detects spiders"; now, how would you do that with a hack if the
>spider does not request the robots.txt file?

By checking/matching the user agents ?

(1) Do the redirection by using the web server config file (possible on
ceritian web servers)
(2) Setup index.html as a script to redirect users/spiders, this script
outputs one of 2 redirect headers.

>Interceptor checks every request coming from Search Engines and
>distinguishes between spiders and people and performs different actions
>depending on who is requesting the files (any specified one!).

This is what the above does ?

>The best thing about Interceptor is that we charge per visitor delivered
>making it a totally result driven service.

Sorry, but I can't believe you are charging people for something so basic ?

>If you want more tech info give me a shout.

We set it up ourselves, this trick is as old as the hills !

Greenspun was talking about this whole thing in Database Backed Web Sites
ages ago, see below


Jamie Unwin : jamieatnml [dot] guardian [dot] co [dot] uk : http://www.guardian.co.uk
The Guardian : 3-7 Ray Street, London, EC1R 3DJ : Tel. 0171 713 4469
jamieatjamie [dot] net : http://www.jamie.net : http://www.poke.com

[Previous] [Next] - [Index] [Thread Index] - [Next in Thread] [Previous in Thread]