Re: UKNM: Search Engines

[Jamie Unwin]

>that detects spiders"; now, how would you do that with a hack if the

>spider does not request the robots.txt file?


By checking/matching the user agents ?


(1) Do the redirection by using the web server config file (possible on

ceritian web servers)

(2) Setup index.html as a script to redirect users/spiders, this script

outputs one of 2 redirect headers.


>Interceptor checks every request coming from Search Engines and

>distinguishes between spiders and people and performs different actions

>depending on who is requesting the files (any specified one!).


This is what the above does ?


>The best thing about Interceptor is that we charge per visitor delivered

>making it a totally result driven service.


Sorry, but I can't believe you are charging people for something so basic ?


>If you want more tech info give me a shout.


We set it up ourselves, this trick is as old as the hills !


Greenspun was talking about this whole thing in Database Backed Web Sites

ages ago, see below



http://www.photo.net/wtr/thebook/publicizing.html


jamie

___________________________________________________________

Jamie Unwin :

jamienml [dot] guardian [dot] co [dot] uk

 :  http://www.guardian.co.uk

The Guardian : 3-7 Ray Street, London, EC1R 3DJ : Tel. 0171 713 4469

jamiejamie [dot] net

 : http://www.jamie.net : http://www.poke.com

____________________________________________________________