[Previous] [Next] - [Index] [Thread Index] - [Previous in Thread] [Next in Thread]

Subject: Re: UKNM: Spidering frames
From: Paul Sanders
Date: Thu, 29 Apr 1999 15:29:47 +0100

>Well, if you have a very, very cunning database driven site, you
>could proably build a spiderable version of it which didn't have
>frames, for arachnid consumption.

Cunning sites are as much of a headache as frames if you want the spiders
to come crawling.

We struggled with this one - take a look at http://motion.state51.co.uk/ It
runs on Apache with mod_perl, nearly all dynamic, and faking HTTP headers.
It's a big fat fly as far as the spiders are concerned.

One really great unforseen benefit - if you are detecting agents before you
deliver the content you can deal with those evil EmailSiphon bastards and
their like. Another of our sites redirects them to a certain large software
company's site...



Paul Sanders - a member of the state51 conspiracy
email: paulatstate51 [dot] co [dot] uk tel: +171 729 4343
state51, Rhoda Street, London, E2 7EF, UK

UKNM is sponsored by Excite UK, visit us at http://www.excite.co.uk.
Email Khalil Ibrahimi khalilatexcitecorp [dot] com (mailto:khalilatexcitecorp [dot] com) to advertise on Excite.
Change your UKNM subscription use http://www.chinwag.com/uknm.html

  Re: UKNM: plug-in take up/stats, Ray Taylor
  UKNM: Spidering frames, Robert Hamilton (brandwid

[Previous] [Next] - [Index] [Thread Index] - [Next in Thread] [Previous in Thread]