[Previous] [Next] - [Index] [Thread Index] - [Previous in Thread] [Next in Thread]

Subject: Re: UKNM: robots.txt - are you at risk?
From: ben
Date: Wed, 28 Jun 2000 19:20:25 +0100

[Sam says: that's probably as much of robots.txt as anyone can take for the
moment, shall we draw a vale over this one (misspelling intended)]

Craig Pickup <c [dot] pickupatdigitrade [dot] co [dot] uk> wrote ..
> Correct me if I am wrong (which is often the case) but the search engine
> bots, spiders, etc will only index pages either specifically submitted or
> linked to from somewhere else. If it ain't linked to the bot won't find
> So if something is of such merit that you don't wish crackers to find it
> then don't link to it (or they will find it anyway) and you will not need
> to include it in the robots.txt file.

You're right but robots.txt allows you to specify certain directories as not
to indexed even though links may take the bot there.

It is these areas that you have to be careful about and ensure that either:-

1) they are password protected in some way of other (.htaccess)
2) Directory Indexing is switched off
3) The directory includes an index page (what it does, doesn't matter it
merely stops anyone else seeing the other stuff in the directory).

Remember we are assuming that kiddie hackers are looking at the robots file
and then going to the areas that are excluded.

Also be careful if like us, you do an awful large amount of on the fly
redirecting of requests. robots.txt operates on the URL version of the name
not the actual location make sure the security is correct on both versions.
(Thankfully for us, everything ends up in 1 of 6 directories so we know
whats going on but each case is different).


the UK's totally managed affiliate marketing solution.
ukaffiliates.com >> the net.working
http://www.ukaffiliates.com / salesatukaffiliates [dot] com (mailto:salesatukaffiliates [dot] com)
telephone: 020 7691 1880 / fax: 020 7691 1881
To unsubscribe or change your list settings go to
http://www.chinwag.com/uk-netmarketing or helpatchinwag [dot] com

[Previous] [Next] - [Index] [Thread Index] - [Next in Thread] [Previous in Thread]