[Previous] [Next] - [Index] [Thread Index] - [Previous in Thread] [Next in Thread]

Subject: UKNM: robots.txt - are you at risk?
From: Tim Ireland
Date: Tue, 27 Jun 2000 16:53:38 +0100

An interesting news article from The Register:

"Our friend 'fravia+' recommends searching for this file, called robots.txt,
in the main directory of a target site, by entering a URL with the following
pattern: http://www.targetsite.com/robots.txt. The robots.txt file is used
to tell search engines which directories and files they should not index.
Nothing listed in a 'robots.txt' file will turn up in a search query; but
once a person has seen the directory and file names it contains, they can
type them directly into their browser to access the various subdirectories
and pages which the site administrators would rather keep hidden. These are
of course the very subdirectories and files most likely to be of interest to

I had a bit of a poke around with this, and AFAICT the most common method
used to beat this amateur hack is to 'hide' this sensitive info in a folder
and instruct the robot not the enter it (rather than provide an exact file
name that it should avoid).

Of course if you have nothing to hide, you could borrow this little gem:

# Don't even bother trying to use our "robots.txt" file to hack us - we
# aren't that stupid!


Tim Ireland

the UK's totally managed affiliate marketing solution.
ukaffiliates.com >> the net.working
http://www.ukaffiliates.com / salesatukaffiliates [dot] com (mailto:salesatukaffiliates [dot] com)
telephone: 020 7691 1880 / fax: 020 7691 1881
To unsubscribe or change your list settings go to
http://www.chinwag.com/uk-netmarketing or helpatchinwag [dot] com

  Re: UKNM: robots.txt - are you at risk?, Craig Pickup

  RE: UKNM: Email list management ?, paul youlten

[Previous] [Next] - [Index] [Thread Index] - [Next in Thread] [Previous in Thread]