UKNM: robots.txt - are you at risk?

[Tim Ireland]

An interesting news article from The Register:

http://www.theregister.co.uk/content/6/11174.html


"Our friend 'fravia+' recommends searching for this file, called robots.txt,

in the main directory of a target site, by entering a URL with the following

pattern: http://www.targetsite.com/robots.txt. The robots.txt file is used

to tell search engines which directories and files they should not index.

Nothing listed in a 'robots.txt' file will turn up in a search query; but

once a person has seen the directory and file names it contains, they can

type them directly into their browser to access the various subdirectories

and pages which the site administrators would rather keep hidden. These are

of course the very subdirectories and files most likely to be of interest to

crackers."


I had a bit of a poke around with this, and AFAICT the most common method

used to beat this amateur hack is to 'hide' this sensitive info in a  folder

and instruct the robot not the enter it (rather than provide an exact file

name that it should avoid).


Of course if you have nothing to hide, you could borrow this little gem:


# Don't even bother trying to use our "robots.txt" file to hack us - we

# aren't that stupid!


;o)


Tim Ireland

www.designercity.com

www.another.com

www.buymybook.co.uk



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 the UK's totally managed affiliate marketing solution.

 ukaffiliates.com >> the net.working

 http://www.ukaffiliates.com /

salesukaffiliates [dot] com (mailto:salesukaffiliates [dot] com)

 telephone: 020 7691 1880 / fax: 020 7691 1881

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  To unsubscribe or change your list settings go to

http://www.chinwag.com/uk-netmarketing or

helpchinwag [dot] com