The Easy Guide To Making a Robots.txt File
Posted: Monday, April 23, 2007
by Will Kalif
Kalif Publishing
If you have a website you really need to have a robots.txt file. It gives search engine spiders specific commands and it is easy to use and easy to maintain. Here is an easy guide to a robots.txt file in five minutes.
There are times when you don’t want a search engine to index a page or a folder on your website. Maybe you have some information you just don’t want to have show up in google. This may include your statistics page, a page of notes, or a dynamic page. And, importantly, if you use google adsense and the search tool that displays search results on your website google mandates you exclude this page from search engines. Which means they mandate you having a robots.txt file.
User-agent: *
Disallow:
The first line tells all spiders to listen up because the following command is for you. The second line means do not index any of the following pages. And it is here you put the url of any pages you don’t want spidered. So if you wanted the spiders to skip your private page it looks like this:
Disallow:/privatepage.htm
If you want the spiders to skip a whole folder you put the url of that folder with a slash like this:
Disallow:/privatefolder/
Simply place this text file in the root folder of your website and you are done. In the future you can add and remove commands easily.
The robots.txt file is a very easy file to write and maintain and it is a very powerful tool that will help you interact successfully with search engines. This disallow command is the simplest and most used command but there are also many other commands you can use and if you have a website it is well worth your time to have a robots.txt file and even to research it a bit further.
For more interesting insights into being a creative webmaster and making your website work for you visit the authors site at: The Creative Webmaster – Forging the Iron of Creativity on the Anvil of a Website
For more practical advice on how to earn money with your small website visit the authors tutorial website at: Earning Money with your small website
This Article has been viewed 300 times. (Not updated in real-time.)
No comments yet.We want your comments! If you can read this, you don't have javascript enabled, so you can't use this comment system. Please enable javascript.