As a website owner one of my main goals is to share relevant information with my readers. This is a very common goal amongst webmasters. If you are anything like me then you want that information to run on auto-pilot. Personally I like to let the search engines send me free targeted traffic through the use of highly optimized articles. Yeah I know, you’ve heard all this before.

We are going to cut straight to the chase and give you the details you need to get your articles indexed faster and get your website crawled deeper by your favorite search engine spiders. Yes, I am talking about a robots.txt file. This robots file is specifically for Wordpress blogs and it works great to keep those spiders on track to devour your content.

First let me recap what a robots.txt file is and what it does for those that may not know. A robots.txt file is a simple file you create with notepad that has a referrence to your sitemap and a list of files and folders. The job it performs is spectacular in getting your content indexed faster. Simply put, it tells spiders where NOT to go. Places like your javascript, css, and downloads folders. Basically anything that does not need to be indexed can be put in this file.

The problem is that most people do not know how to write a good robots.txt file and be certain that it is formatted correctly. If you get this wrong you could be telling the search engines NOT to index any of your website. That would be bad. So I will list a perfectly good robots.txt file you can copy and use for your Wordpress blog.

I suggest that you use it as is unless you know what you are doing. So here it is and I will explain what is going on below the file…

Sitemap: http://www.YOURSITE.com/sitemap.xml

User-agent: *
Disallow: /*.js
Disallow: /*.png
Disallow: /*trackback
Disallow: /*.css
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /tag/
Disallow: /author/
Disallow: /comments/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/uploads/
Disallow: /wp-content/themes/
Disallow: /wp-admin/
Disallow: /images
Disallow: /cgi-bin/
Disallow: /*?*
Disallow: /*?

Obviously the first line is a reference to your sitemap. This ensures that the search engine spiders will find it since your robots.txt file is the first thing they want to look at. All 3 major search engines support and use the robots.txt so it is an important aspect of seo. The second line, User-agent, is very important as you can use it to disallow specific search engines.

As you can see this file has an asterisk following User-agent. This is because on a robots.txt file the asterisk is a wildcard. So this particular file is set for any and all search engine robots. If you want to disallow certain search spiders just do a simple search on Google for a list of search robots then put the names where the asterisk is separated by commas.

Everything following the User-agent line is files and folders we are telling the search engines to stay away from. There is no useful content to be found there anyway so why waste precious crawl time on them. You probably noticed the question marks and dollar signs so I will briefly explain them.

The dollar sign is used to specify the end of a line. It will block the indexing of URLs that end with the selected file or folder. The same applies to the question mark in that it suppresses indexing of URLs that have question marks in them as they are usually conjured up by a database and are not really seo friendly.

I hope that helps you with your robots.txt file for a Wordpress blog and good luck. Just be sure to upload the file to your root directory (where your homepage is).

Thanx and remember…Always strive to improve and success will follow
David Knops Jr
DKWebsites LLC

• SEO Tips for Bloggers on WordPress There are several million WordPress blogs that you can find on the internet, but unfortunately the majority of them are not properly optimized for search...
• Best Wordpress Plug-Ins Blogging with Wordpress offers users so many choices, including a variety of plug-ins that meet different needs. Some are for fun and others are very...
• SEO For Beginners, Part 2 - Spiders Are People, Too OK, arachnids are not people, and search engine spiders are not really people, either. But what I'm driving at is that search engine spiders "think"...

Related posts:

1. Blogs Hit the Mainstream
2. Blogs-Everybody Loves Them
3. WordPress Rules the Blogging World
4. 2 Reasons Why Blogs are Better
5. Blogging with WordPress