I think that the leaner robots.txt file from Jeff Star is a good starting point for WordPress users as it specifies the most important folders that should be blocked from the search engines such as your admin area, includes directory and theme and plugin folders.
What is a Robots.txt file?
The Robots.txt File lets you control what search engine spiders should and shouldn’t crawl on your website. It’s a file that many WordPress users forget to add. For those who do, it can be unclear what folders should and shouldn’t be blocked.
There isn’t a standardised robots.txt file for WordPress. As a result, the contents of Robots.txt varies quite differently from site to site. The great thing about the robots.txt file is that it is publicly viewable so I thought it would be interesting to look at some popular WordPress websites and see how they have configured their robots.txt file.
Please bear in mind that whilst I have an understanding of how you create a robots.txt file and what properties are available (e.g. Allow, Disallow, Sitemap etc) when creating one; I don’t consider myself to be an expert on the Robots Exclusion Protocol or on search engine optimisation. What has become clear to me when researching this post is that there doesn’t seem to be a right or wrong robots.txt file. Most WordPress websites are blocking different directories from search engines and many website owners disagree about what you should and shouldn’t put in the file (or whether you should even have a robots.txt file at all).
WordPress Robots.txt file
Let’s start by looking at how the official home of WordPress handles search engine robots 🙂
WordPress.org actually has a fairly simple robots.txt file. The main focus seems to be on blocking search functionality and the RSS feed.
WordPress.com sites are a little different. All WordPress.com sites have the same robots.txt file. It defines the crawl delay for the IRLbot crawler and blocks a few folders such as the sign up and activation directories.
The Suggested Robots.txt File from WordPress hasn’t been updated in years though it is still a good indicator of what directories shouldn’t be crawled by search engines.
In help me write this article I checked the robots.txt file of dozens and dozens of WordPress websites. To my surprise, the only website I found to be using the suggested robots.txt file was John Chow. I’m surprised more top blogs haven’t used the example used on WordPress or at least a close variation of it.
Allow Everything
I came across a few large WordPress websites such as ProBlogger, SitePoint and Spotify who simply allow everything.
Block WP-Admin and WP-Includes
One of the most common robots.txt file I came across was simply to block crawlers from indexing the wp-admin and wp-includes folder. It’s used by BuddyPress, Digging Into WordPress, WebDesignerWall, WooThemes, StudioPress and DailyBlogTips.
Other Website Examples of Robots.txt files
There are millions of WordPress websites online. Whilst many have the same robots.txt file, a large number of websites have created something unique for themselves.
WP Mods
I can’t recall the original source/inspiration for the WP Mods robots.txt file. I suspect I started with an example I found online and then made some changes along the way. In the file I block folders like wp-admin, wp-includes and wp-content as well as all files that start with wp (e.g. wp-login.php and wp-register.php). Google images is blocked though Adsense is allowed and the sitemaps generated by Yoast SEO are denoted at the bottom.
Mashable
Mashable block the WordPress theme and plugin folders. They also block search engines from indexing the WordPress readme.html and rpc_relay.html. I found this a little strange as it’s much easier to just delete these files rather than block them (plus it’s not like hackers can’t get these files from downloading WordPress anyway). They also block a lot of other HTML files and folders where ad scripts are installed.
Matt Mullenweg
WordPress co-founder Matt Mullenweg has a simple robots.txt that allows Google Adsense and blocks search engines from Dropbox, contact page, login page and admin area.
Smashing Magazine
Apart from the Yoast generated sitemaps, Smashing Magazine blocks all crawlers from indexing their RSS feed. 5 specific crawlers are prevented from indexing anything from the site.
WP Beginner
The interesting thing about the WP Beginner robots.txt is that they only set rules for the Google crawler. Google Images is blocked from the wp-includes folder though Adsense is allowed. Wp-content and wp-admin have been blocked but wp-includes can still be crawled. Trackbacks, feeds and their gallery have been blocked too.
CopyBlogger
CopyBlogger have a simple robots.txt that blocks the feed and trackbacks from being indexed.
WPMU
WPMU have not specified any rules for crawlers to obey. The only thing they confirm is the location of their sitemap (last year Peter Handley from MediaFlow left a comment on SEORoundTable stating that he does the same).
No Robots.txt File
The first thing that a search engine robot looks for when it visits a website is the robots.txt file though many WordPress websites such as the theme store Elegant Themes and the personal blogs of Mark Jaquith and Tim Ferriss do not have one.
Is this necessarily a bad thing? I’m not sure. A year ago Google employee JohnMu gave the following advice to someone on Webmaster Central:
I would recommend going even a bit further, and perhaps removing the robots.txt file completely. The general idea behind blocking some of those pages from crawling is to prevent them from being indexed. However, that’s not really necessary — websites can still be crawled, indexed and ranked fine with pages like their terms of service or shipping information indexed (sometimes that’s even useful to the user :-)).
The main argument for not doing this seems to be that 404 errors are generated if a search engine crawler can’t find the robots.txt file.
Overview
As I noted at the beginning of this article, I am not an SEO expert so I am not sure what the best practice is for creating a robots.txt file for your website. I think that blocking important areas of your site such as your admin area, theme and plugin folders and trackbacks is a good idea. Plus, making sure that important files unique to your own website is recommended (e.g. www.site.com/mydocuments/).
There does seem to be a lack of consistency between websites on what the robots.txt should actually block and there are those that argue that is not needed at all.
How much importance do you place on the robots.txt file? Feel free to share a link to your robots.txt file (or code from it) in the comments area together with your reasons for blocking specific directories and files.
Thanks for reading 🙂
Kevin
