What is a WordPress robots txt file?
Robots.txt is a simple text file stored in the root directory of your web server that instructs bots where they can and cannot go on a website. It is universal to all platforms, including websites powered by WordPress.
To view the robots txt file of any website just add /robots.txt to the domain’s URL. Google has a rather impressive robots.txt file.
The Robots Exclusion Protocol was originally developed in the early days of the web as a solution for controlling what parts of a website were off-limits to search engine crawlers and other web bots.
What is a bot anyway?
Bots (AKA web robots) are Internet software programs that visit websites. Search engine crawlers are a type of bot that visits (crawls) websites to index their content for inclusion in search engine results.
What is a robots txt file used for?
Robots txt files are most commonly used for restricting search engine crawler access to private or non-relevant website files like those located in admin directories and login URLs.
Other uses include limiting the frequency in which bots are allowed to crawl your website and optimizing crawl budgets so that bots are only focused on the most important pages of your site.
Where is robots txt located in WordPress?
The robots.txt file is stored in the same location for every website, including WordPress. Which is located in the root directory of your web server.
Is a robots txt file necessary for WordPress?
No. If a bot visits your website without a robots txt file in place it will crawl and index pages as it normally would.
Does a robots txt file prevent indexing by search engines?
No. The use of a noindex meta tag is the ONLY way to prevent crawler indexing of website content from search engines. Robots txt are essentially guidelines that bots can choose to ignore if they want to.
Google, for example, adheres to the majority of robots txt directives save for any commands that involve managing crawl frequency. Which tells search engine crawlers how often they are allowed to visit.
Google will also index disallowed content if it has acquired external links from other websites.
WordPress robots txt file SEO example
Here’s an example of a simple search engine optimized robots.txt file for a WordPress-powered website.
What the WordPress robots txt file means
It designates which bots (user-agents) the instructions apply to, what sections and pages of the website are accessible (allow/disallow), and the location of the website’s XML sitemap.
- User-agent: * – all web bots can access the website given the following directives
- Disallow: /wp-login.php – web bots are NOT allowed access to this URL
- Disallow: /wp-admin/ – web bots are NOT allowed access to any URLs in this directory
- Allow: /wp-admin/admin-ajax.php – web bots are allowed to access this URL located within a disallowed directory
- Sitemap: https://sitename.com/sitemap_index.xml – the URL location of the website’s XML sitemap
How do I test a robots txt file?
The most optimal method of testing your robots txt file is with Google’s robots.txt tester tool, which is available in Google Search Console for verified websites.
Managing WordPress robots txt SEO
An updated robots txt file assists search engine crawlers with indexing the most important content on your website. Keeping this file up to date often requires the assistance of web development to identify areas of the site that are off-limits to bots. The ongoing optimization and maintenance of a robots txt file for WordPress is a component of a successful managed SEO strategy.
We’re an SEO agency that has optimized websites for hundreds of organizations located throughout the US (and beyond) since 2009.
Like this post? Link here using the graphic below.