What Is Robots.txt? What to Block and What to Allow

There’s a tiny text file, just a few lines long, with the power to take your entire site off Google. It’s robots.txt. It sits at the root of your site, it’s the first place Google’s bot visits when it arrives, and a single wrong line in it can block all your content from search results. At the same time, configured well, it’s a useful tool for guiding crawling. Few files combine so much power with so much risk.

This guide explains what robots.txt is, what it does and doesn’t do, which directives actually work, the crucial difference between blocking and deindexing, and how to test before publishing. With templates you can adapt to your site.

What Is Robots.txt?

Robots.txt is a plain text file at the root of your site that gives search engine bots instructions about what they may or may not crawl. When Google’s bot reaches your site, this file is the first thing it checks, before any page.

Its job is to guide crawling, not indexing. That distinction seems subtle, but it’s the source of almost every serious robots.txt mistake, and we’ll come back to it in detail. For now, remember that robots.txt says where the bot can go, not what it should or shouldn’t show in results.

It’s a central piece of technical SEO, working alongside the crawler, the bot that travels the web. If the crawler is the visitor, robots.txt is the sign at the entrance saying which hallways are open and which are closed.

How Robots.txt Works

The file works through simple rules that combine three main elements.

The first is the user agent, which identifies which bot a rule applies to. You can create rules for all bots at once or for a specific one, like Google’s. In most cases, rules apply to all of them.

The second is the disallow directive, which indicates which paths the bot shouldn’t crawl. It’s how you stop the bot from wasting time on irrelevant areas of the site, like system folders or internal function pages.

The third is the allow directive, which opens exceptions inside a blocked area. It’s used to say “don’t enter this folder, except for this specific file”.

On top of those, robots.txt usually includes the address of your XML sitemap, which helps the bot find the full list of your pages on arrival. It’s a good practice that connects the two files smartly.

A simple, safe template for most sites looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.yoursite.com/sitemap.xml

The Difference Between Blocking and Deindexing

This is the most important point in the guide, and the one that causes the most dangerous confusion. Blocking in robots.txt is not the same as removing from Google.

When you block a page in robots.txt, you stop the bot from crawling it, that is, from reading its content. But that doesn’t necessarily remove it from results. If other sites link to that page, Google may keep showing its address in results without being able to read the content, displaying an empty result with no description. It’s the worst of both worlds: the page appears looking broken and Google doesn’t know what’s on it.

To actually remove a page from results, the path is different: a specific noindex tag inside the page itself, which only works if the bot can crawl the page. And here’s the trap: if you block the page in robots.txt, the bot never reads the noindex tag, and removal doesn’t happen. The two methods cancel each other out when used together by mistake.

The practical rule: use robots.txt to save crawling on areas with no value, and use the noindex tag to take specific pages out of results. Never rely on robots.txt to hide something from search results.

The Mistake That Wipes Out an Entire Site

There’s one configuration that, added by mistake, blocks your whole site at once. It’s a directive that stops every bot from crawling every page:

User-agent: *
Disallow: /

This happens more often than you’d think, for a specific reason: many sites in development use this total block on purpose so they don’t appear on Google while being built. The problem is forgetting to remove it when the site goes live. The site launches, everything looks fine, and weeks later comes the discovery that it never appeared on Google because the development block was still there.

That’s why the first thing to check on any newly launched site, or one that suffered an unexplained traffic drop, is robots.txt. It’s a thirty-second check that can explain a disaster. It’s a mandatory item in every SEO audit, precisely because it’s a common, serious and easy-to-miss mistake.

What to Include and What Not to Include

A good robots.txt is lean. The fewer rules, the lower the chance of error.

It’s worth blocking areas with no search value that would only waste crawling, like system admin folders, internal function pages and areas that generate pointless duplicate content. That helps direct Google’s attention to what matters.

It’s not worth blocking pages you want removed from results because, as we saw, robots.txt doesn’t do that. Nor should you block resources Google needs to understand the page, like certain style and script files, because without them the bot may see a broken version of your site and evaluate it poorly.

And, crucially, never use robots.txt as a security tool. Blocking a folder there doesn’t hide it from anyone who wants to access it; it just asks well-behaved bots not to go in. Sensitive information needs real protection, not a line in robots.txt.

How to Test Before Publishing

Given the file’s power, testing before applying it isn’t optional. A mistake here is costly.

Google Search Console offers tools to check how the bot interprets your robots.txt and to test whether a specific URL is blocked or allowed. Before uploading any change, simulate the behavior there and confirm that important pages remain accessible to the bot.

After publishing, check again. Open your robots.txt address directly in the browser, it’s always in the same place, at the site root followed by robots.txt, and read the contents calmly. Confirm that no rule blocks what should be open. That thirty-second final read prevents most disasters.

Frequently Asked Questions About Robots.txt

Does every site need a robots.txt?

It isn’t mandatory, but it’s highly recommended. Without it, the bot crawls everything it finds, which usually works for small sites. A well-made robots.txt gives you control over crawling and avoids waste, which makes a difference as the site grows.

Does blocking a page in robots.txt remove it from Google?

No, and that’s the most dangerous confusion. Blocking stops the bot from reading the content, but the page can still appear in results if there are links to it, just without a description. To actually remove it, use a noindex tag on the page itself, without blocking it in robots.txt.

Where does the robots.txt file go?

Always at the root of the site, reachable by adding robots.txt after your domain. That location is fixed: Google’s bot always looks there and nowhere else. A robots.txt in any other folder simply isn’t read.

Can I have a robots.txt on WordPress without touching code?

Yes. The main SEO plugins let you edit robots.txt directly from the WordPress dashboard, without accessing server files. It’s the safest path for most users, because it reduces the risk of handling errors.

What happens if I don’t have a robots.txt?

The bot assumes it can crawl everything. For small, well-structured sites, that’s rarely a problem. For large sites, the absence of robots.txt can lead to wasted crawling on irrelevant pages, leaving less attention for the content that matters.

Should I include the sitemap in robots.txt?

Yes, it’s a good practice. Pointing to your XML sitemap inside robots.txt helps the bot find the full list of pages on arrival, reinforcing what you’ve already submitted through Search Console. It’s one simple line that improves discovery.

Don’t Let One Line of Text Take Down Your Site

Robots.txt is small, but the damage it causes when misconfigured is huge. An entire site can become invisible on Google because of a single forgotten rule, and the cause often goes unnoticed for weeks.

Request an SEO audit from Malamute Digital and confirm your robots.txt is allowing what matters and blocking only what it should, with the care of a team that treats technical SEO as the foundation for everything else.