To disallow all crawling, your robots.txt file needs just two lines: User-agent: * followed by Disallow: /. To allow all crawling, you leave the Disallow value empty, add Allow: /, or skip the file altogether. Both setups live in a plain text file at your site’s root, where a single slash decides whether bots see everything or nothing.
At V Digital Services, we have spent over a decade managing technical SEO for clients across dozens of industries, and a misconfigured robots.txt is one of the most common problems we fix. Our web development services cover the full crawl-to-conversion path, so a single text file never quietly drains your traffic. If your robots.txt needs a second opinion, schedule a consultation with us today before a stray rule pulls your pages from search.
This article covers robots.txt syntax, the exact Disallow All and Allow All setups, common restrictions and their SEO impact, how to handle AI crawlers, and how to test the file before it goes live.
Understanding Robots.txt Syntax: Disallow and Allow Directives
A robots.txt file is a plain text file that tells search engines which parts of your site they may crawl. It lives in the root directory of your domain, so the address is always yourdomain.com/robots.txt. Put it anywhere else, and search engine crawlers will not find it. The filename must be strictly lowercase, robots.txt; capitalize a letter, and crawlers treat it as missing. You can write it in any plain text editor.
The format follows the Robots Exclusion Protocol, a convention web crawlers have honored since 1994. That convention became an official IETF standard, RFC 9309, published in September 2022, roughly 28 years after the original idea took hold. The standard added formal definitions, error handling, and caching rules.
Every file is built from a few directives, and each one goes on a separate line. The table below is a quick reference to keep open while you edit.
| Directive | What it does | Example |
|---|---|---|
| User-agent | Names the bot a rule block applies to | User-agent: Googlebot |
| Disallow | Blocks a path from crawling | Disallow: /private/ |
| Allow | Permits a path inside a blocked folder | Allow: /private/page.html |
| Sitemap | Points crawlers to your XML sitemap | Sitemap: https://example.com/sitemap.xml |
| * | Wildcard: matches zero or more characters | Disallow: /*? |
| $ | Anchors a rule to the end of a URL | Disallow: /*.pdf$ |
The User-agent line starts every group and decides whose crawler behavior the rules govern. User-agent: * allows all search engine bots to follow the rules in that group. You can also name a specific user agent, or specific bot, by token: a Googlebot Disallow rule targets Google’s crawler alone, and a Bingbot Disallow rule targets Bing. Writing rules for multiple user agents just means multiple groups, and only one group per user agent is valid.
The Disallow directive blocks specific URLs from being crawled. A disallow rule tells a web crawler to skip the listed paths. That changes its crawl behavior. A robots.txt Disallow only blocks crawling, which is how you discourage search engines from fetching a path without deleting it. To close your whole site, you point it at the root with Disallow: /; to open everything, you leave the value empty, Disallow: with nothing after it. Those two lines are the basis of Disallow All and Allow All, broken down in their own sections below.
The Allow directive permits crawling of specific files inside a folder you have otherwise blocked. When an Allow rule and a disallow rule both match a URL, Google follows the most specific rule, which usually means the longer path wins. RFC 9309 calls it “the most specific match,” measured by character count. On an exact tie, Google uses the least restrictive rule.
Wildcards let you match URLs by pattern instead of listing them. The * matches any run of characters, so Disallow: /*? blocks any URL with a query string. The $ marks the end of a URL, so Disallow: /*.gif$ blocks .gif files and nothing past that extension.
Two details cause silent syntax mistakes. First, case sensitivity applies to directory values, so /Page and /page are different URLs to a crawler, even though field names like “User-agent” are not. Second, the path value must start with a slash. The Sitemap directive is the exception to the group rules: it points crawlers to your XML sitemap and works on its own line, independent of any user agent. Keep the file lean, too, because Google enforces a 500 KiB size limit and ignores anything past it. For auditing the rest of your stack, our roundup of the best SEO tools is a useful starting point.
How to Set Up a Robots.txt File
Here is the process we recommend at V Digital Services. Our team follows these five steps whenever we build or fix a file, and they help avoid the mistakes that pull pages out of search.
Step 1. Check Whether Your Site Already Has a Robots.txt File
First, confirm a file does not already exist. In your browser’s address bar, add “/robots.txt” to the end of your domain, like this: www.example.com/robots.txt. If a blank page appears, you have no file yet. If a page listing instructions shows up, one is already live, so edit that instead of starting over. Many platforms ship a default; on WordPress, plugins like Yoast or Rank Math manage a virtual file you adjust from the WordPress dashboard.
Step 2. Decide Your Overall Goal
One of the biggest benefits of a robots.txt file is scale. You can allow or disallow many pages at once, without manually touching each page’s code. There are three basic goals, each with a clear outcome:
- Allow All: Every search engine robot may crawl all content. Because URLs are allowed by default, a full allow is usually unnecessary.
- Disallow All: no robot may crawl any content, which blocks Google’s crawlers from every part of your site.
- Conditional allow: you set rules that block certain URLs while leaving the rest open.
Once you pinpoint the outcome you want, you are ready to write the file. The dedicated Disallow All and Allow All sections below break down each setup in depth.
Step 3. Write Your Allow and Disallow Rules
Two elements do the heavy lifting. The user-agent names the specific search engine robot to which a rule applies. The disallow line names the URLs or files you want to block from that robot. Each sits on its own line, and a single file can hold many entries.
Use the user-agent line to name one bot, such as Google’s Googlebot, or an asterisk to cover them all: User-agent: *. The disallow line then sets the limit. A lone forward slash (Disallow: /) blocks the whole site. A slash plus a path blocks something specific — Disallow: /bad-directory/ blocks a directory and its contents, while Disallow: /secret.html blocks a single page. Put together, a simple entry reads:
User-agent: * Disallow: /bad-directory/
Give every URL its own line. Crowd several onto one line, and crawlers cannot separate them. For more patterns, Google’s official robots.txt documentation lists a wide range of example entries.
Step 4. Save and Upload the File
Once your entries are done, save the file with care:
- Copy it into a plain text or Notepad file, then Save As “robots.txt” using only lowercase letters.
- Place it in the highest-level directory of your site, inside the root domain, with the exact name
robots.txt. - Keep the structure in order — User-agent, then Disallow or Allow, then Sitemap — so robots read the rules in sequence.
- Subdomains need their own files. For example,
blog.domain.comanddomain.comeach require a separate robots.txt.
Step 5. Test the File Before You Rely on It
Confirm the file works before you trust it. Google retired the standalone robots.txt Tester, so the current checks live in Google Search Console:
- Open the robots.txt report (under Settings). It shows the file Google fetched for each host, the last crawl time, and any errors or warnings — scan for red flags first.
- To check one URL, paste it into the URL Inspection tool at the top of Search Console and press Enter. Under crawl details, it tells you whether the page is “Blocked by robots.txt,” which is the modern version of the old ACCEPTED/BLOCKED readout.
- If a URL is blocked when it should be open (or open when it should be blocked), edit robots.txt on your server, save it, then click “Request a recrawl” in the report so Google re-fetches.
One reminder the old tool gave that still holds: edits only count once they are saved to your live file on the server. The report reflects what is actually deployed, not a draft.
How to Use Robots.txt to Disallow All Pages
Disallow All blocks every cooperating bot from crawling your whole site. You write it with a wildcard user agent and a single slash:
User-agent: * Disallow: /
That one slash is the most powerful line in the file. The User-agent: * targets all the pages for every bot, and Disallow: / blocks the entire website at the root. The impact is total: search engine crawlers stop fetching your content.
This is the right call for a staging site you are still building, where you do not want an unfinished version in Google search results. It also fits a private internal tool. A fast, clean staging build is worth protecting while you work, and our guide to improve web page load time covers the performance side.
On a live site, treat Disallow All as a loaded gun. Blocking search engines entirely can remove your site from search results and cause significant traffic loss as Google drops pages it can no longer crawl. It happens by accident more often than you would think: a developer pushes a staging file to production and the whole site goes dark in search within days. Read this line twice before you save.
When to Use Disallow All vs No Robots.txt
A blank robots.txt and a Disallow All file look similar but do opposite things, and the wrong one can erase your visibility.
Robots.txt allows all by default if no file exists. An empty robots.txt file allows all bots to crawl, and having no file does the same. This is how search engines handle a missing file: RFC 9309 says that if no rule matches a URL, the URL is allowed. The standard goes further on errors. When your server returns a 4xx response such as a 404, crawlers treat it as allow-all and crawl freely. A 5xx server error flips that, and Google assumes full disallow until the file responds again.
So the SEO consequences split cleanly. No file means full access and full crawlability. Disallow All means no crawling, no fresh indexing, and a slow slide out of search results. Use Disallow All only when you genuinely want the site hidden. The same care applies as you grow into multiple locations; our piece on how marketing agencies support franchise growth shows why crawl access scales with your footprint.
How to Use Robots.txt to Allow All Pages
Allow All invites every bot to crawl every page. The minimal block uses an empty Disallow value:
User-agent: * Disallow:
The blank after Disallow means “nothing is off limits.” Two other approaches do the same job, a single Allow: / line or no robots.txt file at all. All three let every crawler reach the entire site.
This is the default state of the web, and for most sites it is the recommended one. Full crawl access means search engines can find, read, and index all the pages you publish, which is the foundation of organic visibility. When you want a page to rank, the first requirement is that Google can crawl it. Pair that open access with clean tracking. Our Google Analytics guide walks through it.
Why Allowing All May Be Best for Small Business Websites
Small business sites rarely have a crawl budget problem. They have a few dozen pages, not a few hundred thousand, so there is little risk of bots wasting their time. In that situation, restricting access usually costs more than it saves.
For a small site, every indexed page is a chance to be found. Blocking sections only shrinks your footprint in search engines. Allow All keeps the door open to your full catalog of services, locations, and posts, and it removes a common source of self-inflicted ranking problems. If you are weighing where to spend limited effort, our digital marketing tips for small businesses put visibility ahead of micromanaging crawlers.
Common Robots.txt Restrictions and Their SEO Implications
Between full-open and full-closed sits the work most sites actually do: blocking a few specific things. Done well, this approach protects crawl budget and quality. Done carelessly, it buries pages you needed.
Robots.txt restrictions block access at three levels. A single file is precise: Disallow: /secret.html stops bots from accessing specific files one at a time, the simplest way to block individual web pages. A directory is broader: Disallow: /admin/ closes a whole folder, and a specific folder user agent rule can limit one named bot to that directory. Use the narrowest rule that does the job. Good restrictions point bots away from internal search result pages, faceted-navigation URLs that create duplicate content, and admin areas, so the pages blocked are ones you never wanted crawled. That is healthy crawl budget optimization: you spend the crawl on critical pages instead of clutter. Restrictions turn harmful when they block specific pages you want ranked, block specific bots you need, or when a stray Disallow: / takes the whole site offline.
One restriction backfires almost every time. Do not block CSS or JavaScript, because Google needs those resource files to render and understand your pages. To keep image crawling in check, instead, you can target Google Images with the Googlebot-Image user agent. And blocking a heavy section eases load on your server resources during a crawl, which is a fair reason to restrict it. Vercel reports that GPTBot made 569 million requests and ClaudeBot made 370 million requests in a single month, which accounts for approximately 20% of Googlebot’s total of 4.5 billion requests.
The table below compares the directives you reach for most. One point to flag before you read it: crawl-delay is the odd one out because Google ignores it.
| Directive | Purpose | SEO Impact | When to Use |
|---|---|---|---|
| Disallow | Blocks listed URLs from crawling | Saves crawl budget; hides pages from search if misused | Low-value or duplicate paths |
| Allow | Permits a path inside a blocked folder | Keeps needed pages crawlable | Exceptions within a disallowed directory |
| Crawl-delay | Requests a slower crawl rate | Ignored by Google since September 2019; Bingbot still honors it | Easing server load for Bing |
| Sitemap | Points crawlers to your XML sitemap | Helps discovery and indexing | Every site, always |
That crawl-delay row reflects a real shift. On September 1, 2019, Google retired support for several unofficial rules inside robots.txt, including crawl-delay, noindex, and nofollow. A noindex directive written in robots.txt does nothing on Google now. To keep a page out of search engines, use a meta robots tag in the page’s HTML, or an X-Robots-Tag delivered through your HTTP headers, never a robots.txt line. That X-Robots-Tag and other page-level meta tags control indexing in a way robots.txt cannot. Crawl delay is the exception worth remembering: Bing still honors a crawl delay even though Google does not, and if you build a Bingbot section, duplicate your shared rules into it.
Blocking AI Bots
The newest reason to add restrictions is AI. As reported by originality.ai, approximately 35.7% of the top 1,000 websites now block GPTBot, a significant increase from about 5% in August 2023, representing a sevenfold rise.
The major AI companies all honor robots.txt, so you can opt out of training-data collection while keeping your search visibility. The bots fall into three jobs: training crawlers, search bots that cite and link back, and user-triggered assistants. Knowing which token does what is the key to the whole game. Here is how to selectively block AI crawlers.
The table below lists the AI bots worth knowing in 2026. Block the training crawlers if you do not want your content used to train models. Think hard before blocking the search bots, because that removes you from those AI search results.
| Token | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ClaudeBot | Anthropic | Model training |
| Claude-SearchBot | Anthropic | Search index |
| Google-Extended | Gemini / Vertex AI training | |
| CCBot | Common Crawl | Open crawl dataset |
| PerplexityBot | Perplexity | Search and answers |
| Bytespider | ByteDance | Model training |
| Meta-ExternalAgent | Meta | Model training |
To block AI bots, give each its own group with a site-wide Disallow:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: /
Google-Extended is a special case: a robots.txt-only token with no crawler of its own, introduced in September 2023 to control whether your content trains Gemini and Vertex AI. Google confirmed in April 2025 that it is not a ranking signal, so blocking it will not hurt your Google Search position.
Most businesses want a middle path: stay out of training sets but keep showing up in AI answers that drive traffic. This recommended pattern blocks the training crawlers while other bots keep their access:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: /
One limit holds throughout: robots.txt only governs cooperating crawlers. Malicious bots ignore it. Stopping those takes a security service to perform security verification on every visitor. Those are the systems whose challenge pages respond with Ray ID codes when they block a request, then read “verification successful” once a real browser clears the check. For a quick view of how these rules interact, our SEO tools roundup includes crawlers that flag conflicts.
How to Test and Validate Your Robots.txt File
Never publish a robots.txt file without checking it. Robots.txt files can be tested using Google Search Console, and the tooling changed recently, so make sure you are on the current method.
The standalone robots.txt Tester that used to live in Search Console was retired. Two tools replaced it. The robots.txt report shows the files Google found for your top 20 hosts, when each was last crawled, and any warnings or errors. The URL Inspection tool checks a single URL and tells you whether robots.txt is blocking it, plus how Google renders the page. Third-party online validators can sanity-check syntax too, but the Search Console tools are what Google acts on.
Here is the validation flow we use. First, open the robots.txt report and confirm Google fetched the file without errors. Second, run a few important URLs through URL Inspection to confirm they are blocked or open exactly as intended. Third, after any edit, request a recrawl from the report. This matters because Google caches robots.txt changes for up to 24 hours, so a fix may not take effect until the cache clears or you prompt a fresh fetch.
Troubleshooting follows the same path. If a page you blocked is still being crawled, check case sensitivity and the leading slash in the path. If pages you wanted open vanish from search results pages, look for an over-broad Disallow. When you also want a fast site behind those rules, our guide to test site speed pairs well with a crawl audit.
Ready to Take Control of How Search Engines Crawl Your Site?
A well-built robots.txt protects your crawl budget, keeps the wrong pages out of search, and steers both search and AI bots exactly where you want them. Get it wrong, and a single line can erase months of ranking progress overnight. The safest move is to have an experienced team review the file before it ships, not after your traffic drops.
Our digital marketing professionals have spent over a decade auditing, fixing, and optimizing technical SEO for businesses of every size. We treat your robots.txt as one piece of a larger strategy, tying crawl control to the rankings and revenue it actually affects. Contact us today, and we will make sure your site stays open to the right bots and closed to the rest.
Frequently Asked Questions
We have written, fixed, and audited robots.txt files across hundreds of client sites, from single-location small businesses to multi-market brands. That hands-on work is where these answers come from. Our SEO team watches how each rule plays out in Search Console, so the guidance below reflects what actually happens, not just what the spec says.
Is Robots.txt Illegal?
No. Robots.txt is a voluntary standard, part of the Robots Exclusion Protocol that was formalized as RFC 9309 in September 2022. It is not legally enforceable and relies on bot cooperation. Well-behaved crawlers follow it, but malicious bots can ignore it entirely.
What Does Disallow in Robots.txt Mean?
Disallow tells a crawler not to crawl specific URLs or paths. It blocks crawling, not indexing. A disallowed page can still appear in search results if it picks up external links from other sites, because the rule stops the fetch, not the listing.
How Do I Disallow a Specific URL in Robots.txt?
Name the user agent, then list the path. To block one page for all bots:
User-agent: * Disallow: /path/page.html
Remember that path values are case-sensitive and must start with a slash, so /Page.html and /page.html are not the same URL.
What Are Robots.txt Restrictions?
The file has real limits. It only governs cooperating crawlers, it blocks crawling but not indexing, and it is not a security tool. Google also enforces a 500 KiB parse limit and ignores some directives, including noindex and crawl-delay. Plan around those constraints rather than fighting them.
Can Robots.txt Guarantee Privacy or Security?
No. Robots.txt is advisory, and the file itself is public, so every path you list is visible to anyone. That can actually expose sensitive directories you meant to hide. For real protection, use authentication, password protection, or a noindex tag, not a Disallow line.