Vibe SEO

Technical SEO

Robots.txt Syntax Explained Rule by Rule With Examples

November 19, 2026

This post teaches robots.txt syntax, so you can open any robots.txt file and say what each line does. It also shows five working examples for a local WordPress site that you can adapt today.

It opens a series that covers every robots.txt rule and every meta robots value, one at a time. After the series, any directive on your site will make sense to you. For the wider picture, read The Complete Technical SEO Guide for Local Business Sites. This post goes deeper on one file.

The parts of robots.txt syntax, line by line

A robots.txt file is plain text. Each line is a field name, a colon and a value. Crawlers read it before they fetch your pages. It tells them which paths they may request and which they should skip.

Here are the parts you will meet:

  1. User-agent says which crawler the lines below it are for. A star means every crawler. Googlebot means Google’s main crawler. Each group starts with one or more user-agent lines.
  2. Disallow gives a path the crawler should not request. A disallow rule with an empty value blocks nothing.
  3. Allow gives a path the crawler may request. You use it to make an exception inside a blocked folder.
  4. The star wildcard matches any run of characters. The path /*?s= matches any URL that contains ?s=, the WordPress search parameter.
  5. The dollar sign anchors a pattern to the end of the address. The path /*.pdf$ matches URLs that end in .pdf and nothing after it.
  6. Sitemap holds the complete web address of your XML sitemap. It stands alone and does not belong to any group.
  7. A comment starts with a hash mark. Crawlers ignore everything after it on that line.

Every path in a disallow rule or an allow rule starts with a slash. The two wildcard characters are the only pattern tools you get. There is no full regular expression support. Google also ignores fields it does not support, such as crawl-delay.

Once you know this robots.txt syntax, a file stops looking like code. It is a short list of groups, and each group has a name and a few rules. Read it from the top, one group at a time.

One point trips up many owners. A disallow rule stops crawling, not indexing. A blocked URL can still show in results if other pages link to it. To keep a page out of results, you use a noindex directive, and the crawler has to be able to read the page to see it. A later post in this series covers that.

Where the robots.txt file must live

The file must sit at the root of the host, for example https://www.example.com/robots.txt. A copy in a folder, such as /blog/robots.txt, does nothing. Crawlers only ask for the root location.

The file covers one host and one protocol. So https://www.example.com and https://shop.example.com each need their own file. The same goes for http against https. If you run a booking subdomain or a separate store, check that it has a file of its own.

Also check that the file returns a normal page and not an error. A server problem on the file can make crawlers hold back from the whole site, so a broken file is worse than none. Open the address in a browser. You should see plain text and nothing else.

A missing file is not an error for Google. With no file, it treats the whole site as open to crawling. That is fine for many small sites. Still, an explicit file shows you made a choice.

Case, size and encoding

Paths are case-sensitive. A rule that says Disallow: /Private/ does not block /private/. Field names such as user-agent are not case-sensitive, but copy the usual capitalisation anyway, because it reads better. Match the case of the real URLs on your site.

Save the file as UTF-8 text. A word processor can add odd characters, so use a plain text editor or the editor in your SEO plugin. Also keep the file small. Google reads the first 500 KiB of the file and ignores anything after that, as its guide to how Google interprets the robots.txt specification explains. A local business file is a few lines, so you will not get near the limit.

Five example files for a local WordPress site

Each example below shows the lines of one file, in order. Change example.com to your own domain. Together they show robots.txt syntax at work. Then read the reason after each one, because the reason is what decides if you need the file.

Example 1: open to everything

User-agent: *

Disallow:

Sitemap: https://www.example.com/sitemap.xml

The empty disallow rule blocks nothing. Use this when you have no pages to hide and only want the sitemap line.

Example 2: the usual WordPress file

User-agent: *

Allow: /wp-admin/admin-ajax.php

Disallow: /wp-admin/

Sitemap: https://www.example.com/sitemap.xml

Nobody searching Google needs your dashboard, so the folder stays closed. One file inside it stays open, because some themes and plugins fetch public content through it. Google follows the most specific matching rule, so the order of the two lines does not change the result.

Example 3: internal search pages

Add these lines to a group:

Disallow: /?s=

Disallow: /search/

Your site search can create endless thin URLs, each built from whatever a visitor searched for. Blocking them keeps crawlers on your real pages. Check your own search URL first, because some themes use a different pattern.

Example 4: wildcards

Add these lines:

Disallow: /*?replytocom=

Disallow: /*.pdf$

The first line skips comment reply links, which duplicate the post. The second skips every PDF. Use the second only if you do not want your PDFs found, because a price list or a menu in PDF form may be something you do want in results.

Example 5: a staging site

# Staging copy, remove at launch

User-agent: *

Disallow: /

The single slash blocks the whole host. This is the most dangerous line in robots.txt. If it ships to your live site, crawlers stop requesting every page. Also, a password on the staging site is safer than this file, because robots.txt is public and only polite crawlers follow it.

Those five files use the same small set of parts. If you can read them, your robots.txt syntax knowledge covers most local sites. Bigger sites add more rules, but they still use the same lines.

Notice the comment in example 5. Comments cost nothing. Use them to record why a rule exists and who added it.

Every robots.txt rule needs a reason you can say out loud

Many files in the wild carry old rules that nobody can explain. A previous developer added them, or a plugin did. Then they sit there for years, and one of them blocks something that matters.

Use a simple test. For each disallow rule, finish this sentence: we block this path because crawlers waste time there, or because it is private, or because it duplicates another page. If you cannot finish the sentence, the rule should not stay. Also remember that robots.txt is public, so it is no place to hide anything sensitive. Anyone can read it, and a long list of private paths acts as a map.

Testing matters as much as writing. After any edit, load the live file in a browser and check that your key pages, such as the home page and your main service pages, are not blocked by a rule you added by mistake. Then use URL Inspection, which reports “Blocked by robots.txt” for a blocked page; the Page indexing report lists the same reason. The next post covers testing rules before you publish. A slip in robots.txt syntax, such as a missing slash, can block far more than you meant.

Your SEO plugin may also write or edit the file for you. In that case, open https://yourdomain.com/robots.txt in a browser and read what is live. The live file is the only one that counts.

What to do next

Open your own robots.txt file now and check it line by line against the robots.txt syntax above. Write one reason next to every rule, and delete the ones you cannot explain. Do the same for every subdomain you run. The next post in this series covers which rule wins when two rules or two groups seem to apply, and why adding one group can silently change another. If you want the rest of the technical work in one place, start from the Technical SEO category.

Rather talk it through with a person? Call 866-VIBE-SEO

Is your site ready to be found by Google and by AI?

Free instant scan. Real checks on your actual page, no signup needed.