Robots.txt for a Small Business Blog: What to Block, What Not To, and a Safe Default (2026)

By · Published 2026-10-31 · Last verified 2026-10-31

Free tool: Robots.txt checker. Paste your robots.txt and test whether a URL is allowed or blocked.

What should a small business block in robots.txt?

For most small business blogs: almost nothing. Google's documentation says robots.txt tells crawlers which URLs they can access and is used mainly to avoid overloading a site with requests. It is not a way to keep a page out of Google. A short file that allows everything and lists your sitemap is the right default. Block only areas with no search value that crawlers might waste time on, such as internal search results or cart pages, and never block the CSS or JavaScript your pages need to display. To hide a page from search, use noindex or a password instead. This guide shows a safe default file, what's worth blocking, and the mistakes that quietly remove a blog from Google.

What robots.txt does, and what it doesn't

Robots.txt is a plain text file at the root of your domain, for example https://example.com/robots.txt. Before crawling, Googlebot reads it to see which paths it may fetch. Three points from Google's guide matter most for a small business:

  • It controls crawling, not indexing. If other pages link to a blocked URL, Google can still index that URL without visiting it. It may then appear in results with no description.
  • It blocks noindex too. Google notes that for noindex to work, the page must not be blocked by robots.txt. Otherwise the crawler never sees the rule.
  • It's public. Anyone can read your robots.txt, so listing a "secret" folder in it advertises the folder.
Two columns: what robots.txt does, such as controlling crawling and pointing to the sitemap, and what it doesn't, such as keeping a page out of Google
What robots.txt does.

A safe default for a small business blog

If you have nothing that needs blocking, this is a complete, valid file:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Google supports the user-agent, allow, disallow and sitemap fields. The sitemap line must be the full URL. Listing it here helps crawlers find every post, as explained in does a small business blog need an XML sitemap. Fields such as crawl-delay aren't supported by Google, so adding them changes nothing there. Blocking Google-Extended does not switch off AI Overviews; see AI Overviews and a small business blog.

No file at all is also fine. Google treats a 404 (and other 4xx errors except 429) for robots.txt as if no file existed and assumes there are no crawl restrictions.

Four parts of a safe default robots.txt: user-agent star, allow all, the full sitemap URL, and a 404 meaning no restrictions
A safe default.

What is worth blocking

Block only URLs that are endless or useless for searchers, where crawling wastes your server and Google's time:

  • Internal search results, such as /search?q=, which can generate unlimited thin pages.
  • Cart, checkout and account pages on a shop, which change per visitor and shouldn't rank.
  • Filter and sort parameters that create thousands of near-duplicate listings.
  • Staging or test copies, though these are better protected with a password, since robots.txt won't keep them out of the index.

Duplicate posts or print versions are a different problem. A canonical tag handles those better than blocking; see when you need a canonical tag. For paid, affiliate or user-added links, use rel values such as sponsored, ugc and nofollow instead.

Grid of URLs that may be worth blocking: site search, cart, accounts, filters, staging better protected by a password, and duplicates better handled by canonicals
Worth blocking, rarely.

Mistakes that hide a whole blog

  1. Disallow: / left over from development. A single slash blocks the entire site. It's the most common reason a newly launched site gets no traffic.
  2. Blocking /wp-content/, /assets/ or script folders. Google's guide warns that if missing resources make a page harder to understand, you shouldn't block them. Google renders pages much like a browser does.
  3. Blocking /blog/ to "hide" drafts. This blocks your published posts too. Keep drafts unpublished instead.
  4. Using robots.txt to remove a page. It can leave the URL indexed without a description. Use noindex, and let Google crawl the page so it sees the rule.
  5. A typo in a path. Rules are case-sensitive and match from the start of the path, so /Blog and /blog are different.

If posts are missing from Google, also check the other causes in why a new blog post isn't indexed.

Grid of robots.txt mistakes: Disallow slash left from development, blocked CSS and JavaScript, blocked blog folder, removing pages with robots.txt, case typos and the 24-hour cache
Mistakes that hide a blog.

How to check your robots.txt in five minutes

  1. Open yourdomain.com/robots.txt in a browser and read every Disallow line.
  2. In Search Console, open Settings and the robots.txt report to see the version Google fetched and any errors.
  3. Use URL Inspection on one blog post and confirm crawling is allowed.
  4. After changing the file, give it time: Google generally caches robots.txt for up to 24 hours.
  5. Keep it small. Google ignores content beyond 500 KiB, which a blog will never need to reach.

Robots.txt is also the place to check before adding structured data or any other technical change, since a blocked page can't show any of it; see structured data for a small business blog.

Vertical Pinterest graphic with five points on robots.txt for a small business blog
Pinterest pin: robots.txt for a blog.

Where PilotaSEO fits

PilotaSEO publishes posts on a schedule and lists them in a sitemap, so a robots.txt that allows your blog and points to that sitemap is all it needs from you. If you're unsure whether yours blocks anything important, the five-minute check above will tell you.

robots.txt, noindex or a password: which to use

Most robots.txt mistakes come from using it for a job it doesn't do. A quick way to choose:

  • You want Google to spend less time on endless or useless URLs (internal search, filters): robots.txt Disallow.
  • You want a page out of Google's results (a thank-you page, an old offer): a noindex robots meta tag, with the page left crawlable so Google can see it.
  • You want a page private (a staging site, client files): a password or login. Neither robots.txt nor noindex stops a person from opening a URL.
  • You have two versions of the same post: a canonical tag pointing to the main one.

For a small business blog, the first case is rare, the second and fourth come up now and then, and the third belongs to whoever runs the website. That's why the best robots.txt for most blogs is the short default above. If you work with a web designer, ask them to remove any leftover Disallow: / on launch day, and check the file yourself the same week.

One last habit: after any website redesign, platform move or new plugin, open the robots.txt again. Site builders and SEO plugins sometimes rewrite it, and a change you didn't make is the hardest kind to notice until traffic drops.

FAQ

Does robots.txt stop a page from appearing in Google?

No. It stops crawling, not indexing. A blocked URL can still be indexed if other pages link to it. Use noindex or a password to keep a page out.

Do I need a robots.txt file at all?

Not necessarily. If the file returns a 404, Google assumes there are no crawl restrictions. A short file with your sitemap URL is still a good habit.

Should I block CSS and JavaScript files?

No, not if your pages need them to display properly. Google needs them to understand the page.

Does Google support crawl-delay?

No. Google supports user-agent, allow, disallow and sitemap, and ignores crawl-delay.

How quickly does Google see robots.txt changes?

Google generally caches the file for up to 24 hours, so changes can take about a day to be used.

Sources