IGNITR

If you’ve ever poked around your website’s files, you may have stumbled on a plain text file called robots.txt sitting quietly at the root of your domain — something like yourbusiness.com/robots.txt. It looks unremarkable, but this tiny file gives instructions to every search engine crawler that visits your site. Get it wrong, and you could be accidentally telling Google to ignore your entire website.

This post is part of our ongoing technical SEO 101 series, where we break down the behind-the-scenes settings that quietly shape how well your small business site performs on Google.

What Robots.txt Actually Does

Robots.txt is a simple set of rules that tells search engine crawlers (like Googlebot) which parts of your site they’re allowed to visit and which parts to skip. It lives at the root of your domain and follows a basic format:

User-agent: *
Disallow: /admin/
Allow: /

In plain English, that example says: “Every crawler, please stay out of the /admin/ folder, but feel free to crawl everything else.” It’s a set of polite instructions — well-behaved crawlers like Google and Bing respect it, though it’s worth knowing it’s not a security wall. It won’t stop bad actors from finding a page; it just tells honest crawlers where not to spend their time.

What It Doesn’t Do

A common misconception: blocking a page in robots.txt does not remove it from Google’s index or guarantee it stays hidden. If other sites link to that page, Google can still discover and even index the URL without visiting it — you’ll just see a bare listing with no description. If you truly want a page kept out of search results, the correct tool is a noindex meta tag on the page itself, not a robots.txt block. In fact, blocking a page in robots.txt can actually prevent Google from ever seeing a noindex tag placed on it, because Google never crawls the page to read the tag.

The Mistake That Costs Small Businesses Traffic

The single most damaging robots.txt mistake we see is a leftover rule from a development or staging site:

User-agent: *
Disallow: /

That single line tells every search engine to stay away from your entire website. It’s common during a site build — developers block crawling so an unfinished site doesn’t get indexed — but if that rule doesn’t get removed before launch, the business can go months wondering why they’re invisible on Google, when the real answer is sitting in one line of a text file nobody remembered to check.

What Small Businesses Should Typically Block (and Not Block)

For most small business websites, robots.txt should stay simple. Reasonable candidates to disallow include internal search result pages, admin or login areas, staging subdomains, and duplicate parameter-based URLs (like ones created by filters or tracking tags). You generally should NOT block your CSS or JavaScript files — Google needs to load those to understand how your pages render, and blocking them can hurt how your site is evaluated. You also shouldn’t block core content pages, product pages, or blog posts you actually want found.

How to Check Yours

Checking your current file takes 10 seconds: type your domain followed by /robots.txt into a browser (e.g., yourbusiness.com/robots.txt) and see what comes up. If nothing loads, that’s usually fine — search engines will just crawl normally. If you see a blanket Disallow: /, that needs fixing immediately. Google Search Console also has a way to test and validate your robots.txt file directly, so you can confirm important pages aren’t accidentally blocked before they cost you traffic.

The Takeaway

Robots.txt is one of those files most business owners never think about until something goes wrong. It’s not glamorous, but a stray line of code in it can quietly undo everything else you’re doing right in SEO. A quick periodic check — especially after a site redesign or migration — is cheap insurance against invisible-to-Google syndrome.

Not sure what your site’s robots.txt is doing, or want a technical SEO check-up as part of a broader website health review? Check out our plans or reach out to our team — we’re happy to take a look.