robots.txt isn't a vault — it's a note taped to the door
John Mueller dropped a reminder recently that, honestly, shouldn't be news to anyone by now: robots.txt isn't law. It's a request. Google can ignore it. Not out of spite — but because somewhere on your end there's a config error, or the file itself is unreachable.
And here's the thought I keep coming back to: people treat robots.txt like a safe. Write a Disallow line, and boom — it's locked away. In reality it's closer to a note taped to the door that says "do not enter." Most passersby will respect it. But if the door's broken, or the note's written badly, or it's simply not visible from the side the crawler's knocking on — someone walks right in.
Where the ignoring comes from
The mechanics here are simple, nothing mystical. Google can drop robots.txt rules when:
- the file itself is unreachable (5xx server error, timeout — in this case Google either waits it out or, in some cases, crawls as if the file didn't exist at all);
- there's a conflict or outright mess in the syntax — directives contradict each other, and the crawler interprets it however it sees fit;
- there's a stronger signal somewhere else — say, a page is already indexed via external links, and a crawl block alone isn't enough for Google to drop it from the index.
That last point deserves its own conversation, because it's the one people mix up the most. robots.txt controls crawling, not indexing. Those are two different things. Block a page in robots.txt and it can still sit in search results — just without a snippet — because Google saw links pointing to it and decided the page exists and deserves a spot in the index. A crawl ban isn't a ban on being there.
Why this hurts especially bad for CSS and JS
Separate conversation entirely: blocking folders with stylesheets and scripts in robots.txt. The reasoning is usually laughable — "those files aren't for people, why let the bot see them." The catch is that Googlebot renders the page almost like a browser does, and if it can't reach your CSS/JS, it sees a broken, half-rendered version of your site. That hits directly how the page gets evaluated — yes, even if your HTML underneath is flawless.
I've seen this on live projects more than once: a site that's technically "optimized" on paper, while in reality Googlebot is staring at a bare skeleton with no styling, because someone, at some point, blocked the /assets/ folder "for security." Nobody checked it. Nobody thought through the consequences. This isn't some rare edge-case bug — it's a fact, and it sits quietly in projects for years.
What to actually do about it
Stop treating robots.txt as your only line of defense. It's a crawl filter, not an indexing control tool, and it's definitely not protection for sensitive data — that's what noindex meta tags, the X-Robots-Tag header from your server, and real access-level authorization are for.
Practical hygiene here is simple.
Check the syntax and logic of your rules during development — not after the fact, once traffic has already tanked. Don't block CSS and JS — obvious advice, and people keep stepping on that rake anyway. Check the robots.txt report in Search Console regularly — it shows you exactly what Google is reading and how it's interpreting it. And for the truly critical cases — pages that must never, under any circumstance, show up in search — back up the block with noindex. Don't just trust a single Disallow line to hold the fort.
Google isn't your enemy, and it's not sabotaging you. It's just doing what's technically logical given the set of signals you handed it. Feed it contradictory signals, and you'll get a contradictory result. Clean up the basics, and most of these "surprises" simply stop happening.
If you'd rather not untangle this yourself, you can hand the technical audit over to our team.