Google IGNORES disallow search in Robots.txt file

1 comment

Bill Hartzer has published an example of a site ranking #1 for their competitive keyword phrase even though their site has robots disallowed.


seen data in GWT

Ah yep more than a few times I've seen URLS in GWT that are blocked in Robots.txt that Google is cralwing, one example was it was crawling the parameters of a URL that is clearly blocked. Also getting to these URLs is not easy so it really worked at it

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.