WMW - the bots are back

38 comments

WMW is apparently back in Google and MSN. This thread dances around the what-did-you-do subject, but it looks like deals have been made with the SE's.

Comments

Four weeks into the ban

Four weeks into the ban, I looked at the Alexa chart for WMW ranking since their ban. The effect of their robots ban was fairly dramatic (the earlier spike was WMW being off air for a few days in late October), but there is some evidence of a rebound over the last week.
[img]http://traffic.alexa.com/graph?w=379&h=216&r=6m&y=t&u=webmasterworld.com/&u= [/img]
Whilst one can argue Alexa accuracy in absolute terms,I would reckon that the relative movements given are correct.

Interesting to see how this picks up now.

Late to the party

Greg Boser blogged about this on Friday - Welcome Back Brett. Nice URL there.

I was doing some test surfing this morning using a new user agent/header checking tool Dax just built. Just for fun, I loaded up WebmasterWorld with a Slurp UA. Suprisingly, I was able to navigate through the site. I was also able to surf the site as Googlebot and MSNbot.

I quick check of the robots.txt with several different UA’s showed that MSN and Yahoo are now given a robots.txt that allows them to crawl. However, Google is still banned, and humans still must login in order to view content.

Apparently, it’s been this way for awhile because both engines already show a dramatic increase in page counts.

MSN 57,000
Yahoo 160,000

No Follow ROFLMAO

No Follow ROFLMAO

Brett, you woose!

Seems there could well be more than just a grain of truth in all those "rumours"

looks like Brett may have

looks like Brett may have banned WG, WG is back, ban may have been automated for using a Googlebot user agent

I wish

I wish someone else would do a cool forum

See how it now works

/robots.txt has the story, /robots2 is the actual, working robots.txt. If you follow that.

WMW

It would be great if we could drop the WMW links from TW - if Bett wants to keep a walled garden around his site then that's his perogative, but it's not useful to human users if they're forced to login just to read chatter on another forum.

?

Can't agree there. Threadwatch. Not that uncommon to see posts marked (requires login) - if BugMeNot doesn't get you in, then fine, but it's a choice. Nick always banged on about linking liberally, and I agree. Others may not.

(Didn't link to robots examples in my last post - didn't feel right, but that's rare)

I agree on keeping linking to WMW

Different policies there.

Wrong policies there maybe, but different.

In spite of the fact I got a smack from a WMW mod this morning for one of my infrequent posts there, I will continue to link to them when there is something worth linking to.

I think the web has moved on from the "walled garden" anyway.

I think not linking should

I think not linking should be reserved for people who are using spam to market garbage and maybe sometimes also those who are intentionally trying to piss you off with the goal of obtaining the link...otherwise linking is always cool IMHO.

<sarcasm>

<sarcasm>
wow.. what an amazing idea cloaking robots.txt and requiring logins
bretts so smart..
</sarcasm>

Can someone please explain to me?

Why do people still bother with WMW? There are so many other forums that are much more open than WMW, why bother?

Because despite how rude the

Because despite how rude the anglophile denzians are there, there is good info there if you can find it and its where a lot of people got started.

so many other forums Some

so many other forums

Some people hate Threadwatch too...I say just find where you like and fit in, and post wherever that may be.

BTW WMW is still pretty big and used to have even greater market force before so many independant blog type channels sprung up. I actually met my second client from WMW wanted ads and still work with them today, but on other larger projects.

In addition to being the 800

In addition to being the 800 pound gorilla of search engine forums, WMW has the most search engine reps of any forum. Having Googleguy, AdsenseAdvisor, AdwordsAvisor, Tim (Tim Mayer), MSNDude and a few others is a major plus.

And a helpful Community

One of the things that I like most about WMW community is the prompt help you get from other fellow members whenever you need it.

In my case my good WMW friend Dayo_UK made me aware that my site had a minor canonical problem. Other fellow members helped with a 301 redirect. Google took care of the rest :-)

thats true reseller... if

thats true reseller... if you write english well.

You guys need to quote more...

...from WMW threads referenced here. Apparently, the Googlebot User-Agent trick no longer works ;-]

Grade school

You guys whining about WMW going bye bye reminds me of those who are not able to leave their childhoods behind. You know, that guy you used to party with from your old town who still lives with his mom at age 36? Yes, he is still buying beer for high school kids.

(Don't hate me because I a right) ;-)

Some people hate Threadwatch?

Say it isn't so!

Blasphemy!

I've never been smacked by a mod here yet but I'm slightly kinky so it might backfire if you try.

"I've never been smacked by a mod here yet "

Relax Bill, no sado-masochists here - well not that I know of.

This was expected

The site dominated the search for almost every major query about anything webmaster related. If you needed .htaccess help, robots help, php help, Google help, there was bound to be a thread popping up with your answer. WMW was bound to take a major hit from banning robots and I think it was arrogant and misguided to believe otherwise.

As for linking, the policy should remain liberal with linking. If it is considered newsworthy, it should be considered link worthy. Banning certain sites from being linked to is not what made Threadwatch what it is.

I never understood the whole thing with banning bots anyway. It almost seemed like an arrogant thing to do and thumbing your nose at people. A "we don't need search traffic" approach. Apparently they do though. The other bizarre thing was the solutions they weren't using. Bots can be a pain, but backbreaking they are not. Even letting many run wild won't completely kill your site. This isn't 1997, and bandwidth does come fairly cheap these days. Lets not forget that a site search will add much more stress than any rogue bots will.

About Banning Bots

This isn't 1997, and bandwidth does come fairly cheap these days.

WMW is a 100% dynamically generated site, much like my web site, and there were many days it ran at a crawl when the bots were pounding on it. Not because of bandwidth, because of CPU usage, and I've had it happen to my site so I know where he's coming from.

I understand the reason behind the total lock out of all bots was to see which bots ignored what rules, which ones broke thru security measures, and which could be housebroken to behave better, etc.

I've blocked lots of bots but NEVER Google, but then again I don't get quite a much traffic as WMW gets even on a good day. I think it was just to take some pressure off the site while he resolved other issues but it remains to be seen.

if Bett wants to keep a

Quote:
if Bett wants to keep a walled garden around his site then that's his perogative, but it's not useful to human users if they're forced to login just to read chatter on another forum.

Well said. If idiot boy wants to make himself irrelevant, let him. Who gives a flying fuck about wmw? It's been a fucking farce for years, now it's just embarrassing...

Savvy marketing move

Personally, I think WMW took a serious business issue and turned it into brilliant marketing:
- hecka lot of buzz, generally
- top SE and SEO players involved, publically and behind the scenes
- authoritative sources commenting on BT's move and WMW's status as a leading site
- increased awareness of an important issue that was causing serious issues from a business perspective

Everyone's talking about it, regardless of what they're saying ... and the issue is getting addressed better than it might have if everything had been handled any number of other ways, which is good for WMW and lots of other big sites too.

Talk about turning a problem into an opportunity. Gotta love those robots files too. This one is going into my "Marketing 501" notebook as a case study.

You honestly think that guy

You honestly think that guy planned all this ? You give him way to much credit.

He is one guy that was lucky to start a website when he did and about what he did. Thats it... thats the secret to Bretts success. Luck.

He is one guy that was lucky

Quote:
He is one guy that was lucky to start a website when he did and about what he did. Thats it... thats the secret to Bretts success. Luck.

Ahh, but he did it I, you and many other's didn't. For that you have to doff your cap to him.

old saying "the harder I

old saying "the harder I worked the luckier I got"

Planned?

Naw.. Clearly it started as a business/technical issue. How it was managed though, is a different matter. For those who did not notice or were not there, he actually started to make something of it publically at the last PubCon, but the uptake, even from the SE reps, was slow or non existent. That was a choice, to begin to handle it publically, and in a way that might have gotten some attention. Others may have kept it all behind the scenes. Just like others might have just put up normal, standard, revised robots files, rather than turning them into instruments of marketing. Like someone at WMW said, "first time I've ever bookmarked a robots.txt file." hehe.

Hey, life isn't about avoiding problems; it's about what you do with the problems you have. This problem could not have been handled much more effectively IMO.

And ditto Jason's and Graywolf's comments. BT has put himself in a position to be successful. In this case, credit where credit is due, that's all.

I usually go and look at

I usually go and look at "foo" these days. 5 or so years ago WMW was the only place to go for seo.Heh, remember when nick was a mod on the css forum :)and did not call Tabke "idiot boy".

So times have moved on and there are definately a few more places to go these days but I think BT has enough new blood coming along to fill the shoes of the old pros who hang out in other places these days. WMW is still an institution and I am sure most of you guys here started posting there first. I would say the interface is a getting a bit sour and needs a major overhall. He could also make a shitload out of adsense but I guess it is nothing to what he makes on pubcon. I guess he could do many things but just leaves it because if it aint broke and all that..

Ahh, but he did it I, you

Ahh, but he did it I, you and many other's didn't. For that you have to doff your cap to him.

Absolutely agree with that and graywolfs statement, he saw the opportunity, took it, and managed it into something very successful.

Congrats to him.

// added..

All I read are the libraries and the css/programming sections anymore. The libraries are the best feature of that site you'd think more forums would have that.

Well Nick...

ROFLMAO - Tell us what you REALLY think?

And Brett's robots.txt being used as a blog, what a scream LOL

Traffic

Posts seem to hang around on WMW for a lot longer than they used to do. Has he lost a ton of traffic?

Marketing?

This is no marketing trick and any exposure it has garnered from this is not good exposure. Take a look at the Alexa graph on top (I know it's not a good gauge but you can clearly see a trend). If this "marketing" technique was so brilliant, I'd foresee more than his traffic being essentially cut in half since this move. Of course it drew attention from some of the bigger names in the industry, but for the most part, it was at the extent of a joke being made about it.

Before I come across as bashing WMW, I'm not. I still feel they have the most informative site for webmasters on the web. There is not a single site out there that has more tips, guides, and help than WMW. Finding that information without search is another story.

However, it isn't what it used to be. It isn't where the pros hang out and you rarely come across a top notch thread that gets you thinking. If reading 600 page Google update threads with guys whining that their site that they spent the past 3 years of their life building 24,000 reciprocal links for the same term isn't ranking anymore and how Google is broken is your thing, then it's a great site. Otherwise, the greatest element of WMW, the massive archive of information, was buried when the bots were banned.

And for those who cast some apocalyptic notion that rogue bots are bringing down a site, let's be real. There were much better solutions to this problem. The fact there is no search and a ridiculous solution to stopping these bots just makes the site look amateurish.

My New Blog

My new blog is written in the comments of a javascript include file.

Cool heh :/

>> they have the most

>> they have the most informative site for webmasters
Of course it is.

>> There is not a single site out there that has more tips, guides, and help than WMW
Spot on.

>>Who gives a flying fuck about wmw?
Er, only about a few million visitors a day.

>> Ahh, but he did it.
And is still committed to the site he built.

going down hill

wmw has lost both traffic and interest, has been going that way for years. remember the ol days of real information sharing and ideas, long gone. wmw will never be the same again, search info is just too sensitive these days, who's really willing to spill the beans? (unless it's to some lazy client unwilling to do the work themselves?)

Banning certain sites from

Banning certain sites from being linked to is not what made Threadwatch what it is.

Quite agreed - my point is simply that links to closed-walled content isn't so useful for human users.

2c.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.