Brett Tabke Challenges Jakob Nielsen to Block the Bots

33 comments

Brett Tabke founder of WebmasterWorld, writes an open letter in his blog to expert usability consultant Jakob Nielsen. Nielsen recently wrote a controversial article: Search Engines as Leeches on the Web.

Very interesting and worth reading. Let's see if Nielsen takes the challenge, and practices what he preaches.

Comments

So all tabke's other posts...

To quote tabke's robots.txt blog:

Quote:
To test out the theory that a community site could liberate itself
# from search engine dependence, we tried a test in November.

So, all his other posts to his community (of which I'm a part) that this was to deal with problem bots was lying bullshit? Perhaps I missed something here, I stand to be corrected.

i still dont get the "block

i still dont get the "block the bots" rationale. whether or not anyone likes it, search engines are a huge source of traffic. even if you currently don't need that traffic, if you dont try for it, a competitor will. once your competitor gets all that traffic, and once htey start converting that traffic, and once they have enough money to price you out of the market.....

also is there a reason why he's using his robots.txt file as a blog? that seems so weird to me.....i cant see a single advantage.

What kind of a challenge is that?

Quote:
#
# So, this all leads me to a challenge Jacob, and that is, to block
# the search engines for 90 days to Useit.com. No guts - No Glory.#

Ummm yeah. Why would anyone take that challenge? Because that went so well for Tabke right? I mean, he blocked the SE's for 90 days and the end result was........unblock and let the SE's back in again. Gee, that went well.

Color me wrong again, but I don't see a lot of glory around for that little experiment.

(I'm not trying to personally hammer Tabke here, I don't really know him as a person. But these recent actions, unless someone explains it real plain-like to me, sure don't seem very bright to me - and my comments above about misleading the community still stand to be corrected.)

--

Since Jakob was for the most part talking about PPC in his article, I don't really see what blocking the robots have to do with anything.

But then again, I never could understand how blocking good robots would stop rogue robots either, so maybe it's just me.

I mean, he blocked the SE's

Quote:
I mean, he blocked the SE's for 90 days and the end result was........unblock and let the SE's back in again.

90 days? It was more like 30...if that. Although he said he he was going to do it for 90.

All that comes to mind when I read this is...

"your pill?"

Seriously, what's the point, the Interweb is this neato place where some people link to eachother and crawl eachother. It's really just a big informational & social event.

If you want to be a private party, be a private party. If you want to be social, let the good bots roll...

You people are missing the major points

A. Brett jumped the shark and didn't notice Jacob's mind numbing tirade was mostly focused around PAID search which I don't think WMW has ever done and blocking search engines would still permit PAID search listings. Apples and oranges.

B. Brett's blog is in ROBOTS.TXT which disables all bots so robots.txt is blatantly being cloaked except to the bots he likes.

User-agent: *
Disallow: /

C. WMW used to suffer massive meltdowns at various times when the site just crawled before Brett did this last round of robot wars, and having similar issues, I'm a believer in bot problems.

Besides, you all seem to think 3 search engines is the end-all-be-all to bots on the web and you are sorely mistaken as the flood of idiot bots is what I've been battling for quite some time.

jumped the WHAT?

jumped the shark - http://en.wikipedia.org/wiki/Jump_the_shark means past it's prime

I think you mean Jump the gun - http://www.answers.com/topic/jump-the-gun

Brett may have "jumped the shark"

but he did what a million other people are tempted to do everytime the next Alertbox comes out. Nielsen is, and has always been, easy shark bait.

Not to change the subject, but this, in Brett's blog, is a scream.

Blogging Today : Please Stand By
# 1/4/2006

Can't imagine Jakob will bite on the challenge. He has too many books to sell online and he can afford server Performance testing.

My Guess

Quote:
So, all his other posts to his community (of which I'm a part) that this was to deal with problem bots was lying bullshit?

I doubt it. My guess is that it was a drastic, dumb step he made after a lot of frusteration. He felt his community was strong and he didn't need the engines. He felt confident enough to thumb his nose at Google and such. 30 days later and traffic going down the toilet, he killed that idea. He wasn't lying, he looked dumb and is now just backpeddling.

PAID search which I don't

PAID search which I don't think WMW has ever done

I am almost 100% certain I saw WMW Google AdWords ads at one point.

Lets hear what Brett said....

Hi Folks

For the sake of further discussion.

If you are interested in reading what Brett wrote already last November about Attack of the Robots, Spiders, Crawlers.etc, here is the relevant thread of WMW

I hope this helps.

Blocking Robots and Guerrilla Warfare, Heh.

I have often thought that one could conduct guerrilla warfare with the engines by using robots.txt to give some engines exclusive access to some limited but critical portions of your site and deny it to others.

Of course, with all protests you need to publicize it and get other webmasters to go along. Something like: "You won't find our reviews on search engine X, but you will find them on search engine Y and here is why..."

Brett says he only let the

Brett says he only let the spiders in because he needs them for site search. He could always try upgrading to a proper forum software package, instead of having to use third-parties to provide basic features.

No alexa

Jumped the shark is the correct motorcycle.

Tricky Wiki

If you believe everything you read in the wiki as gospel, it will be a long weird trip

Jumping the shark in TV is usually when they go off the deep end altering the storyline in a weird way, or resort to stunt programming, and if blocking all bots and then calling Jakob out to do the same wasn't stunt programming then I don't know what it was besides being the best link bait in SEO history making so many new links that he may not need search engines ever again.

For those that scoff at Brett needing search engines, you may be surprised that some of us can maintain 300K visitors a months without one as I've lost Google for 3 months before and it was just a dent in my traffic, not the end of the world. It's funny how those that decry search being the nexus of the web usually have very short-sighted marketing and dropping out of a search engine could be fatal to them.

As a matter of fact, in my web stats total referrers lists Google as 4th below direct bookmark traffic and a single link from some very popular web site, and about 40% of the rest of the traffic is from a network of links and always has been. I could go cold turkey and survive as even my competitors have links to me if you can believe that. Not purhcased links, no blackhat, not even greyhat, site is just an authority in it's area with thousands of links which is what I suspect Brett learned but he needed to disconnect first to discover about his site what I already knew about mine.

Brett says he only let the spiders in because he needs them for site search. He could always try upgrading to a proper forum software package, instead of having to use third-parties to provide basic features

Give me a break, they all suck, most of those searches either return so many results it's a why bother searching for a needle in a haystack or just the opposite they couldn't find their own ass with both hands and a flashlight.

If you want to find something in a forum, the search engines get the job done.

Period.

Maybe SE's are just a dent in your current traffic

..but I think anyone is nuts to ignore the traffic from them. If you've got a stable base of users you can likely 'maintain' your traffic for a short period of time - but at the huge expense of new visitors.

And sorry, I ain't buying that in most cases new traffic isn't coming from the SE's. You think WMW doesn't get almost all of their new visitors from the SE's? Folks doing searches on CSS or SEO or something and up pops a thread? then 6 months later they sign up as supporters or convention attendees. Sure, he's got his 'already know about the site' bulk of his traffic, but how long before new registrations dries up if they're not in the SE's? Not today maybe, but it'd happen. And WMW knows that too, sure as sugar. If there wasn't money involved, he'd have left it out of the SE's. But money is involved, so back into the SE's the site went.

One thing you gotta say for

One thing you gotta say for Brett, whatever the reasons may or may not have been the robots.txt thing he sure knows how to start some link baiting discussions, whether it be the original deed, the robots.txt cloaking, the robots.txt blog or the robots.txt NNG challenge

If you wanna learn what viral truly means in this industry just look at what has happened in this saga.

Personally I think he has taken a LOAD of lemons and made some pretty decent lemonade

Oh Wheel....

I had traffic before search engines, I'll have traffic after search engines.

Before the search engine was the directory, before that the link pages, once you find one relevant site you start a journey to find others of a similar bent, today it's blog rolling.

And new traffic doesn't have to find YOUR page via a search engine, just anyone linked to your site, or the article about your site, etc.

But you're right, I wouldn't toss the search engines either as it's all part of the mix.

My point was people that think SEs are the end-all be-all of web existence are the new breed that never lived online before the SE existed and we all did fine before the paradigm shift called Google happened.

Actually, we did better as all the directories were built buy hand and there was no directory spam, no MFA sites would've been allowed in the listings, wow - how DID we survive before being ruled by faulty algorithms that SEOs spend all day gaming?

Just for the record

Brian

Brett says he only let the spiders in because he needs them for site search.

In fact Brett was under tremendous pressure by several WMW members, also paying members asking him to reinstall Google site search.

You can see for yourself here on this thread

old

Quote:
My point was people that think SEs are the end-all be-all of web existence are the new breed that never lived online before the SE existed and we all did fine before the paradigm shift called Google happened.

In the context of SEO, that is the case. That's what pure SEO is. Let's not complain that the SEO's are putting too much importance on search, ok?

Quote:
Actually, we did better as all the directories were built buy hand and there was no directory spam, no MFA sites would've been allowed in the listings, wow - how DID we survive before being ruled by faulty algorithms that SEOs spend all day gaming?

Yeah, that was yesterday. Now I wonder if you didn't have legacy sites from those old days (with their trusted link networks and granted authority) what would you be saying? Start from scratch today the same way?

ThreadWatch started as an SEO forum, and while WMW-style general webmaster stuff is important let's not over generalize.

Old Dog, New Tricks

Yeah, that was yesterday. Now I wonder if you didn't have legacy sites from those old days (with their trusted link networks and granted authority) what would you be saying? Start from scratch today the same way?

I launch new sites all the time but incorporate the new with the old.

Lots of marketing methods have merit as link building is far from dead and on topic reciprocal links are far from useless as many decry. I do some things for SEs and others are done for networking as I'm a firm believer in organic traffic and not putting all my eggs in just one basket.

We only depend on search engines because we, the webmasters, gave them the power.

If you all block the bots tomorrow Google stock would be tanked in a month and it would be back to everyone fighting to get sites into Yahoo and DMOZ all over again.

We, the webmasters, have the ultimate power.

We, the webmasters, gave it up willingly and now complain about how it impacts our lives.

Perhaps a 30-day organized Google-ban would send a message we aren't to be treated like cattle?

You control the crawlers, how you use that power is up to you.

30 day ban

Quote:
Perhaps a 30-day organized Google-ban would send a message we aren't to be treated like cattle?

Great idea!
Would you like to lead it by example then? :)

utopian

HautDawg has a blog that makes $1 per day and he's happy to participate in AdSense. Sure he wishes he could rank for "ringtones" but he's ok with the $30 a month he gets for his 72nd page apearance for "fly mp3 phonez". He never heard of robots.txt and probably never will.

Go ahead and get every webmaster on earth to block the bots. It'll just make HautDawg and his budz richer.

That's Fine

If all the real content went on strike with bot blocking and left nothing online but crap the net results would be the same - mass abandonment of the search engines.

Just wrote this...

...in another post. One should never just rely on SE's for traffic, it's webmasters 101. I was commenting on Google pushing and pulling.

Already had a Ban

Quite by accident for about a week, so I've been off the SE juice recently ;)

The response was quick too, one crawl with blank pages and my listings slipped from top 10-20 spots down to 80 or vanished and the minute I fixed the bug and they were crawled again they popped right back where they were before.

You go next...

No effect

Yeah, banning those bots had no effect at all on his traffic and I'm certain he didn't need it:

http://www.alexa.com/data/details/traffic_details?q=&url=http://www.webmasterworld.com

Alexa?

Do real webmasters use Alexa?

My web site traffic is very stable but Alexa reports wild swings and the truth is those swings are just Alexa toolbar users coming and going and have nothing to do with reality. Hell, if all WMW the moderators alone installed Alexa it would skew the results off the charts.

So, if you think that Alexa is a valuable measurement of anything useful then you must be drunk, step away from the keyboard.

Good for comparisons

It's true that Alexa is not real reliable data, but it is interesting to compare a comparable site, in this case, say another seo or webmaster forum, and see how the graphs correspond. If they both have a similar up or down pattern, then it's probably just user behavior for that month. But if one goes way down and the other business as usual, it's fairly telling.

Try it with that Alexa WMW graph and see for yourself. Just plug whatever other forum you'd like into the "Compare Sites" and have at it. It's a pretty cool tool when used that way.

Comparing over Time

Alexa is good for comparing data over time. The ranks, estimated traffic, etc is all relative with them. Fact is that Alexa registered a signifigant drop in users going to that site during the exact times that the bots were banned and the site was out of the search engines. So unless the traffic of users using Alexa at WMW substantially decreased in that exact period, then re-installed it as soon as the bots came back, I'd have to say he took a pretty nice hit in traffic.

I didn't say that search

Dr. Nielsen responded in a recent interview to Brett's challenge:

I didn't say that search engines are bad or that there's any harm to a website from being listed in a search engine. I just said that they are taking too big a share of the value that's being created by the many people and companies that build original websites.

I also think that websites are advised to rely less on search engines for their business. It's better to emphasize loyal users in your Internet strategy, for example by having an email newsletter.

Right now, search advertising is found money for many sites because so many other companies continue to be clueless about the Internet and waste their budgets on banner ads that don't work. As long as competition for keywords is low, you can buy clicks for much less than you make from those users. But that's not going to continue. Sooner or later, keyword prices will be so high that most sites won't be able to get a positive ROI. But as long as prices are low and you can rack in fat ROIs, then of course you should keep doing it.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.