WMW - we don't want no steenking spiders.

34 comments

In this thread, Brett (did they piss him off at Vegas?) declares that robots.txt is going to stop 'em all at the gate for the next three months.

As stever points out, that effectively knackers the WMW site search well and good. Ours is not to reason why.

Comments

No idea why Brett hasn't

No idea why Brett hasn't upgraded the forums to have it's own site search. Possibly the only forum I can think of that has to rely on a third party to help you find information in it.

>Ours is not to reason

>Ours is not to reason why

WMW moved hosts recently, remember? Think about it....

WW site search

This was a topic debated at a bar with a few mods. One promised a search was on its way.

hehe, then he shouldnt have.

hehe, then he shouldnt have. They know nothing.

Why?

Does anyone know why they wouldn't want to get indexed by the search engines?

From robots.txt

# GNU Robots.txt Feel free to use with credit
# given to WebmasterWorld.

LOL.. perhaps it just leftover text from the last version, but I don't think you need to credit WMW for the following two lines:

User-agent: *
Disallow: /

LOL.. perhaps it just

LOL.. perhaps it just leftover text from the last version, but I don't think you need to credit WMW for the following two lines:

am also sorta surprised that it would have the info about crediting without mention that it blocks all bots. Imagine a newbie webmaster grabbing that code because they think that maybe WMW would probably have what a good search engine friendly robots.txt should look at.

"did they piss him off at Vegas?"

Well did they?
What's this all about.
wasnt there unfortunately.

$andwidth issues

$andwidth issues

plus non profit making conferences

yeah higher hosting costs plus non profit making conferences, things must be a little tight over in Austin theses days :)

You all are assuming WmW

You all are assuming WmW won't get any indexing by the search engines.

I'm sure it'll get indexed...by google anyway.

Quote:
non profit making conferences

I can't believe that.

Doesn't make sense

Quote:
You all are assuming WmW won't get any indexing by the search engines.

A good search engine would not index it with that in the robots.txt. That's telling them not to.

Quote:
yeah higher hosting costs

Can't be more than a couple hundred a month for a dedicated server. I'm sure Brett can afford that.

Quote:
plus non profit making conferences

Those conferences have to be making a TON of profit.

There has to be reasons why they don't want to be indexed by the search engines. Nobody knows?

No freebies?

Those of us who don't pay to read there sometimes check out cached copies of pages. This would stop us from reading for free, but it's not something I'd consider a great loss.

Qwerty, I don't think their

Qwerty, I don't think their paid stuff has been indexed and cached. But you do have to be logged in (free I believe) to read some stuff, so it could be that.

I agree, no great loss for me either as I don't generally read there either.

I suppose it will cut down

I suppose it will cut down on the amount of newbies.

so was this a dare or something?

Starting to look that way to me. It must have been onehelluva bragging party at Vegas:

Quote:
- Well, I could ban ALL spiders and they'd still keep us.

- Cannot!

- Can too!

BT takes a few more pulls on the Pabst.
- When I get back I'll show you!

(personally, i think it was the $andwidth charge$)

WG: Brett Loses His Mind Part 2

Oilman: Blink And It’s Gone

Danny: Brett Tabke drops the nuclear bomb of banning all spiders

I guess BT will be known

I guess BT will be known from here on out as the man who couldn't make 2.1 million pages of well indexed, unique content profitable.

I still don't get it

Maybe it's just a publicity stunt? It sure is getting press apparently.

>getting press yeah, it'll

>getting press

yeah, it'll be the highest ranking robots.txt on the internet. what's the adsense policy on that?

By my read it's an

By my read it's an "experiment". Ban via robots.txt and start logging. All IPs that are not logged in as WMW members are by definition violating robots.txt and therefore EVIL. Analyze that (user agents, IPs, patterns, etc) and use it to devise a strategy for operating with robots.txt back in place (albeit with rules).

I like this part:

Quote:
I hate talking about it at all. It is like talking about security problems in public (given I believe that the majority of bots we see here are owned by members). However, it is better brought up by us, than someone else.

He's right. Anybody sensible just stopped spidering WMW until Brett's robots.txt gives the "ok" again.

Personally I'm not sure he's

Personally I'm not sure he's gone mad here. Those crawlers must cost a fortune in bandwith, and after all they never pay Brett a cent for all that traffic. As any good webmaster, what he really wants to have is users, not bots.

OTOH, the site search is a real problem, and always was. He runs the risk of even more me-too posts, as members aren't able to search for previous discussions on some subject. This, of course also diminishes the value of the past discussions, as if you can't find them you can't read them.

< speculation >
(1) Did he lose data with the move to new host?
(2) Will he run a Google Search Appliance?
< /speculation >

All IPs that are not logged in as WMW members are by definition violating robots.txt and therefore EVIL

Eh... AFAIK, you don't need to be logged in to just read the site? Using a standard browser such as Firefox or IE.

Enforced log-in

>>Eh... AFAIK, you don't need to be logged in to just read the site? Using a standard browser such as Firefox or IE.

also - everyone will have to login to access the site starting now.

Classic So, when faced with

Classic

So, when faced with increasing irrelevance, what do you do? Lock out spiders and non-logged-in users of course...

Re: getting press

But what a silly way to get attention. There has to be a better way to grab buzz.

I have puzzled over it for a few minute then decided...

who really cares

Success

A result. No TBPR, zero results on a domain name search. Slightly quicker than the 26 (?) steps to Google heaven.

Brett is posting that he can finally sleep at night. Now, 'who really cares'? - well, I'm more intrigued than concerned. I know Rackspace ain't cheap, but I'm with Littleman (above) here.

Nick - quick - you might be in with a chance to bid for Update Names here. Trouble is, they never get to the top of the alphabet - 'Update Wanker' will have to remain a pipe-dream. Bollocks might be in with a shout, however.

Could just be an interesting

Could just be an interesting study to see how well the user base is retained with limited new referrals coming in?

If I wanted to bump up the value of a site, I'd like to be able to say to potential buyers, "OK so we get xyz visitors from SE referrals each month - but we don't need them - that's how stable our community is".

WmW is a monster of a site and must be hell to run. It isn't monetised, but you could be sure if it were to be that a large part of the sanctimonius spammer crowd (white hats ;)) would march out in protest at this travesty.

WmW doesn't need SE referrals but it does need its community, particularly a lot of key individuals. IMO, banning all bots from the site would have been an easier choice to make than deciding to monetise the site, where the risk is greater.

Not monetized?

>> a large part of the sanctimonius spammer crowd (white hats ;))

I presumed that this was the target market these days...

Hmmm, I didn't realise how

Hmmm, I didn't realise how much I searched for old articles on WMW until today - as someone who's very much still learning, I do at least a couple of searches a day.

By doing this Brett isn't just sticking two fingers up to Google, he's critically imparing the usability of the site. For me it's pretty much useless now.

Oilman, get outta bed and update the spider watch

Looks like Google might be the winner in the instant gratification department
http://forums.searchenginewatch.com/showthread.php?p=67026#post67026

Oilman needs to update the spider watch, he's as slow as Yahoo.

well i posted over on his

well i posted over on his blog ... but he has me on moderation !! ... so a few links going his way :)

DaveN

all updated...

was travellin to Seattle last night and the inlaws craptastic wireless put me behind :)

sorry for the premod kids but you know how those blogspammers are ;)

>to Seattle Damn lousy

>to Seattle

Damn lousy border patrol we have.

lousy border patrol

>Damn lousy border patrol we have.

This should apply to canucks as well...

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.