Googles Secret Army? - Eval.google.com

28 comments
Source Title:
eval.google.com
Story Text:

Slashdot spot a story about recruitment of manual spam hunters, complete with flash animation at eval.google.com.

Unfortunate choice of url...

Comments

Been doing this for ages

This has been going on for a while, cool to see the screens but nothing really new Int'l Sea­rch Quality Coord Nov 15 2001

yeah, we talked about

yeah, we talked about similar job postings last year - it'd be odd if they didn't do some user studies

scary stuff tho

If those screens are real its a bit scary. I always thought the system was just set up for evaluators to make sure that some english site doesn't pop up in Japanese results, for example, because of Japanese link text. Maybe it still is, but some of those tools in the screens look people's rankings can be in the hands of a few others who aren't "that" skilled in the art.

"If those screens are real

"If those screens are real its a bit scary. "

Agreed here - Google as a mathematically objective determiner of relevance is an ideal I can subscribe to, even if the technology doesn't always work well.

However, what we seem to be getting is "DMOZ does rankings".

Some degree of human intervention was rumoured for a long time, but 2 things concern myself about this project:

1. The reliance on human subjective interpretation for ranking pages, rather than an attempted objective standard

2. The alleged nuking of affiliate sites, excepting Google partners.

Wow...

Do they give 'em all pointy white hats when they start the job? ;)

Penalty confirmation

Check out the footer of the jpeg here

Seems that the 'offensive' filter could well be responsible for knocking out a few domains this past update..

"Offensive" hotel pages

Yes, anyone in travel should take look at that.

Interesting that they believe that affiliate links in themselves may be enough to damn a page (take a look at the site they cite, iagora.com )

They would appear (for better or worse) to be edging towards a DMOZ banning of any affiliate site.

Make me re-look at some of my pages!

Text from image for those

Text from image for those without 20/20 vision.

Quote:
As you know very well by now, we try to maintain a distinction between the page that is the result of search and the broader site on which the page is located. The application of this distinction sometimes results in the need to assign Offensive ratings to some pages while merit-based ratings to other pages on the same site. For instance, a domain that more often than not provides price comparison and reviews may have pages on which an affiliate link to one merchant site is all that the user can find: no reviews; no price comparison no nothing. Such pages should be labelled Offensive. That, even though the other pages on the same domain provide quality information, unique content, helpful functionality, etc

Seems from the dates on the images and flash movie these are from last September - wonder how much has changed since then?

Older than i thought

The oldest mention on a forum i can find is April 2003
http://www.webmasterworld.com/forum3/11972.htm
even gets a reply from GG

Started appearing in stats at least as long ago as September 2002
http://www.projecto-oasis.cx/stats/www/ref_200209.html

Searching for

Searching for eval.google.com on Yahoo is much more interesting than searching anywhere else.

Thanks GW, just spotted

Thanks GW, just spotted another bug, we've had "Drupal - Marketing & Technology Discussed" on every title tag since the upgrade hehe....

There is a new thread at WMW, with GG and OP

The WMW thhread has input from GG and from the original poster, Henk van Ess. GG appears more concerned with the ethics of the info getting published, than with what they are up too!

For those banned or too idle to read the WMW thread, the relevant bits are

Quote:
walkman, your comment illustrates a misconception that I've seen in a couple places. The system that was up at eval.google.com was a console to evaluate quality passively, not to tweak our results actively. But when Henk van Ess submitted his own blog to Slashdot, he asserted "Real people, from all over the world, are paid to finetune the index of Google," and that made it sound like people were reaching in via this console to tweak results directly, which just isn't true at all.

I have serious reservations about Henk van Ess taking information from one of his own students (who presumably signed a non-disclosure agreement when the student agreed to help rate the quality of our results) and posting that information online. I also believe these web pages said things like "Google Proprietary and Confidential," but it appears that the screenshots have been cropped to exclude that information. Those are the two things that really made me sad, not the "breaking news" the Google evaluates its own results quality. It shouldn't be a surprise that Google evaluates the quality of its results in lots of ways--the fact is that every major search engine evaluates its relevance in many ways.

and the reply from Henk van Ess

Quote:

GOOGLEGUY
But when Henk van Ess submitted his own blog to Slashdot, he asserted "Real people, from all over the world, are paid to finetune the index of Google," and that made it sound like people were reaching in via this console to tweak results directly, which just isn't true at all.

Google Guy, do I read between the lines that you think my postings are irrelevant and misleading? That would be a shame.

Let's go along with your reasoning. If you say agents don't have any influence on the index, I have a question for you. Why pay them for something if it has no effect om the index? Must be charity then.

GOOGLEGUY
I have serious reservations about Henk van Ess taking information from one of his own students (who presumably signed a non-disclosure agreement when the student agreed to help rate the quality of our results) and posting that information online.

I like your posts, but I guess you don't like mine. I'm not aware of restrictions. The pages were shown in a public class. I'm a professional reporter for 20 years. If Google thinks the information is classified, why Debbie (Frost) didn't tell me? I asked her for a comment...

Let's go back to the content. Check the discussions on Search Engine Watch or many other professional SE-boards. Google Guy, do you really think it's irrelevant to talk about Google's Human Quality Evaluation?

Seems GG is having fun with

Seems GG is having fun with stories being slashdotted again eh? Sure fire way to get him all hot under the collar it seems...

Has anyone seen

Has anyone seen Eval.google.com in their logs and later noticed a significant change to what was a stable ranking?

Download guidelines for reviewers

Guidelines from google for their reviewers:

Detecting Spam (word doc)

Rating Results(pdf)

Wow...

That Detecting Spam doc is amazing....

full of wonderful examples aswell, i'll bet the site owners are just sooooo happy...

That detecting spam doc

there's a bit of me that finds it hard to believe that Google are issuing unprotected word docs with wonky formatting, not using any type of corporate identity (wrong font, no branding), using words like 'punished'(even if it's "quoted"), and with no secrecy statement or disclaimer on them.

Most of me believes it.

When it comes to spam, i

When it comes to spam, i find anything at all being "leaked" or "discovered" about Google's policies and practices suspect.

Generally i keep it to myself as i know it sounds whacky :) but that whole patent thing, with all the nice recognizable names and simple to understand clauses just had me thinking, yep, another bit of disinformation from $G - i just don't think they can keep up with it, and continuing the tradition of FUD amongst the enemy (that would be you and me) seems to be the only course either open to them, or perhaps the only one they want to follow. (cheapest..)

place your bets

well either it's so bad it's genuinely true or it's a fake (which of course doesn't mean the person publishing it doesn't think it's entirely genuine) or it's ultimately deniable spin. I'm not betting against any of those at the moment :)

Whatever it is there's one 'fact' in there even I could prove is wrong, and was wrong 12 months ago.

If you look at the providence of the document

...it looks to be genuine.

If you can bear reading through the whole of the WMW thread, and concentrate on the stuff from Henk van Ess, I think you can conclude that the strong balance of probability is that it is genuine.

Sheesh, its most illuminating to. What is the scale of this operation? Anybody like to say how many are employed, how many sites a day they vet, and get a monthly total?

GG is peeved

Hes posting about copyright issues at WMW.. lol

So far at WMW he has only attacked Henk, not the validity of the documents.

Now, this is a fun dance to watch.

more on it

here at the search bistro site. i haven't read enough to form an opinion one way or another.

And someone has added this comment on Henks site..

..its wasn't me, but I do agree with it.

Quote:
Also this GoogleGuy thing is getting stupid know, a lot of companies depend on Google Adwords and on the Google SE so they want and need to know the exact guidelines so it is time that Google comes out with realnames, a real spokesman with a name, real guidelines, real information ,real customer support and no standard reply emails! Henk well done, GoogleGuy get your real name and information out !

Its all a bit the "good old days" of the divine right of kings, when kings considered themselves to be deities and above mere motrtals.

There is a ding dong punch up between GG and Henk at WMW

Quite interesting thread there now bwetween the two of them, I particularly like from GG

Quote:
I sincerely appreciate it if you've stopped posting documents and taken out the employee's name.

Eat your heart out Deep Throat!

Heh

That's the most fun WMW thread I've seen for ages!

They advertised these jobs lately in the UK, been trying to get my mate to sign up...

yeah that is too funny. Two

yeah that is too funny. Two years ago this kind of stuff never would have flown, like RC says though 'everything on the net is measured in dog years'

well...

after reading some, (and knowing it's the hip thing to hate Google lately) here's my two SERPs:

* i don't think it's that big of a deal for Google to pay people for quality control. from what i've read, that's all it is - people look over SERPs and grade them. those results are given to engineers to tweak the algorhythm

* as a journalist, if these are copyrighted, internal documents, they shouldn't have been posted. it'll be interesting to see how/if Google goes after Henk van Ess. could he have written it up w/out breaking copyright laws? yes. would it have been as sensationalistic? nope. w/out the screenshots, it would've been YANB (yet another new blogger) posting info and i doubt he would've gotten a slashdot mention, even if he still posted it to them himself. slashdot is getting anti-google as much as they are anti-microsoft, imho (not that i spend a lot of time there, tho...)

anyway, those are my (basic) thoughts on the matter. i'm hoping that as the project eval continues, it helps make the SERPs cleaner, which i'm sure it will. i can't see webmasters paying these people to whitelist inappropriate sites for extra $$ and google not noticing the trends. that is, i'm sure there's meta-moderation going on. and since it's not 'live editing' of SERPs, it would have to get by the engineers as well...

Newsworthy

It isn't newsworthy that Google hire these people...

What is interesting for the seo community and newsworthy is the contents of the documents as it gives insight into how the big goog operates.

Doesn't matter how they gather the data, it's the fact that they look at it and have policies on how to handle aff sites.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.