First! http://digg.com/videos/comedy/Video_A_First_Commenter_Tells_All

Not to mention why people contribute to Open Source software.

I suspect that Amazon were hoping that Mechanical Turk would be attractive to companies trying to get work done by people living in low-wage countries - where $2.31 for a 20-min task doesn't seem like such a bad idea.

According to the BBC (http://news.bbc.co.uk/1/hi/business/4436692.stm), the average pay of a software engineer in India is around $5/hour - offering $2.31 for 20 mins work isn't unreasonable in those circumstances.

Another good one is the ESP game. If you haven't done so, watch the Google Techtalk video on Human Computation and you'll see what I mean.

Chris

Chris beat me to it. The Human Computation video is the best thing I've seen in ages. One of those moments when you realise that the person you're listening to is scarily smart.

Cheers,
Colin

I just got a email from the HR department--they said they will be increasing our intrinsic rewards and henceforce will no longer be paying us a salary. Wow, I've never felt so rewarded, but somewhat betrayed. Fortunately, the new hires will not be paid in the first place and they will only feel pure intrinsic reward for the duration of their careers.

Wow, I never expected my dad to show up in your blog. I'll have to ask him how he knows Mary Poppendieck.

A couple of things:

1) I visit (and buy things at) Amazon's site regularly (as Jeff said, primarily because of the reviews). Until this article, I'd never heard of their Mechanical Turk program.

2) I keep all my Amazon reviews in a big Word document here. Over the years, it looks like I've contributed 213 pages (or 97,055 words) of free reviews to Amazon. If they want to give away money, I'll gladly accept 5 cents a word :) .

3) While my son was growing up, I gave him an allowance of $1 per year of age (starting when he could understand what was happening). The purpose of that allowance was to allow him to learn how to handle money. I let him know that was what I was doing and that the allowance would stop when he reached 16. At that point, if he wanted money, he'd have to get a job. He seems to handle his own money OK, so I guess that was a success.

You make a very valid point about the monetary rewards. Arguably, one of the most popular uses of the mechanical turk has been for a task which there has been no monetary reward at all. In January 2007 when database pioneer Jim Gray disappeared, some folks got a Digital Globe satellite to fly over the area of the Pacific where his boat may have been. The problem is that the satellite returned black and white images with a feature resolution of slightly better than 1m. Jim's boat would have then showed up as a little white speck about 6-7 pixels long and 2-3 pixels wide. Machine techniques to filter the data would have been very difficult because of the small feature and the number of potential false positives. However, by using the Turk and providing no rewards, people were able to search through thousands of the images pretty quickly. All a person had to do was check whether or not the image may have something interesting to look at some more in it. A perfect example of a non-monetary reward motivating people, many of whom were highly paid computer professionals, to take some time out of their schedule and do something.

I'm certainly in the camp of not tying allowance to chores. I know for certain that my son would gladly do without his allowance to avoid doing the dishes, if I gave him the option.

I participated in the Mechanical Turk program while it was still in its beginning. However, back then, the jobs offered differed from those which are offered now. They were not spammy and were quick to do. Most of the jobs offered were to identify among a series of 5 pictures the one that described most accurately a business or an address that was supplied. It was paid 5 - 10¢ per image, but it took 20 seconds to complete the task.

I think that this approach was pretty nice because the complexity of the task was quite big for a computer but trivial for a human being.

I stopped participating when I left my job where I had access to the internet and pretty much nothing else to do. Managed to get a book out of them with this.

I have worked at Mturk now for almost a year and have made over $1000 doing various types of HIT's. The ones that mostly paid anything worth a darn and kept me at it were the RQ's(research questions)asked from nownow and as well Askville, but things have changed over the last week and I am not really sure what is going on. It is hard to only do the answering at Askville.com now as I don't know what the "coins" are going to be worth, maybe only products, not cash.
I think the idea is a good one for business' such as castingwords and others and was good for us who could manage to get Great Answer Votes from the RQ's and thereby stay in the top 30 earning the bonus.
The bonus made it all worth the competition which is a driving force for me too, I like competition.
Uclue pays more, but I can't get in there yet, they already have their set people.

My questions for the father mowing the lawn is: do the other teenagers have jobs? Do they have an actual respect for the value of a dollar, or are they just spoiled and lazy?

Patrick Wagstrom: Did they find him?

Minor clarification: there aren't 128 tasks in Mechanical Turk right now, there are 128 *types* of task, each of which has up to several thousand actual paying tasks in them. For example, the top task type today is "GIS - Image Tagging" which represents 2,397 five-cent jobs or around $120 of actual work.

Telos: no, Jim Gray was never found.

The irreversible nature of monetary rewards reminds me of working in retail.

I used to work in IT for a local retailer with 20+ department stores and 80+ lower-end stores. Their constant lament was that the shoppers would only buy items on sale. Items not on sale would barely move off the floor. They even tried to target certain items that would never go on sale, so as to condition the shoppers to actually pay normal price from time to time.

It didn't work very well.

-Eric

I've used mTurk for a few months now (requestor and worker) and while I tend to agree that it is a "solution looking for a problem", I think that it is a valuable tool in a small website or developer's toolbox.

That noone has really leveraged it effectively yet does not mean that noone will. I do think there will be some success stories to come out of it in the future.

You should read Douglas Rushkoff's new book, "Get Back in the Box". Much of it is devoted to the concept that the most effective workers do their job mainly for the intrinsic satisfaction they get from it, and that adding external motivators decreases their productivity.

Paradoxically, perks like on-site massage and stress reduction classes end up making jobs more stressful by implying that the job itself should be stressful.

Matthew Martin hasn't got it quite right. The discussion is more about keepng the office open at the weekends so you can, if you choose, come in an work on an interesting project or one of your choosing that may possibly benefit the company. A lot of people would come in (similar to Google's 20% of your time is spent doing whatever project you want to).

Admittedly, you have to get past the basic pay for your job first, and the intrinsic rewards are not a replacement for doing more company work, but for doing different work to your daily job, work you want to do and are not forced to do.

Check out "Punished by Rewards" by Alfie Kohn for more on intrinsic motivation.

I really wonder how all of these parallel markets will integrate in the end. I especially wonder about who will benefit most of all of this free capital. Amazon certainly does. So do advertisers. The value of Digg, Flickr, Del.icio.us isn't a matter of technology, but of attention and participation. Playgrounds are the new economy. Just build and amusement park and let the people in as long as they bring something in return, then cash in on their behalf. Artists, writers, engineers, all of their work, you can benefit from, freely, without signing a single check, or even noticing their name. They are your new consumers.

I think the McLuhan people call this a reversal...

Not long ago I had a chance to speak with Peter Cohen, director of Amazon's Mechanical Turk, and the guy who originally championed the idea for the service. We were discussing the value of documentation. I was bemoaning the fact that my employer might adopt an advertising driven revenue model. Peter said there were only two ways of making money on the Internet: advertising or subscription.

I pointed out that Wikipedia seemed able to make its mark with neither. I've worked on products where we used a Wiki to get the community to drive creation of content. We didn't have the success of Wikipedia with our documentation project, but there is some prestige associated with contributing to a Wiki project. I wish I would have thought to mention the value of the Amazon user reviews to bolster my point.

I think the real value of the Mechnical Turk service is the way the community springs up around it. Philip Greenspun says the one thing the Internet is good for is building communities. All the profitable Interent sites that survived the bust in 2000 had built communities around themselves: Amazon and its user reviews, e-bay and its transaction voting. Mechanical Turk has already seen its own grass roots effort to build a community spring to life and Amazon had little to do with it.

I'm not sure I agree that the effort has failed already, but I do wonder if Amazon and Peter Cohen have the right stuff to move it forward.

Interestingly, the author behind mTurk should know better -- his thesis was about convincing users to contribute human effort without realizing it. The ESP game, which someone else mentioned, is his work. Amazon seemed to like it for their various tasks for which they could find no usable software.

I tried it out at launch, and it seemed promising, but a few years later and the system hasn't really improved much. I always figured it was intended as a platform for testing AI techniques against humans, without key people knowing it. In that light, hourly rates aren't as important as HITS x price; you figure if you're the only one working on AI, you'll probably wind up with 90 percent of the HITS. Unfortunately, if this was supposed to be a platform to attract financially motivated AI engineers, it hasn't improved much.

But maybe Amazon's goal of finding talented data miners and such has already been met, and the test, as it were, adequately represents the real world problems they face on the internet. In which case, I'd be nice to see some report from the author about it, but I doubt Amazon's about to announce their findings to competitors.

The company I work for (www.vocalabs.com) does something similar. Actually, a few similar things. And we've had no problems with it. But there are some major differences. The biggest is that we're not trying to generalize into a new kind of work, we're just taking things that existed before the Internet which happen to be a little more efficient online.

The key here is that MTurk is a solution waiting for a problem, whereas we had problems, and a computer-mediated, distributed solution happened to be the best solution. I suspect there are plenty of situations like ours, where people work online. It's just that in most cases, it's not an attempt to realize science fiction.

I think MTurk is a credible idea, just with a bad implementation. They failed to realize that it doesn't scale with employers. Reputation matters, and I might trust Amazon to pay me fairly, but I don't trust J-Random-Third-Party. Especially if they can subjectively declare your work unworthy of payment. Nor do I want to work for someone whose values I don't agree with. Furthermore, the pay scale matters. If it's not a decent monthly income, you need to repackage it as entertainment, volunteerism, or both.

Here are the things we can do. You can decide for yourself whether or not they are like the mechanical turk. In both cases, the computer does a lot of the work and the person fills in at one crucial step.

Thing #1: we do usability testing for telephone applications. This isn't much different from taking a survey, except that the participant makes a phone call first. As with regular surveys and in-person usability tests, we give a token payment. (Typically a buck.) But, as others have mentioned, the higher the reward, the lower the motiviation. And the more we pay, the more we have to deal with people gaming the system, rather than just helping out.

Thing #2: we have a virtual call center of people who administer customer satisfaction surveys. This is something a computer can do, but having a machine for a complaint department is just plain a bad idea. This is contract employment, but as with the Turk, specific tasks are sent out by the computer. (A survey is triggered when someone hangs up from calling our client's customer support line.) Our survey administrators expect to make a reasonable income for their time, but the time might be half an hour now and then.

ok, I have to ask. Were you inspired to write this because the "mechanical turk" was the article of the day(a few days ago) on wikipedia?

inquiring minds want to know

5$ and hour?

No wonder their code S&$#s.

Um, one thing you forgot. Writing a review about a movie is inherently *fun*. Writing a plot description isn't. Posting your opinion to your own blog is *fun*. Posting what somebody else tells you to write to a blog you don't care about isn't. Submitting links to Digg which others will find entertaining is *fun*. Submitting junk faxes that you know people will hate getting is not.

The free activities of the social web work partially because of the merit-based rewards of community standing, but they also work because they fulfill people's inherent need to express themselves, to see their mind validated in print. The job is automatically its own reward!

As the first poster pointed out, free and open source software works off a meritocracy as well - in addition it is a marketable skill that's in demand, so it looks good on a resume. BUT (and this will be unbelievable to those who aren't a programmer by nature) programming is also FUN! To a certain kind of person, inventing new programs that make the computer do nifty stuff for them is rewarding all by itself. Even though I've never put an open source project out there (because I'm just not that good at programming) I have quite a few little programs I've written for myself and my household to solve little problems over the years - and every one of them counted as recreation, in the same way that needlepoint and origami are fun and relaxing hobbies (for being creative acts) and crosswords and mazes are fun (for being stimulating problem-solving exercises).

Hence, when you posted the "Buzz-Fizz" problem, the comments were immediately flooded with solutions, even though that wasn't the point of the post and even after you asked for it to stop. You're waving a cat in front of a dog and expecting it not to lunge for it, against thousands of years of evolution.

I did a lot of the work involved in one of the most prolific users of Mechanical Turk for a long time, and we had a very mixed bag of results.

The biggest challenge for using Mechanical Turk is how to decide whether a particular worker's answer is correct. If a computer program could judge the answer, you wouldn't need a human in the first place. If someone is going to decide on each response, then it would probably be easier for them to just do the work in the first place.

Amazon ran into this problem first. The first big set of tasks was to choose a photograph that best showed the storefront for a business. Many people tried to earn quick bucks by either picking any photo at random or writing a script to submit a random answer.

We eventually came up with a scoring system that rated workers on their agreement with other workers. A computer program compared previous results with answers to decide whether we should trust a particular answer. If the computer program couldn't decide, it referred the matter to a human (me) for authoritative judging. It took a while until it could judge all the answers without human intervention, but it eventually got there.

MT is a failure only in the sense that it hasn't revolutionized the relationship between computers and humans, which is a pretty tall order. There just aren't that many tasks that are easier for a human than for a computer AND which can be farmed out to potentially unreliable workers.

It's a success in that for the problem spaces for which the above qualities are both present, MT works great. Reading numbers off of documents, figuring out the name of an album from a photograph of the cover, etc. are all the sorts of tasks that MT does well in.

MT has also been used for things that it's not so good, such as naming your "top 3" of something or other. (This was one of Amazon's seed tasks.)

We thought that most of the MT workload would come from overseas, namely China, Korea, and Indonesia, where paying 1 cent for a few seconds of work might be a good deal for both parties. However, we found that most of our workers were in the US, and as a group, they really wanted to get paid a lot for doing very little. There were some exceptions, of course, but most of my correspondents were somewhat indignant that they were not able to make a living off of determining not-so-subtle characteristics in data.

Give it a few years, then look again.

The ESP game is an example of using intrinsic value (FUN!) to extract additional value.

Game:
http://www.espgame.org/

Presentation by the creator:
http://video.google.com/videoplay?docid=-8246463980976635143

So I should read the comments and not dupe, but I get bonus points for including links. :)

The $5/hr for coding in India has to be purchasing power-adjusted. In the local currency you can live quite comfortably.

Most of us grew up on video games and hi-score lists. Most of us have been training our brains to release feel-good chemicals when we receive points. It feels really good, and thats often reward enough.

I think Amazon could do pretty well by changing their model. Requestors buy points from Amazon, and then assign a point value to a task. Put up a hi-score list for the workers, and offer a point to dollars exchange rate. I bet you'd see lots of workers playing solely for the points.

How much were you not paid to write this article? ;-)

About as much as I'm not being paid to contribute to it! :-)

fascinating post.

> The theory of intrinsic motivation goes a long way toward explaining
> why Amazon's unpaid user reviews are so popular and effective, and yet
> the paid Mechanical Turk service appears to be withering on the vine.

But if we see software development as a game, then everyone is in it for the extrinsic awards?

An example of "Mechanical Turk Service" with a system of intrinsic motivation is Google Image Labeler : http://images.google.com/imagelabeler/
It uses a high score system (like folding@home) to motivate users to label images.

IMHO the real problem for MTurk lies elsewhere:

There's no efficient global micropayment system (yet) to support execution of the cheapest tasks.

With 2 Billion people on this world eager to work for $1 an hour (or much less), MTurk and similar systems would be ideal to distribute work (and wealth) across the globe.

As proof I'd like to point to 'project markets' like rentacoder, getafreelancer and scriptlance which, supported by paypal/e-gold and similar on-line payment systems, prove to be *very* efficient in matching projects with people.

PS Shameless plug (remove if not allowed Jeff):
If you're interested in this space, keep an eye out for my soon-to-be-launched blog on the topic http://www.peopleasbits.com

< tangent >
One of the reasons MTurk *may* not have been promoted as much as it could have been, may be the very negative reviews it received from people in the USA, offended by the fees offered for the HITs.

It's always interesting to see how true globalism is seen as a *threat* to capitalism and not accepted as proof of the much-touted supply/demand theory. What will happen once China gets on a roll. I've heard people say "we'll take care of innovation", but haven't we all heard that before (cf. Japan, Hyundai, etc.)?!?
< /tangent >

Eerie - I've been scratching my head over the same topic (why does the bonus they're dangling in my face just not do it for me?) and came to some articles by Alfie Kohn on the topic - http://www.alfiekohn.org/business.htm
What I took away is that people want to work with a good team on a good product and be compensated properly for it. This is probably why things like goofy t-shirts with in-jokes on them are better rewards (excuse me, external inducements) than money: they're tangible proof that someone out there understands and appreciates what you're doing, and that makes perfect sense to me.
When it comes to birthdays, do you give loved ones a wad of twenty dollar bills or do you give them a present that shows that you know what they want? Which would you rather get?

Go to http://stardustathome.ssl.berkeley.edu/ and help look for particles of cosmic dust captured in silicon aerogel by a space-probe.

The scientists expect 40-100 particles hidden in many thousands of slides (actually they are like movies, your browser receives 20 or something separate images taken at slightly different focus.

You use your mouse to change the focus, and what is on the surface of each cell is at one focus, and any trapped dust particles at another focus).

When you sign up, you go through some practice slides followed by a qualification, using images of particles trapped from a comet's tail in a similar experiment, except that many thousands of particles were caught so were much easier to find. But the scientists aren't sure what interstellar dust particles will look like, they just guess the comet tail particles will be slightly similar. So they can't use an image-recognition computer programme because there is nothing to train it with. They mix in images of these training images from time to time (a bit too often in my view) to keep you on your toes. Also, it's how they determine score: number of cells viewed is one way, but also the difference between the number of random training videos you got right and wrong is how they rank people.

Let's not forget that the person inside the mechanical Turk was a slave. That is very much in line with pay offered through Amazon's service with the same name.

"Let's not forget that the person inside the mechanical Turk was a slave."

Actually, that's not true. Tou can read the Wikipedia article or any of several books about the chess-playing Turk to learn more about it.

Keep in mind that the hidden person had to be not just a capable chess player, but a phenomenal one; it often beat the best players around. This wasn't cheating--this was real chess playing. That's hardly a skill that a "slave" would develop. The people who were the secret players were talented co-conspirators with the Turk's inventor, never "slaves."

Don't tie in money with the work done? Sounds like socialism to me! Lol Love your blog, even though I'm not a software developer!

Nobody's going to ask what happened to Google Answers? That was a good example of a paid "HIT" service.

http://en.wikipedia.org/wiki/Google_Answers

According to waxy.org, the entire archive just got pulled.

http://answers.google.com/answers/

I actually blogged about mechanical turk when it first came out, figuring that it had the potential to change the nature of how we work, but had my doubts about that actual process ... and also wondered about the notion that one could find people who were low paid enough that the kind of work involved would be worthwhile and who nonetheless had regular internet access. (I also raised the issue with the good folks at Amazon about the joys of taxes ...)

As to intrinsic vs. extrinsic rewards - I remember when I was about ten and my dad made an offer to me of one penny for each dandelion weed I could get out of the back yard. After about twenty five dandelions I found that the motivation for the reward just wasn't there. On the other hand, I suspect that if he had just gone out with me and we had both pulled dandelions without any discussion of reward, we'd probably have taken care of the entire lawn.

I've actually found that I prefer for those projects that I have the biggest personal emotional stake in that I don't have a dollar value tied up in with it, though certainly if along the way I make some money I won't argue. Once you start tying a job to a paycheck it ceases being an endeavor to master and simply becomes x amount of dollars per hour, and you then do anything you can to avoid having to actually do any more work than you absolutely have to for that amount.

Good post.

I guess I came to mTurk after all the hullabaloo about "changing the way people work," because I just saw it as a chance to earn enough to buy a couple DVDs.

Mturk does have the potential to change how we work, unfortunately it is based on a bizarre premise that is often touted by people in favour of globalisation, that there is a whole lot of quite literate and skilled people in developing countries who are willing to work for peanuts. This is not the case. If you're educated enough to write properly, you probably are being paid more for your work than what mturk offers, no matter where you live.

I have been turking for 3 weeks now and though I've done many interesting things and some of the repetitive HITs feel like certain video games, I just want to get paid and that is the part that apparently takes the longest.

The majority of my hits are still pending and I think that over time, I will probably stop turking altogether because the novelty is wearing off. Could mturk help by being scrupulously honest about payment and offer intrinsic rewards to keep us going? Of course. But companies have spent the last twenty years deliberately avoiding meeting our emotional needs because they think it's cheaper and easier to operate that way. It's demanding and expensive to keep people happy enough to do a good job. Mturk is a short-cut around that, just like outsourcing is.

I've been watching Mechanical Truk for awhile and also thought it was a failure. I've changed my mind. I think they were just using us for beta testing. Powerset is the real goal.

http://whatis.blogs.techtarget.com/2007/05/11/jeff-bezos-mechanical-turk-and-powerset/

If you want to earn with Google. Join Google Adsense Programme today. Google charges no fee to join this service. You can start earning in dollars instantly. For more information visit: http://magotras.googlepages.com/earnwithgoogle

If you face any problem in creating Google Adsense account or ur webpage, just click on post a comment on my blog and I will be haappy to guide you through the process.

I've been reading a lot about Mechanical Turk because I have a transcribing business. I pay a good rate to people in the Philippines. This is what one of them had to say...

"My reason from not continuing to work at Mechanical Turk is because of the tasks available. When there is a task, it is not assigned to you, it is available. When you start working on that task, somebody is also doing that task so it's like a race to the finish. Sometimes you spend an hour working, then when you finish it somebody had just finished it and that work is already done. Besides the pay is too low. You will need to work on a lot of tasks just to earn $10 and that is for already a week or so work."

Seems the Amazon engineers are quite proud of their Turk. But they've created a monster. Outsourcing is one thing. Pitting low-paid workers in foreign countries against one another is just plain ruthless.

I second the opinion of tim.

What is cheaper than all labor force in the world?
Pay 10,000 "turkers" a cent for a ridiculous "micro-task" that insults the human brain! Just make a "game" out of it, promise rewards or bonuses (which doesn't necessarily need to exist) for good work and rake in the added value. Just wrap "at most one idiot gets paid a bit" or "mass-exploitation" into euphemistic jabber about "competition" and "community" and noone will hear the shot.

But I don't care if bored housewives spend their whole day comparing images of products for a cent. What makes crowdsourcing really evil is the devaluation of work beyond the already dramatic negative effects of outsourcing, the devaluation of knowledge, expertise and education. They break into the translation and transcription business, marketing research and consulting business... everything that can be done via the internet and everything that has to do with brainwork is threatened to be devaluated to a few cent. What a luck that brain-surgery can't be done @home. ("Is this brain convolution needed or not? Click 'no' to remove it")

But this is only one half of the mess, the other half is that enough people are sufficiently brain-damaged to offer their manpower, knowledge and skills to be exploited, not for charity, not for other good reasons but for a few bucks of which Amazon gets even its share back because most turkers order stuff off Amazon.com for their money. A transfer to a bank account costs $4 and many have to work a whole week to earn that much.

Is MT a failure? If you see it from a idealistic viewpoint characterized by the above mentioned jabber, yes. If you see it from the viewpoint of Amazon.com, surely not.
It's capitalist's dream of a global galley on which everybody rows voluntarily and shares bread and water with the owner of the galley. Tim, you're invited to join them when they ruined your business...

Oh, I forgot to rant about the abusive possibilities of MT: Populating forums and blogs artificially is not bad per se, I know that it's hard to get a community going, but to do it with turkers is even cheaper than the existing services. And it explains why many posts are so wonderfully brainless. The funniest thing I've found so far (and I'm watching MT merely for a few weeks) was a completely boring dating agency website ("lowered expectations") trying to push their popularity on StumbleUpon.com. Need a vote? Only 1 cent @ MTurk.com...

This comment was not paid by Amazon - Mechanical Turk...or was it? :D

Darn...truncated a whole text block:
...explains why they are so wonderfully brainless: If you ever plan to visit Seattle, Toronto or Kirkland (wherever that might be), don't check the "I love Seattle/I Love Toronto" etc. blogs for restaurant or hotel recommendations. At least most of the euphoric comments on the menu or the rooms on these blogs are not real and have been written by people who doesn't even know where Seattle is or who write things like "Canada is a Wonderfull city". One of the poor guys who was lurked to lie on these blogs by "Data Services" could not completely overcome his honesty and called himself "Michael Turk"...

fuck google answers and uclue, all these assholes are are just another fucking clique. why should anybody reward 0.000000000000000001% of the internet when there are so many willing and able people able to give much better indepth answers then these sets of assholes?

mechanical turk? it's not a social community, just a virtual sweatshop full of lonely assholes on the internet, and the motherfuckers and greedy assholes willing to exploit them. who the fuck wants to write a 500word essay for 20cents? i wouldn't, but i'd be willing to hunt down these people and blow their heads off for 20cents.

The mechanical turk offers some very interesting avenues to consumer opinions, provided you are able to ask the right questions.

I agree Larry, perhaps the best thing to do with MT is to use it as a very cheap focus group.

it is a take it or leave it mturk. what i can say is everyone is clever enough to make decision whether they want to continue work in mturk. whether they want to "continue" be abused by mturk requesters it is entirely their choice. however if it does prove earn more than their current job, why not?