Mechanical Turk
Posted
November 4, 2008 
Comments(6)
Comments(6)
I may be a bit slow on the uptake, but I’ve just discovered Mechanical Turk. It’s a service offered by Amazon, described as “artificial artificial intelligence”. Here’s the blurb off their site:
Get results from Mechanical Turk Workers. Ask workers to complete HITs – Human Intelligence Tasks – and get results using Mechanical Turk…
As a Mechanical Turk Requester you:
- Have access to a global, on-demand, 24 x 7 workforce
- Get thousands of HITs completed in minutes
- Pay only when you’re satisfied with the results
My interpretation is that by paying them a small amount for each task assigned to them, Amazon have a huge pool of people at their disposal.
Could this be used for certain kinds of online usability testing? Has anybody tried using the service for that? I wonder how precisely you can specify which workers are assigned to your HITs?
From what I understood of the API when I looked at it (at least six months ago) there were ways to filter by way of requiring “turkers” to qualify to do your request or “HIT” by taking some sort of test. I don’t think you could otherwise specify though and… of course you wouldn’t know if people were lying :)
Which kinds of usability testing activities are you thinking would be appropriate? There may be a way around this but I think the way it works would mean you’d have to rely on responses and not be able to directly observe?
Card sorting is an obvious choice, but other ranking activities such as “rank the following features/qualities in terms of importance to you” could also be a good fit.
I have considered using “the turk” for doing transcription. I figured breaking a long audio file into numbered bits and shipping it off to “the turk” to be typed up and strung back together would be cheaper than having transcription done by a service.
However, I haven’t managed to get around to doing it yet. I think the service would be good for all sorts of pattern recognition stuff (like language) that real artificial intelligence [did I just write that?] can’t do.
Interesting idea about the card sorting and questionnaires though.
MT is one of my favourite things on the Internet. Agree there must be ways to use it to evaluate & improve user experiences & have been daydreaming about them for a while now.
Some interesting Mechanical Turk links i’ve collected:
Using MT to evaluate search results
“A reader noticed that stealth search engine Powerset is using Amazon’s Mechanical Turk service to gauge user reactions to search results. ”
Using Mechanical Turk to cleanup business data
“I had a list of over 6000 business names, addresses and url’s of dubious quality that I needed to make sure was accurate. For a brief moment, I thought about checking them myself. But after doing several dozen by hand, I realized that I was violating the principle of Don’t Be Silly(TM)… Enter Mechanical Turk, and the hordes of awesomeness to save the day.”
Moral psychology on Amazon Mechanical Turk
“For a while now I’ve been thinking that recruiting subjects online could lend itself to collecting some really interesting behavioral science data. A few months ago I tried doing this with Amazon Mechanical Turk. To that end, I tried out running one of the standard moral psych survey questions to see what would happen — the so-called ‘trolley problem’”
Why People Participate on Mechanical Turk?
Using MT to answer that question:”This questions comes up often when I describe the tasks that Turkers complete on Mechanical Turk. Therefore, I decided to run a set of studies that will aim to answer this question. Instead of trying to impose my own interpretation, I will let the Turkers speak for themselves.”
Cheap, Easy Audio Transcription with Mechanical Turk
“Bracing for a good four or five hours of rewinding and writing and rewinding, I remembered that this is The Future! So, instead, I tossed the job over to the global anonymous workforce at Amazon Mechanical Turk instead.”
(Andy Biao has also used MT for identifying smaples in mashup: Girl Turk)
Amateur economist
“I recently saw a thread that highlights the distinction between expected value and utility. Would you take a more likely but lower payoff instead of a less likely but higher payoff? Similarly, the St. Petersburg Paradox takes the problem to its logical extreme. By constructing a game that has a series of increasingly rare payoffs of increasingly larger size, a game with infinite expected value is created.”
The Turker’s Gospel
“The Turker’s Gospel is a new version of the Gospel books of Matthew, Mark, Luke, and John, reinterpreted from the King James Version of the Bible by workers at Amazon’s Mechanical Turk.”
MT being used to recruit link spammers
“Amazon’s Mechanical Turk has fallen prey to social media spammers and it is now full of requests to spam bookmarking services for pennies per link.”
Why are you here right now?
A crowdsourced ebook
The Sheep Market
My favourite. “TheSheepMarket.com is a collection of 10,000 sheep made by workers on Amazon’s Mechanical Turk. Workers were paid 0.02 ($USD) to ‘draw a sheep facing to the left’. Animations of each sheep’s creation may be viewed at TheSheepMarket.com.”
The sheep thing is cool. But would be better if they had some sort of aggregation of all people’s sheep :)
I think he has had some exhibitions with all the sheep printed out and mounted in a grid on a wall, that’s a simple but effective aggregation.
I love that some people drew something other than a sheep or drew one facing the wrong direction.