Posted on

First Experiment With “Mechanical Turk”

The completed tasks, not 20 minutes after publishing them!  Click on the image for full-size.

An initial experiment (my first) with Amazon’s Mechanical Turk, a service that allows workers around the world to complete simple online tasks.  While most of the tasks are weird surveys or spamming attempts that pay between $0.05 – $0.25 per job, I decided that paying a reasonable fee would be fair and more likely to be completed.  I asked participants to visit this site and determine the color they saw when their eyes were closed looking at a bright white screen.  Prepaid for 100 people to complete my task, I assumed it would take a week or two.

I took a shower and came back to my computer to find… 100 responses!  In less than 20 minutes the whole project was finished, to my astonishment.

The first 100 responses – not all what I would call “accurate”

A few thoughts on the process:

  1. The results were almost entirely quality – about 93% took more than 90 seconds to complete the task and returned useable values.  The others apparently did not read the instructions very carefully and returned color names like “pink”, “black”, and “rainbow”.  While interesting, I wanted completely objective answers.  Mechanical Turk allows you to reject jobs that don’t meet your criteria so I did; a few sent back emails saying they were upset that I had rejected their one minute of work, but I think that’s likely par for the course with this system.
  2. It is clear that not all the values are “good” – I can’t imagine the scenario that someone sees bright blue.  In the future, I’ll likely proof the data first.
  3. Mechanical Turk will return your results as a CSV file, which is very useful.  It also includes lots of great “extra” data, including time spent finishing the task and the exact time the task was completed.
  4. I am very much thinking of setting out another batch to be completed.  While $100 isn’t cheap ($110 actually, since Amazon charges a fee), the data was fast and quality.  I have been considering a sliding scale so that the price goes down over time, making those who answer early get paid more, those that wait less.

If you are feeling egalitarian and want to help the project without getting paid, you can head over to this page and email me your results.