Link To TurkPrime

Effective Mechanical Turk: The TurkPrime Blog

How to run successful experiments and get the most out of Amazon's Mechanical Turk

Monday, November 20, 2017

Strengths and Limitations of Mechanical Turk

Hundreds of academic papers are published each year using data collected through Mechanical Turk. Researchers have gravitated to Mechanical Turk primarily because it provides high quality data quickly and affordably. However, Mechanical Turk has strengths and weaknesses as a platform for data collection. While Mechanical Turk has revolutionized data collection, it is by no means a perfect platform. Some of the major strengths and limitations of MTurk are summarized below.
A source of quick and affordable data
Thousands of participants are looking for tasks on Mechanical Turk throughout the day, and can take your task with the click of a button. You can run a 10 minute survey with 100 participants for $1 each, and have all your data within the hour.
Data is reliable
Researchers have examined data quality on MTurk and have found that by and large, data are reliable, with participants performing on tasks in ways similar to more traditional samples. There is a useful reputation mechanism on MTurk, in which researchers can approve or reject the performance of workers on a given study. The reputation of each worker is based on the number of times their work was approved or rejected. Many researchers use a standard practice that relies on only using data from workers who have a 95% approval rating, thereby further ensuring high-quality data collection.
Participant pool is more representative compared to traditional subject pools
Traditional subject pools used in social science research are often samples that are convenient for researchers to obtain, such as undergraduates at a local university. Mechanical Turk has been shown to be more diverse, with participants who are closer to the U.S. population in terms of gender, age, race, education, and employment.
There are two kinds of potential limitations on MTurk, technical limitations, and more fundamental limitations with the platform. Many of the technical limitations of MTurk have been resolved through scripts written by researchers or platforms such as TurkPrime, which help researchers do things they were not previously able to do on MTurk including
  • Exclude participants from a study based on participation in a previous study
  • Conduct longitudinal research
  • Make sure larger studies do not stall out after the first 500 to 1000 Workers
  • Communicate with many Workers at a time.
There are however several more fundamental limitations to data collection on MTurk:
Small population
There are about 100,000 Mechanical Turk workers who participate in academic studies each year. In any one month about 25,000 unique Mechanical Turk workers participate in online studies. These 25,000 workers participate in close to 600,000 monthly assignments. The more active workers complete hundreds of studies each month. The natural consequence of a small worker  population is that participants are continuously recycled across research labs. This creates a problem of ‘non-naivete’. Most participants on Mechanical Turk have been exposed to common experimental manipulations and this can affect their performance. Although the effects of this exposure have not been fully examined, recent research indicates that this may be impacting effect sizes of experimental manipulations, comprising data quality and the effectiveness of experimental manipulations.


Although Mechanical Turk workers are significantly more diverse than the undergraduate subject pool, the Mechanical Turk population is significantly less diverse than the general US population. The population of MTurk workers is  significantly less politically diverse, more highly educated, younger, and less religious compared to the US population. This can complicate the way that data can be interpreted to be reliable on a population level.

Limited selective recruitment

Mechanical Turk has basic mechanisms to selectively recruit workers who have already been profiled. To accomplish this goal Mechanical Turk conducts  profiling HITs that are continuously available for workers.  However, Mechanical Turk is structured in such a way that it is much more difficult to recruit people based on characteristics that have not been profiled. For this reason while rudimentary selective recruitment mechanisms exist there are significant limitations on the ability to recruit specific segments of workers.

TurkPrime offers researchers more specific selective recruitment opportunities, and has some features in development to help researchers target participants who are less active and therefore more naive to common experimental manipulations and survey measures. TurkPrime also offers access to PrimePanels, which has access to over 10 million participants, who can be selectively recruited, and are more diverse.


Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior research methods, 46(4), 1023-1031.

Friday, November 17, 2017

Upcoming New Content on the Blog

Greetings Reader,

We would like to inform you of upcoming new content to the blog! We have been posting sporadically, but plan to have weekly content for you in the future. Posts will cover a host of topics relating to conducting research online. We will aim to provide content that can be useful to both novice and more experienced readers. Content will explain features of MTurk and TurkPrime, as well as Prime Panels, that people may want to better understand. Posts may also often have suggestions for best practices based on our knowledge of how to get the most out of online research on MTurk and beyond. We will also discuss hot topic issues as they arise, and are additionally happy to take some requests from readers for future posts as well.

The TurkPrime Team hopes that you will find these blogs informative. We are committed to providing information to our users that can enhance their use of TurkPrime, and their knowledge of issues in online research.

Tuesday, October 31, 2017

New Feature: Select MTurk Workers by Big Five Personality Types

Run Studies Targeting Specific Big Five Personality Types!

TurkPrime introduces a new Big Five personality types qualification: Now social science researchers can run studies targeted to the Big Five: 
  • Extraversion
  • Agreeableness
  • Conscientiousness
  • Openness
  • Neuroticism

Each personality type can be specified by selecting a range of values from 0 (low levels of extraversion, agreeableness, etc) to 6 (high levels of the personality trait), as shown below. Note, the scales take reverse scoring into account. 

Each worker is classified for a particular trait based on answering the Ten-Item Personality Inventory (TIPI) (Gosling, et al, 2003). The workers' personality score (0.0 - 6.0) is the average of the responses they provided for that trait across multiple occasions. 

Coming soon: Non-Naivete scores to target your studies to workers who have not been exposed to many social science studies. 

Reminder: To reach a large audience that has had relatively little exposure to studies in the social and behavioral sciences and are fresh faces use TurkPrime's Prime Panels. Prime Panels reaches over 20 million participants in the US, and more around the world. 

Tuesday, February 28, 2017

New Feature! Restarts now available when Hyperbatching

Now available "Restart HIT"-- using HyperBatch!

The TurkPrime "Restart" feature (which Restarts HITs that have become sluggish; see our blog post on Restarts to see all the benefits of this feature)
can now be used in conjunction with HyperBatch. This was a highly requested feature from our users and we are happy to announce that it is now LIVE. The only difference between using the restart feature with HyperBatch and using the restart feature without HyperBatch is that you will only be able to restart the HIT once per day with HyperBatch. This helps ensure that our system will not be over-burdened. 

However, as always, when workers view HITs, they now find your HIT at the top of the MTurk HIT Dashboard. When restarted, HITs that have aged and have become hard to see by MTurk Workers will have the visibility of a brand new HIT - because TurkPrime has created a new HIT and associated it with the same specifications as your original HIT!

Additionally, when using Restart, Workers who have completed your original HIT are automatically excluded from taking the newly created HIT and are unable to accept the HIT because TurkPrime has created a Qualification Type that excludes them! 

There is nothing that a Requester needs to set up to enable the Restart feature. It appears in the TurkPrime Dashboard as soon as your HIT goes Live.

For more on the benefits of using Hyperbatch, check out our blog post

Friday, December 2, 2016

Verified US Region Targeting

Verified US State and Region Targeting

Problem: Many researchers wish to target participants from specific states or US regions of the United States like from the Northeast or the West. The issue that they often encounter is that using the MTurk Geographic Qualification specifying the US state is often not adequate to ensure that the participants reside in the specified state. 

The MTurk state may be incorrect since workers move since setting up their MTurk account and it has been reported that MTurk uses the worker bank location as the worker's state which may have never been the state of the worker's residency. TurkPrime internal quality tests have shown that up to 25% of the worker reported states using the  MTurk Geographic Qualification are inconsistent with the worker's state as reported by their IP address.

Solution: The TurkPrime Pro feature to verify location by Ip address and the US Region selector will only qualify workers whose IP address has been verified as being located in the study's required state or US region. All other workers will be disqualified from taking the study.

In addition, to run regionally targeted studies, TurkPrime now includes a region selector which automatically includes IP state verification to endure quality results.

Dynamic Secret Completion Codes for SurveyMonkey

TurkPrime Supports Dynamic Completion Code for SurveyMonkey

Users of Qualtrics and Google Forms have long enjoyed the ability to integrate dynamic secret codes for each participant in their TurkPrime study. Dynamic codes ensure that each MTurk worker participating in your study receives a unique code completely eliminating the possibility that workers shrae secret codes.  In addition, with auto-approval enabled, those workers are automatically approved without any need for reserarchers to manually check the worker supplied secret codes and approve workers.

Now, TurkPrime users who host their studies on SurveyMonkey can also use dynamic completion codes as follows:
  1. Check off "Dynamic Completion Code For Qualtrics" when you design your TurkPrime study
  2. Redirect participants at the end of your SurveyMonkey study to

Friday, September 9, 2016

How to Create a Universal Exclude Worker List


Requesters may observe that some workers, even those with high Approval ratings, may not perform to their expectations on a study. Sometimes this may result in rejecting their work which affects the Worker approval rating. But, often the work is not acceptable for research but is not worthy of rejection, or, it may simply be the policy of the research lab to approve all assignments for IRB or some ethical standard they may follow. 

At this point the researcher may wish to exclude these workers from all future studies. MTurk has an option to Block a Worker (available through the API) but our experience has been that this solution is somewhat draconian and extreme: the effect of a Worker Block can trigger the suspension of the Worker's MTurk account. 

(Source: When I was a newbie Requester in 2012 I blocked some workers who gave me inconsistent and poor responses . I learned the hard way by having my Turkopticon rating suffer and the MTurk Worker discussion groups spread the bad word. I responded to the Worker complaints and Unblocked them to undo the damage to their reputation -- and mine!)


Create a Universal Exclude List using the TurkPrime Worker Group feature. This exclude list simply excludes all specified workers in this group from taking a study with this Group Requirement. When you design your studies, just add this exclude group to your Worker Requirements and none of the workers in this exclude group will be qualified to take your study. 

This will achieve your goal of blocking undesired Workers without tarnishing their reputation.