Link To TurkPrime

Effective Mechanical Turk: The TurkPrime Blog

How to run successful experiments and get the most out of Amazon's Mechanical Turk

Friday, September 9, 2016

How to Create a Universal Exclude Worker List


Requesters may observe that some workers, even those with high Approval ratings, may not perform to their expectations on a study. Sometimes this may result in rejecting their work which affects the Worker approval rating. But, often the work is not acceptable for research but is not worthy of rejection, or, it may simply be the policy of the research lab to approve all assignments for IRB or some ethical standard they may follow. 

At this point the researcher may wish to exclude these workers from all future studies. MTurk has an option to Block a Worker (available through the API) but our experience has been that this solution is somewhat draconian and extreme: the effect of a Worker Block can trigger the suspension of the Worker's MTurk account. 

(Source: When I was a newbie Requester in 2012 I blocked some workers who gave me inconsistent and poor responses . I learned the hard way by having my Turkopticon rating suffer and the MTurk Worker discussion groups spread the bad word. I responded to the Worker complaints and Unblocked them to undo the damage to their reputation -- and mine!)


Create a Universal Exclude List using the TurkPrime Worker Group feature. This exclude list simply excludes all specified workers in this group from taking a study with this Group Requirement. When you design your studies, just add this exclude group to your Worker Requirements and none of the workers in this exclude group will be qualified to take your study. 

This will achieve your goal of blocking undesired Workers without tarnishing their reputation.

Reusable Include and Exclude Groups

Problem: You are running a longitudinal study and have identified 1000 workers who you want to allow to take your second phase studies. How do you easily group those workers for easy access. 

Or you want to exclude certain workers from taking a number of your studies and wish to group them for easy exclusion in future studies. How can you do that?

Solution: Use the TurkPrime Worker Groups feature available in the Manage Workers menu. This allows you to create reusable worker groups for either inclusion or inclusion in studies. You simply select whether this group is an include or exclude group, give it a clear name and specify the workers in the group. 

Workers will not know why they were targeted for inclusion or exclusion since the MTurk Qualification name used is a random sequence of characters. The TurkPrime Group Name will not be visible to Workers. This is important so that you can run blind studies (i.e. the workers who are specified for inclusion will know they were targeted because they are exhibiting a certain personality, for example)

When you design your study, and specify Worker requirements, add this group to the worker requirements and all workers in that group will be wither included or excluded, as needed.

Thursday, September 8, 2016

MTurk Panels on Your Own Requester Account

Studies with Panels  for just $0.15 - 0.75 / complete

Now you can run Mechanical Turk studies using your own Requester account and specify over two dozen demographic traits!.  The traits include gender, ethnicity, age, marital status and sexual orientation. But it does not stop there! The available options also include occupation, medical and health history, cell phone use and much more.

The cost ranges from $0.15 - $0.75 per completed assignment. For example, if you run a study with a panel of 100 White Males 40 and under with a cost of $0.42 / complete the TurkPrime Panel Fee is 0.42 * 100 = $42.00. The panel fee is determined by the incidence rate of your particular panel so that harder to reach demographics cost more...but are capped at a maximum of $0.75 per complete.

In addition, TurkPrime displays the feasibility of the study to run to completion. This is not a guarantee that the workers will take your study. since the MTurk workers, ultimately, decide whether they will accept and complete your study based on many factors including worker payment, requester rating on TurkOpticon and clarity of your study, among others. 

If you want to be certain your study will run to completion we recommend using the TurkPrime Lab Services of either Prime Panels or MTurk Panels where TurkPrime manages all user interaction, reaches out to workers to complete your study and guarantees the study will run to completion.

Studies with MTurk Panels will display the panel traits in the study dashboard along with a tag marking it as a panel study, as shown below.

Tuesday, August 23, 2016

Survey Groups Ensure Unique Workers Across Multiple Concurrent Studies

Problem: Ever want to run multiple simultaneous studies and needed to ensure that each worker only took a single study? There is no simple way to block workers who completed one study from accepting anopther study run at the same time....until now. 

(The TurkPrime exclude feature will exclude workers who completed one study from taking another subsequent study but not if both studies are run at the same time.)

Solution: The TurkPrime Survey Group (arriving August 25, 2016)  feature allows a user to assign multiple studies to a survey group. Then, any worker who accepts any study in that group will be disqualified from accepting any other study ion that group. Your studies will be guaranteed to have a mutually exclusive group of respondents.

The Survey Group option is also a broader exclusion than the standard Exclude Worker option: Survey Group excludes workers who even only accepted one study from taking the other studies in the group, while the standard Exclude Worker option only excludes workers who completed the study but will not exclude workers who accepted but did not complete the first study from taking the second study.

Tuesday, July 26, 2016

Google Forms Integration with TurkPrime

Google Forms can be used to deliver a study with TurkPrime in a similar manner to other survey platforms (like Qualtrics and SurveyMonkey).

In Google Forms set up your survey and then set up the secret completion code display:
1. Click on Settings (the gear icon)
2. Click on Presentation
2. Change the confirmation message to include "Your secret completion code is ABCDEF". (Of course, replace ABCDEF with your own code)

Then get the Google Form link 
1. Click on SEND
2. Click on the Chain / Link Icon
3. Click Copy (or just copy the URL)

On TurkPrime you can then select your panel if desired, add the  Google Forms survey  link URL and in the Tab (How workers are paid) enter the fixed secret completion code.  (e.g. ABCDEF)  

Tuesday, April 12, 2016

Demographic Consistency

Demographic Consistency Over Time on Mechanical Turk


Ever wonder if workers are being honest with you when they answer a survey? Or, if you specify that your study should be taken only by Women, whether some workers take the study even though they are not women?


Now, all studies that have the Demographic Question feature enabled, will receive 2 additional columns  in the CSV download for each Worker:
  1. gender
  2. gender consistency score
The gender column specifies the worker's gender based on the worker's responses to that question in previous surveys. Each worker who completes a study may have taken TurkPrime studies many times before. With each study where the Demographic Question is enabled, TurkPrime asks Workers to optionally share some demographic information like gender so they can qualify for HITs that require those demographics. (Through TurkPrime Panels Lab Service, researchers run many studies available only to workers who satisfy specific demographic requirements.)  It is the responses to these questions that is shared with you, the TurkPrime users.

The gender consistency score specifies the consistency with which the gender question was answered as specified. For example, if the GenderConsistency is 66%, that means the MTurk Worker responded that they were female only 66% of the time. If the field is blank, that means that TurkPrime has not received a response from that Worker when asked to identify their gender.

This may indicate an innocuous error on the Worker's part, multiple workers sharing an MTurk account, or negligence on the part of worker. In any event, that information is now shared with TurkPrime users.

Friday, February 19, 2016

New Safety Feature: Assignment Rejections are not Automated and Require Manual User Action

Many researchers set up their studies to use the TurkPrime AutoApprove feature so that they do not need to manually approve worker assignments based on the secret code that workers enter. On occasion, a researcher may set up his study incorrectly which results in many worker assignments getting automatically rejected. This was a significant cause for distress among the Mechanical Turk workers who received rejections for their work which then went unpaid and also lowered their MTurk approval rating.  This was also a sore point for TurkPrime researchers who had to deal with upset workers and correspond with them, and often reverse their rejections

To alleviate this issue, TurkPrime will give researchers greater control of the rejection process when AutoApprove is used: 

Only worker assignments that have the correct secret codes will be auto-approved while workers with invalid secret codes will require manual rejection. (If a rejection is made in error by a researcher, it may still be reversed.) This manual rejection is made using the same interface as is available for non-AutoApproved workers as shown below.

Researchers will see the secret codes the workers entered and whether they were correct. they will then have the option to approve, reject or leave their status undecided.

The manual rejection process must be completed within the Amazon Approval window (default of 7 days). Otherwise, Mechanical Turk will automatically approve the pending assignments that were not resolved buy that time.