Link To TurkPrime

Effective Mechanical Turk: The TurkPrime Blog

How to run successful experiments and get the most out of Amazon's Mechanical Turk

Tuesday, August 23, 2016

Survey Groups Ensure Unique Workers Across Multiple Concurrent Studies

Problem: Ever want to run multiple simultaneous studies and needed to ensure that each worker only took a single study? There is no simple way to block workers who completed one study from accepting anopther study run at the same time....until now. 

(The TurkPrime exclude feature will exclude workers who completed one study from taking another subsequent study but not if both studies are run at the same time.)

Solution: The TurkPrime Survey Group (arriving August 25, 2016)  feature allows a user to assign multiple studies to a survey group. Then, any worker who accepts any study in that group will be disqualified from accepting any other study ion that group. Your studies will be guaranteed to have a mutually exclusive group of respondents.

Tuesday, July 26, 2016

Google Forms Integration with TurkPrime

Google Forms can be used to deliver a study with TurkPrime in a similar manner to other survey platforms (like Qualtrics and SurveyMonkey).

In Google Forms set up your survey and then set up the secret completion code display:
1. Click on Settings (the gear icon)
2. Click on Presentation
2. Change the confirmation message to include "Your secret completion code is ABCDEF". (Of course, replace ABCDEF with your own code)

Then get the Google Form link 
1. Click on SEND
2. Click on the Chain / Link Icon
3. Click Copy (or just copy the URL)

On TurkPrime you can then select your panel if desired, add the  Google Forms survey  link URL and in the Tab (How workers are paid) enter the fixed secret completion code.  (e.g. ABCDEF)  

Tuesday, April 12, 2016

Demographic Consistency

Demographic Consistency Over Time on Mechanical Turk

Problem:

Ever wonder if workers are being honest with you when they answer a survey? Or, if you specify that your study should be taken only by Women, whether some workers take the study even though they are not women?



Solution:


Now, all studies that have the Demographic Question feature enabled, will receive 2 additional columns  in the CSV download for each Worker:
  1. gender
  2. gender consistency score
The gender column specifies the worker's gender based on the worker's responses to that question in previous surveys. Each worker who completes a study may have taken TurkPrime studies many times before. With each study where the Demographic Question is enabled, TurkPrime asks Workers to optionally share some demographic information like gender so they can qualify for HITs that require those demographics. (Through TurkPrime Panels Lab Service, researchers run many studies available only to workers who satisfy specific demographic requirements.)  It is the responses to these questions that is shared with you, the TurkPrime users.

The gender consistency score specifies the consistency with which the gender question was answered as specified. For example, if the GenderConsistency is 66%, that means the MTurk Worker responded that they were female only 66% of the time. If the field is blank, that means that TurkPrime has not received a response from that Worker when asked to identify their gender.

This may indicate an innocuous error on the Worker's part, multiple workers sharing an MTurk account, or negligence on the part of worker. In any event, that information is now shared with TurkPrime users.

Friday, February 19, 2016

New Safety Feature: Assignment Rejections are not Automated and Require Manual User Action

Many researchers set up their studies to use the TurkPrime AutoApprove feature so that they do not need to manually approve worker assignments based on the secret code that workers enter. On occasion, a researcher may set up his study incorrectly which results in many worker assignments getting automatically rejected. This was a significant cause for distress among the Mechanical Turk workers who received rejections for their work which then went unpaid and also lowered their MTurk approval rating.  This was also a sore point for TurkPrime researchers who had to deal with upset workers and correspond with them, and often reverse their rejections

To alleviate this issue, TurkPrime will give researchers greater control of the rejection process when AutoApprove is used: 

Only worker assignments that have the correct secret codes will be auto-approved while workers with invalid secret codes will require manual rejection. (If a rejection is made in error by a researcher, it may still be reversed.) This manual rejection is made using the same interface as is available for non-AutoApproved workers as shown below.

Researchers will see the secret codes the workers entered and whether they were correct. they will then have the option to approve, reject or leave their status undecided.

The manual rejection process must be completed within the Amazon Approval window (default of 7 days). Otherwise, Mechanical Turk will automatically approve the pending assignments that were not resolved buy that time.

Monday, January 18, 2016

Anonymize Mechanical Turk Worker ID's

We recently launched a ground-breaking feature that helps protect Mechanical Turk worker identities. It has been reported in the literature that Mechanical Turk WorkerId's can be used to identify the worker. This is because Amazon uses the same value for both the WorklerId on Mechanical and elsewhere on Amazon properties like Amazon.com product reviews.

The Anonymize Worker ID's feature anonymizes  Mechanical Turk Worker Id's, as discussed below.
When you enable the Anonymize Worker Ids feature, all WorkerId's that appear in your study's downloadable CSV file will appear encrypted. For example, if the WorkerId is "ABCDEFGHJKL", it will instead appear as "TP_1UPKSI2WHSJ4". This encrypted TurkPrime Worker ID begins with letters TP and can be used in all operations on TurkPrime where an Amazon WorkerId can be used:
  • Exclude and Include Feature supports both Amazon Worker Id's and the encrypted TurkPrime WorkerId's.
  • Reusable Worker Groups can specify both Amazon Worker Id's and the encrypted TurkPrime WorkerId's.
  • Bonus Workers can specify both Amazon Worker Id's and the encrypted TurkPrime WorkerId's.


Thursday, January 7, 2016

Block Multiple Responses from the Same IP Address

Block Duplicate IP Addresses

Our most popular feature request has been to allow the blocking of multiple responses from the same IP address.



MTurk workers with different MTurk WorkerId's may share an IP address. This might be for a number of reasons:

  1. different workers may be using the same Internet connection. For example, they may both be in the same apartment, school or Starbucks and access MTurk over the same WiFi connection. 

    Some researchers will want to avoid this so that their sample is more diverse and representative of the population they are targeting.
  2. One worker takes HITs using different MTurk accounts. Nearly all researchers will want to avoid this.

When this option is selected a worker who has the same IP address as a previous workers who accepted the study will see a message that he/she is ineligible for the HIT. 

If the worker, nevertheless, accept the HIT, they will not see the hyperlink to the study. Instead they will see a message to return the HIT because another worker at that IP address has taken the HIT

Tuesday, January 5, 2016

HyperBatch - Run Batched HITs at Hyper Speeds

HyperBatch Feature 

You can now run your Amazon Mechanical Turk studies at Hyper speeds. We worked hard to make the experience identical to our classic MicroBatch option so that you simply launch your study and TurkPrime does all the heavy lifting for you

How it works

When you launch a HyperBatch study with 100 participants, for example, your study appears on TurkPrime exactly as a standard study. However, the study, in fact, is broken down into multiple HITs with 9 or less participants. When you pause or change a study, all of the HITs associated with your study are paused or updated, respectively.

Workers who completed one HIT in the study are disabled from taking additional HITs. TurkPrime automatically assigns the Worker a Quailification that prevents them from taking your study twice.

All existing features such as secret code, include / exclude / groups etc work exactly as you expect them to.