TurkPrime Blog

TurkPrime Blog

How to Minimize Dropout Rate on Mechanical Turk

Posted by Leib Litman on Mar 8, 2015 11:04:00 AM

Brief overview

It is generally thought that pay rate does not affect data quality on Mechanical Turk. For example (Buhrmester, Kwang, & Gosling, 2011) showed that whether Workers are paid 5 cents or one dollar for a survey study, the internal reliability of the surveys does not change. They did show however that fewer Workers will take the surveys that pay less. We recently replicated these findings for both US and India-based Workers (Litman et al, 2014). Here we show that low pay rates have two effects on Workers: 1) Workers are more likely to return a HIT before completing it and 2) Workers spend less time answering questions. We examined 30 MTurk studies that were run over the last 6 months. The findings show that 36% of the dropout rate variance is explained by the length and pay rate of a survey. These results show that low pay rates do more than just slow down the rate at which Workers take HITs. Low pay rates may also negatively impact the representativeness of data due to high participant dropout, and they may also decrease how much attention participants pay to each question. Based on these findings we recommend against low paying HIT We also recommend against overly long surveys, unless Workers are appropriately compensated. To minimize dropout and to maximize time on task, compensation for HITs should not be below $4 per hour and should be closer to $6 per hour or more.

Read More

Topics: amazon mechanical turk, dropout rate, mechanical turk

Determining Completion Rate and Dropout Rate on Mechanical Turk

Posted by Leib Litman on Mar 6, 2015 4:43:00 PM

What is the completion rate and dropout rate?

Dropout rate is defined as the percentage of participants who start taking a study but do not complete it. Dropout rate is sometimes referred to as attrition rate, and is the opposite of completion rate (dropout rate = 100 – completion rate). On MTurk, completion rate is defined as the number of Workers who submit a HIT divided by the number of Workers who accept the HIT. Note that, for the definition of completion rate used here, Rejected Workers are counted as completes.

Read More

Topics: amazon mechanical turk, completion rate, dropout rate, mechanical turk