Link To TurkPrime

How to run successful experiments and get the most out of Amazon's Mechanical Turk

Monday, June 8, 2015

Bounce and Completion Rate

In the latest release of TurkPrime.com we added many new features and fixes among them 2 additional metrics for every survey:

  • Bounce Rate
  • Completion Rate

The Bounce Rate is the percentage of Amazon workers who previewed your survey but decided not to accept it. An open question is whether and how this self-selection of participants affects the representativeness of the participant pool. In addition, a high bounce rate may be an indicator that there is something wrong with your survey



We have already discussed the importance of the Completion Rate:
Completion rate is an important indicator of data quality. A low completion rate indicates that there is a selection bias which may be influencing the representativeness of the results. A very high dropout rate may also mean that there is something wrong with the study. It is typically good practice to report completion rate in the method/results section of a paper. Indeed, some editors require authors to use the CHERRIES checklist for survey research (Eysenbach, 2004), which asks about a study’s completion rate.




Under the Hood

Bounce Rate is 100% -  [Accepted Assignments] / [Previewed Assignments]. Previewed Assignments is set to be the total of unique IP addresses that previewed this assignment. Since nearly all workers have unique IP addresses this is a reasonable estimate of the total number of previews.