In a recent research article (https://psyarxiv.com/jq589/), we reported on the size of the MTurk worker pool. Using observed participation rates from the TurkPrime database, we found that over each of the last three years there have been approximately eighty to eighty-five thousand US workers active on MTurk each year (Figure 1). What these numbers do not illustrate is how often new workers join the platform—a number that is important for researchers who want to sample naive participants.
Using data from TurkPrime, we were able to ascertain that in each month from January 2016 through April, 2019 there were between 15,000 and 30,000 US workers taking at least one HIT and approximately 4,600 new workers joining the pool each month. As shown in Figure 1, the number of new workers per month has fluctuated over time. Nevertheless, within each of the last three years more than half of MTurk workers were new to the platform (Figure 2).
Figure 1. The number of new US workers per month from January, 2016 - April, 2019. On average 4,683 new workers joined the pool each month.
Figure 2. The number of new and unique US workers taking a HIT posted through TurkPrime each year.
MTurk’s popularity has led to several questions about whether the participant pool is overused. One key to answering this question is ascertaining how many workers are active on MTurk per year. In addition, another key is finding out how often new workers join the platform.
The numbers from our database reveal that MTurk has more workers than previously estimated and that there is a significant number of new workers joining the pool on a regular basis. Even though more research needs to be conducted, we believe these numbers offer a reason for researchers to be optimistic about the possibility of sampling participants from Mechanical Turk.
In addition to discussing how often new workers join MTurk, our article addresses several issues relevant for sampling on Mechanical Turk. The full preprint of the article is available here.
Difallah, D., Filatova, E., & Ipeirotis, P. (2018, February). Demographics and dynamics of Mechanical Turk workers. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (pp. 135-143).
Stewart, N., Ungemach, C., Harris, A. J., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making, 10, 479-491.