Amazon Mechanical Turk: The Human Sciences’ Labor Problem

Abstract

This article investigates the rise of Amazon Mechanical Turk (MTurk), Amazon Web Services, Inc.’s crowdsourcing labor platform, in social science research since 2005. A new “digital sweatshop,” the platform hired online workers to do precarious, extremely low-wage tasks to support artificial intelligence (AI) and survey research, while effectively stripping workers of all protections except those they built for themselves. Bringing together labor history and the history of science through an investigation of MTurk, this article intervenes in the historiography bidirectionally. Interpreting research participation as work, it argues, first, that the history of knowledge production is a labor history. To understand the ensuing conflict between workers and researchers on the MTurk platform, one must understand its labor context. Their struggle lay at the intersection between social science's notion of ideal research subjects and the concerns, interests, and vulnerabilities of crowdsourced participants as a class of exploited and unprotected workers. This article asks, second, how the labor conditions of research subjects impacted the knowledge produced from them. As in other industries, dialectics of labor exploitation shaped (and spoiled) the knowledge products that digital piecework yielded. The “labor” being deskilled, in this case, was being human.

“Bot Panic”

Academic social science and the “gig economy” collided in 2018.¹ Max Hui Bai, a psychology graduate student from the University of Minnesota, noticed abnormalities in his data. He asked online researcher forums, “Have [sic] anyone used Mturk in the last few weeks and notice any quality drop?” MTurk, short for “Mechanical Turk,” was a crowdsourcing labor platform developed by the online bookseller Amazon, which over the past decade has become newly popular among social scientists seeking online research participants. Through Facebook groups and word of mouth, Hui learned that over ninety studies worldwide identified similar data abnormalities, spotting duplicated GPS locations (e.g., the same string of numbers, “88639831,” appearing in multiple respondents’ coordinates) and nonsense answers (e.g., respondents indiscriminately replying “nice” or “good” to all open-ended questions). Burgeoning anxiety in social science about the reliability of data obtained from new, cheap, and convenient online platforms exploded into a “bot panic.” Researchers wondered whether the data corruption was new, or statistically significant, or a threat that jeopardized thousands of already published studies conducted using these methods.²

MTurk aspired to provide any individual or organization with online access to “a global, on-demand, 24 × 7 workforce.”³ Opening publicly in 2005, it was an early leader in what became a multi-billion-dollar online outsourcing industry—the whole industry, the World Bank projected, earning $2 billion in 2013, and $15–25 billion in 2020.⁴ Competitors like CrowdFlower, Clickworker, and Toluna joined MTurk's niche of “crowdsourced,” “piecemeal,” “micro,” or “gig” labor.⁵ Academic researchers—particularly social scientists—swarmed MTurk for research, conducting up to fifty thousand studies there annually and publishing thousands.⁶ MTurk was the new “digital sweatshop,” hiring online workers to do precarious, extremely low-wage tasks while effectively stripping them of all protections except those they built for themselves.⁷

Bringing together labor history and the history of science through an investigation of MTurk, this project intervenes in the historiography bidirectionally. It argues, first, that the history of knowledge production is a labor history. An extensive discourse in the history of science has examined what happened when scientists grew anxious about the validity of their data, whether in physics, biology, medicine, climate science, or psychology.⁸ When a device or procedure produced signal, a meditation on the reliability of that signal ensued. Yet to understand the validity of MTurk-produced research data, one must understand its labor context. The methodological tumult triggered by Hui's “bot panic” lay at the intersection between social science's notion of ideal research subjects and the concerns, interests, and vulnerabilities of crowdsourced participants as a class of exploited and unprotected workers. Interpreting research participation as work, I trace labor extraction on MTurk as an inheritance of the long history of labor deskilling, demonstrating how it implicated social scientists as buyers of labor, paying (cheaply) to harvest crowdsourced data. Understanding research conducted on MTurk through its exploitative labor relations sheds light on researchers and scientists as agents in a capitalist order, and on MTurk as a platform providing access to a growing class of deskilled workers furnishing raw materials for behavioral data extraction.

This article asks, second, how the labor conditions of research subjects impacted the knowledge produced from them. Just as conditions of production impacted the quality and composition of products in the industrial age (when goods were material products), they likewise impact the quality and composition of products in the so-called digital age (when goods include knowledge products). Although some may label the United States a postindustrial society, by pointing to flagging labor union membership and manufacturing that relocated offshore, at the cutting edge of modern-day labor, industrial logics of how employers exploited and how workers resisted have not only persisted but even intensified. Dialectics of labor exploitation shaped (and spoiled) the knowledge products that digital piecework yielded. The “labor” being deskilled, in this case, was being human.

(Human) Raw Materials

Up to now, whenever scientific research has concerned humans, it has inevitably required human raw materials: cadavers for dissection, skeletons for anatomy teaching, or live human subjects for experiments. To produce knowledge, the human sciences have demanded a supply of humans.⁹ In medieval and early Renaissance Europe, for instance, anatomists and their students bought, stole (robbed graves), or bribed (hangmen or gravediggers) to acquire cadavers and skeletons for research, medical training, art, or display in natural history cabinets.¹⁰ Laws like the Anatomy Act of 1832 in England expanded anatomists’ pool, allowing them to procure cadavers not only from prosecuted and hanged individuals but also from those who died destitute in workhouses or were otherwise unclaimed after death.¹¹ The traffic in dead human bodies for science proliferated to the United States’ burgeoning medical schools and, in some cases, became a cornerstone of eugenics.¹² By the 1950s, long-standing practices of body-snatching and looting gave way to formal body donation programs for medical education and research.¹³

But human raw materials were useful—even necessary—for science both dead and alive. A similarly ethically fraught history followed human experimentation on living human subjects, from earlier experiments on adults and children to present-day, risky Phase I pharmaceutical trials on healthy, paid “volunteers.” Scientists widely sourced markets of human raw materials to fuel research, but their human subjects were unevenly distributed in race, class, and social station, concentrated among the most vulnerable.¹⁴

As social science professionalized in the first half of the twentieth century, practitioners faced similar shortages in human research participants. Throughout the 1940s and 1950s, social psychology pooled diverse populations, drawing subjects from radio listeners, voters, soldiers, civilians, residents of housing projects, and industrial factory workers, surveying them in their “natural habitats.” Starting in the late 1950s, however, these studies relocated to the laboratory, and undergraduate students, the researcher's “familiar, captive, and largely friendly data base,” became the primary research population. The transition was thorough. By 1980, over three-quarters of studies published in top journals used undergraduates as their sole subjects.¹⁵

This “undergraduate” solution created its own problems, as anxiety about the status of experimenters and their subjects plagued psychological laboratory research beginning in the late 1960s.¹⁶ Social scientists, for decades, critiqued their field's heavy reliance on undergraduates.¹⁷ Many argued that undergraduates were too narrow a pool for deriving generalized claims about human behavior—unrepresentative, as they were, of humans along numerous axes, including likelihood for social compliance, attitude changeability, behavioral consistency, and introspection. Further research argued that using participants only from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies skewed several behavioral metrics, including visual perception, fairness, cooperation, spatial reasoning, and self-concept. But undergraduates proved a pool too convenient and cheap not to use. In the face of critiques, researchers rarely justified their use of undergraduates, and these studies persisted, resulting in the average American college student being over four thousand times more likely than someone outside the West to be a social science research participant.¹⁸

In the new millennium, academics flocked to MTurk, which appeared to offer an alternative, abundant human subjects pool. The academic database JSTOR heralded in 2018 that MTurk gifted social science its next methodological revolution, providing a “sudden availability of a conscientious, diverse, non-student source of survey respondents,” which “has led to something of a golden age in survey research” and “may change the practice of social science” altogether.¹⁹ As one behavioral scientist explained, when transitioning from graduate school to a job at Yahoo Research, he realized, “we had no undergraduates and no classrooms.” How were his experiments to proceed? In 2008, he began using MTurk workers, or “Turkers,” as research subjects, thereby continuing his scholarship even after moving from university to corporation.²⁰ This new pool appeared promising, as Turkers were demographically more diverse than both standard online research pools and traditional American college students, and MTurk data appeared to be at least as reliable as data obtained through traditional methods.²¹ Cheap survey research was a boon to other sectors, too, like Pete Buttigieg's presidential campaign, which reportedly spent $20,000 on MTurk survey research, polling Turkers as approximations for American voters.²²

But this solution soon posed new problems à la Hui's “bot panic.” Clamor about MTurk data integrity sounded. How bad was the problem? This outcry spawned a cottage industry of MTurk metastudies—research about doing research on MTurk. Dubious data already existed on professional online survey platforms like Survey Sampling International, but MTurk research magnified the problem. Some argued that the “bot epidemic” was overblown, as only approximately 20 percent of Turkers provided false data (e.g., by circumventing location tagging requirements, or by providing multiple submissions using blacklisted, duplicated, or missing IP addresses). An additional 5 to 7 percent of Turkers engaged in “trolling” or “satisficing”—the former connoting various common, abusive online behaviors, and the latter a portmanteau of “satisfy” and “sufficing,” referring to work that provided the minimum response required to pass the lowest bar of acceptability. “Satisficing” was not unlike industrial America's “soldiering,” employers’ word for workers’ strategies of artificially restricting output or doing the least amount of work that avoided punishment. The combined impact of low-quality responses from “trolling” and “satisficing” attenuated researchers’ treatment effects by a sizable but not alarming 10 percent.²³ Yet other researchers argued that MTurk's corruption of a quarter (25–27 percent) of research data was significant and frustrating and warranted its discontinuation.²⁴ To understand researchers’ conflicts with their new digital workforce, one must return to the labor context.

The Hoax: Artificial Artificial Intelligence

Amazon originally designed MTurk for internal use. Hosting millions of webpages, the company needed to remove duplicates. Combing through webpages was a task that algorithms performed imperfectly but that—Amazon discovered—humans performed easily when paid a few cents per page. Enjoying its creation, Amazon began allowing other companies to use its site in 2005, while it collected a cut of profits.²⁵ Calling itself “a crowdsourcing marketplace,” MTurk aimed to “harness the collective intelligence, skills, and insights” of crowdsourced workers for artificial intelligence (AI) training, market research, or anything its users could envision and pay for.²⁶ It called its workers “Turkers,” its jobs “Human Intelligence Tasks” (HITs) or “microtasks” or simply “tasks,” and its job-posting employers “requesters.” The pay was low as 1 cent per task, not including a 20 percent site-use fee paid to Amazon, which had doubled from 10 percent in 2015. Researchers estimated that MTurk facilitated $100 million in tasks each year, garnering tens of millions in revenue for Amazon annually.²⁷

MTurk drew its name from the famous eighteenth-century Mechanical Turk, a chess-playing automaton dressed as an Orientalized Turk, invented by Johann Wolfgang Ritter von Kempelen for the court of the Austrian empress Maria Theresa. The device toured the globe, defeating challengers like Benjamin Franklin, Frederick the Great, and Charles Babbage from 1770 to 1850. By design, it was a hoax. An illustration from a 1789 pamphlet (fig. 1) showed that sitting inside the alleged automaton was an unseen human chess master, manipulating its moves like a puppeteer from within. The eighteenth-century Mechanical Turk captured intrigue because it fed burgeoning anxieties about industrialization's runaway mechanical creations and long-standing Orientalist fascinations and fears about the liminal, disciplined, productive, and docile body of Muslim subjects.²⁸

Like its historical precursor, Amazon's MTurk was designed as a human-labor-concealing trick for the modern automaton, AI. The construction of machine learning technologies required human labor because it needed to be trained on existing data sets to learn patterns. Data had to first be labeled and sorted by humans before it was fed to machines. Before AI could identify pictures of apples, a person had to do so first.²⁹ MTurk sold labor in units called Human Intelligence Tasks (HITs), which juxtaposed crowdsourced laborers’ human intelligence with their artificial intelligence counterparts. Thus, MTurk supplied, at scale, the human crutch to adolescent AI. In the words of its patent application, MTurk integrated intelligence—both human and artificial—into a “hybrid machine/human computing arrangement.” Like a massive collective cyborg, composed of any willing worker on the internet, MTurk coalesced its computer and human elements into “one embodiment,” or so its architects hoped.³⁰

Like the human chess master manipulating pawns from within the belly of the eighteenth-century Mechanical Turk, anonymous online laborers animated AI as the ghost in the machine, powering “ghost work” in a “ghost economy,” where 8 percent of Americans have labored.³¹ Adam Selipsky, then–vice president of Amazon's product management and developer relations, explained, “Usually people get help from computers to do tasks. In this case, . . . computers get help from people to do tasks,” “turning the traditional computing paradigm on its head.” According to CEO Jeff Bezos, MTurk specialized in tasks “easy for a human but extraordinarily hard for a computer.” A joker, Bezos called this class of AI's prerequisite human labor “artificial artificial intelligence.”³² The “artificial artificial” in this case was human.

Companies used MTurk in the idiom of its forebear—through “invisibility by design”—to conceal human labor within an AI artifice. Automated text transcription companies, for instance, found that it was cheaper to pay for four deskilled human intelligences to check their transcriptions than to construct an adequately sophisticated artificial intelligence. They profited from their concealment systems by paying Turkers as little as 19 cents, while charging clients 42 to 75 cents, per minute of audio.³³ The more successfully start-ups hid “human computation” in their technologies, the higher their valuations as “software” companies.³⁴

Most journalists and academics writing about MTurk interpreted its namesake, the eighteenth-century Mechanical Turk, as evidence of the platform's insidious labor-concealment. The ever-tricky Bezos hid the human element!³⁵ Yet this compelling perspective omitted the most interesting irony in the platform's cheeky name. In electing the historical Mechanical Turk as namesake, MTurk did not conceal this sleight of hand; rather, it alerted audiences to the ruse, offering self-reflexive commentary on AI's disguising of its human labors. A hoax that announced itself as such, MTurk encapsulated the digital economy's need for a shadow labor force and its cavalier incorporation of duplicitous mechanisms into its technologies. Pointing out its own machinations through its name, MTurk operated like Poe's purloined letter—hidden effectively by being left in plain sight.

MTurk as Digital Piecework

At the cutting edge of the so-called digital age, industrial logics have not only persisted but intensified.³⁶ MTurk resembled “piecework,” or “piecemeal work,” the historical category of labor compensated based on quantity of “pieces” completed, rather than time or skill.³⁷ Breaking skilled labor into smaller, more easily repeatable increments had been a hallmark of efficiency practices like Taylorism, which transferred discretion, authority, and knowledge from workers to managers. MTurk continued this miniaturization of the units through which human labor, time, and energy were sold online.³⁸ In digital piecework, however, the kind and rate of “deskilling” transformed. The unit of work shrank from “jobs,” “services,” or even “projects” to “microtasks.” The micro in MTurk's “microtasks,” like the micro in Frank and Lillian Gilbreth's “micro-motion study,” signaled this intensified task-decomposition.³⁹ MTurk's creators fantasized that their invention may one day become “a computer system [that automatically] decomposes a task . . . into subtasks for human performance.” At this telos of deskilling, even the deskilling process itself will be automated.⁴⁰

The labor being deskilled in this case was being human. Pushing deskilling to its extreme, logical conclusion, MTurk made the base qualification for many of its tasks simply possessing human cognition. The bulk of its tasks were designed for “any person” to complete with little conscious thought or training, such as recognizing patterns, interpreting images, catching innuendos, identifying cultural references, and deciphering context. Deskilled labor's race to the bottom resulted in this form of work for which all humans purportedly qualified by virtue of being human.

Through MTurk's conduit, fields outside AI discovered the profitability of common human cognition. Dozens of studies in medical research, for instance, demonstrated how Turkers, as laypersons, provided time-, cost-, and labor-saving alternatives to expert medical judgment. One study on the surgical procedure cricothyrotomy found that the averaged assessment of a group of thirty Turkers was comparable in accuracy to that of three surgical experts. Whereas trained surgeons took sixty days to evaluate the procedures, Turkers completed the same assessments in ten hours, earning 50 cents per survey.⁴¹ Other studies found comparable results, indicating that “surgery-naive,” “ophthalmologically naive,” and medicine-naive Turkers scored as well as doctors in assessing procedures like urinary bladder closures, robotic partial nephrectomy (RPN), digital retinal imaging, and drug indication annotations.⁴² Averaged crowdsourced intelligence posed an ironic refrain to the classic tagline “4 out of 5 doctors recommend this product” by posing this question: If the averaged judgment of 9 out of 10 laypersons was as good as that of 1 trained pathologist, was that “good enough” that we could dispense with the expert?⁴³ MTurk's wisdom of the crowd gave medicine exciting opportunities to “scale” innovations and profit. Even hallowed medical expertise could be deskilled through digital piecework.

This contrast between human and artificial intelligences became ubiquitous, such as in law, which deployed these logics in their inverse. When a crime-forecasting algorithm was “no better” than Turkers’ lay judgment at predicting recidivism, researchers argued that the court should not use it to make “life-altering decisions.”⁴⁴ Turkers’ common human cognition became a ruler against which to measure expertise.

Employers reaped the benefits of microtask labor. It enabled them to experiment with new areas of business, contracting workers at the smallest unit of obligation to fulfill emergent needs, without taking on the liabilities and financial burdens of a full employer-employee covenant. Like social media companies, which evaded Federal Communications Commission regulation and content moderation duties by classifying themselves not as publishers but merely as “platforms,” MTurk abdicated responsibility for employer-employee relations, worker protection, pay, and fairness by claiming to be not an “employer” but merely a “marketplace” for labor transactions. Amazon insisted that requesters and workers were bound by their own voluntaristic arrangements; it merely held space for their initial meeting. Abstaining from Turker-Requester disputes over unpaid work, Amazon left workers no recourse for unfair, abusive, or arbitrary treatment.

Requesters’ power over Turkers—their authority to reject or accept work at will—was near total, tempered only by the latter's crowd-sourced solutions to hold requesters accountable. Requesters could “reject” unsatisfactory work, and workers could “return” tasks they declined to perform, but workers were paid for neither “rejects” nor “returns.” Requesters admitted to arbitrarily rejecting a certain percentage of all work to cover Amazon's site-use fees, effectively cheating workers out of pay, since requesters could keep (and not pay for) work they rejected.⁴⁵

This microtask employment strategy was a cousin to the widespread, dodgy Silicon Valley cost-saving strategy of “permatemping”—or permanent temping—in which companies permanently retained contract workers purportedly filling in for temporary roles, forming a persistent underclass of workers who were ineligible for employee benefits but served alongside full-time workers in identical or complementary functions.⁴⁶ Like others precariously employed in the gig economy, like Uber or TaskRabbit workers, Turkers did not qualify for protection under the Fair Labor Standards Act, leaving them out of provisions such as minimum wage, overtime, and workplace benefits, although the gig worker reclassification rule proposed by the Biden administration in October 2022 could alter this terrain. The legal status of the “crowd” remained unclear.⁴⁷ Critics labeled digital piecework “the biggest paradigm shift in innovation since the Industrial Revolution,” responsible for “wiping away a century of labor struggles.”⁴⁸ Being free agents in this free market meant that Turkers were free from labor rights and protections, too.

For all this, Turkers were paid basement wages. One user reported that short transcriptions earned $2.30, surveys $6.85, and Google search descriptions $15.15 per hour.⁴⁹ But a study of 2,676 Turkers from 2017 performing 3.8 million tasks showed that the median wage was $2 per hour. Only 4 percent of the study's participants earned more than $7.25 per hour, the federal minimum wage.⁵⁰ MTurk's defenders argued that piecework—in its historic and digital forms—intended to differentially reward workers based on output, so experienced workers should earn several times more than novices for their efficiency.⁵¹ But even muckraking reportage demonstrated digital piecework's pervasiveness in all knowledge industries. The very media organizations that printed MTurk exposés, like the New York Times, ProPublica, and Pew Research Center, used MTurk for other projects. MTurk embroiled even its critics in its labor extraction.⁵²

Companies excused MTurk's low wages, claiming that Americans “do this for fun” and that “poor people in the Third World” consider this “a good salary.”⁵³ But defenders and critics agreed that “drone image tagging” could never amount to a “career.”⁵⁴ Yet these common beliefs conflicted with the reality of Turker demographics. The majority of Turkers resided in the United States, with India a distant second. An increasing number of Turkers—25 percent in 2016—depended on MTurk for all or most of their income.⁵⁵ MTurk echoed broader trends as 44 percent of US workers in 2011 considered themselves “free agents,” generating $141 billion in the US economy annually in pursuit of full-time or supplementary income (e.g., among retirees).⁵⁶ Almost one in four American adults worked in the “platform economy” in 2015,⁵⁷ and the same number resorted to platforms due to lack of other opportunities in their region.⁵⁸ For many in the United States, gig work's microtasks were the best professions the modern economy afforded.

Like its forerunner, online piecework worsened intersectional oppression. During industrial piecework's surge in the early twentieth century, employers used incentive-pay schemes to induce high output from immigrant women, already marginalized from American society and dependent on this supplemental income. Although Turkers were more diverse than undergraduates, digital pieceworkers resembled their historic predecessors in concentrating among marginalized populations. Those who relied on MTurk for all or most of their income tended to be Hispanic, younger, and less educated and live in lower-income housing.⁵⁹

Resistance Online

Turkers were spatially distributed like those who engaged in industrial “homework” (industrial piecework done at home) in the late nineteenth and early twentieth centuries. Physically scattered, “homeworkers,” unlike industrial workers who congregated physically on shop floors, had few opportunities to collectively organize. Some feared that crowdworkers, composed of digitally distributed online strangers, suffered from similar barriers to collective action that “homeworkers” faced.⁶⁰ But crowdsourced labor extraction provoked crowdsourced labor resistance. Absent protections from governments or employers, Turkers collectively became their own guarantors-of-last-resort. Much of the behaviors that academics found frustrating in their MTurk research subjects arose out of workers’ strategies for self-preservation in the online gig economy.

Piecework effectively passed costs for training, professionalizing, acquiring tasks, and taking breaks from employers to workers. To make subsistence wages, Turkers, like contractors, spent enormous, uncompensated time prowling the site for their next piecemeal task. But unlike traditional contractors, Turkers’ contracts came in increments of a few cents each, making them appear more disposable to employers, while also requiring them to secure many more units of “microcontracting” work to string together full-time employment. Turkers were paid only for their productivity, never for breaks between periods of productivity when they rested or looked for more tasks. By contrast, breaks, rest, reprieve, and downtime during work among hourly and salaried workers were historically paid for and even engineered by employers. At the height of Progressive Era zeal for reform, scientists argued that introducing rest periods was the most cost-effective way to improve worker efficiency.⁶¹ Within the gig economy, however, “downtime” became an externality shouldered by workers themselves.

As if a perverse reply to last century's time studies, controlling one own's time became digital piecework's benefit and curse.⁶² Turkers could complete tasks at their self-appointed schedule, unrestricted by the traditional workday's parameters. Yet workers reported that the platforms that granted them the most time flexibility also made them feel most burdened, as they spent their uncompensated time creating for themselves the structures that their employers omitted, developing informal tools, practices, and communities necessary to complete their jobs. Among crowdsourcing platforms, workers rated MTurk as ostensibly “freest” and most time-constraining, as they needed to remain constantly “on call” to make a consistent income.⁶³ This “invisible labor of finding tasks” bred constant, hounding anxiety, as workers felt they could never leave the computer lest they missed the chance to seize the latest marginally higher-paying microtask.⁶⁴ Work advertised to give them control over their time morphed into work that required them to give up all of their time, to be ready to work at a moment's notice.

Because piecework paid workers per unit worked, it incentivized them to work as much as possible, often beyond their physical limits. Historically, nondigital piecework augmented musculoskeletal disorders (MSDs) and physical suffering among pieceworkers.⁶⁵ Amazon's warehouses have likewise received citations from the Occupational Safety and Health Administration (OSHA) at federal and state levels for creating ergonomically hazardous workplaces through notoriously intrusive productivity monitoring practices. Moving at breakneck speeds, workers refrained from taking bathroom breaks or shaking out their hands, increasing their likelihood of developing MSDs.⁶⁶ Piecework's high physical demands, poor working conditions, low worker control over jobs, and little supervisor support, along with pieceworkers’ typically low socioeconomic status, yielded adverse health consequences.⁶⁷

Like traditional pieceworkers, online pieceworkers reported elevated levels of repetitive strain injuries, arising from the pressure to chase extreme productivity to eke out additional income. Turker Kristy Milland set an alarm on her computer to notify her when requesters posted new lucrative tasks paying over 25 cents. After twelve years, she left MTurk with a ganglion cyst on her wrist the size of a marble. Without paid sick leave and unable to cover postoperative surgery costs, she treated her “Bible bumps,” as they were called, with the folk remedy of smashing them with a heavy book. That treatment, however, radiated the pain up her forearm and elbow. She left with mental scars, too, from sorting graphic images, like ISIS videos of “wicker basket[s] full of human heads,” echoing the psychological toll experienced across the online content moderation industry.⁶⁸ When companies did not pay for downtime, workers elided over their need for breaks to pursue the next task.

MTurk's free-for-all ethos empowered not only corporations and academics but also Turkers to declare new terms for workplace engagement, devising their own ways to combat workplace abuse. Because anyone with an Amazon account could post MTurk tasks, privacy concerns abounded. Some, for instance, asked Turkers to transfer strangers’ personal data (e.g., credit card information or disability status paperwork) from photos onto spreadsheets without explanations of the photos’ origins or the task's purposes. Others appeared odd or “creepy,” asking Turkers to film themselves doing niche activities, record their video or audio, disclose their personally identifiable information, or provide scans of their health insurance card. Turkers suffered stalking, spamming, phishing, and scamming on MTurk with virtually no recourse. Reporting false personally identifiable information could lead to their work being rejected or the worker becoming blacklisted on the site. If Turkers aborted a task for any reason, including privacy concerns, suspicious requests, unclear instructions, or technical malfunction, their “completion rate” decreased, adversely impacting their ability to secure future tasks. The allowable margins for incompletion were narrow, as many requesters, including academics, required Turkers to possess above 95 percent ratings to qualify for their tasks.⁶⁹

With time, Turkers acquired tacit knowledge about screening suspicious requests. For example, most avoided the “foot fetish guy on MTurk.” As one worker explained, “I have like 40% of requesters filtered out, because I know better now. But when you start you just try everything. and that's what screws you over.” Turkers felt safer disclosing to seemingly trustworthy requesters like academics, but ultimately income trumped other needs. They reported overriding privacy concerns and discomfort to proceed with dubious tasks for money.⁷⁰

Turkers also devised collective solutions through online forums, such as Turker Nation (moderated by Kristy Milland with the Bible bumps), mturkgrind, Turkopticon, MTurk Crowd, and an MTurk Reddit. They shared homegrown strategies for outsmarting the platform, such as using multiple Amazon accounts, distraction-elimination systems, alerts for new tasks, and bots to auto-complete surveys.⁷¹ Commentators saw the browser extension Turkopticon, which allowed Turkers to rate requesters on communicativity, generosity, promptness, and fairness, as the most promising instance of Turker resistance. Built by socially engaged academics, this tool attempted to correct MTurk's information asymmetries, wherein requesters could see and filter Turkers by ratings, but Turkers could not rate or filter abusive or “shady” requesters. Its cocreators, Lilly Irani and Six Silberman, began Turkopticon in 2008 as a graduate school class project, naming it after the Panopticon, the all-seeing prison that compelled its inhabitants (prisoners) to self-discipline their behavior, designed (but not built) by Jeremy Bentham and made famous by Michel Foucault.⁷² This crowdsourced tool enabled Turkers to counter-surveil their protected employers, forcing on the platform a reciprocal transparency. Most professional Turkers used Turkopticon. In its first six years it hosted over 40,000 users, who posted 200,000 reviews of 34,000 requesters.⁷³

Self-fashioning workplace protections added to Turkers’ invisible labor. MTurk was not merely an online pool of research subjects but a complex gig workplace rife with harassment, resistance, and myriad financial incentives for all parties.

Managing Human Subjects

Universities and institutional review boards (IRBs) struggled to keep pace with regulating this research frontier. The University of Washington's finance office warned of “countless opportunities for issues, fraud, [and] lack of accountability” on MTurk, noting that, although researchers heavily relied on it, should excessive risks emerge, the “use of [MTurk] may be prohibited for all of campus.”⁷⁴ In 2013, the US Department of Health and Human Services acknowledged IRB's deficiencies in regulating MTurk research, writing that “current human subjects regulations, originally written over 30 years ago, do not address many issues raised by the unique characteristics of Internet research.” Although some Turker forums recommended that Turkers escalate abusive behavior to researchers’ IRBs, existing reports have yielded little reform.⁷⁵

Researchers, left on their own and with their data quality at stake, published “best practices” for their communities. Most recommended designing “attention checks” into experimental protocol to weed out inattentive, trolling respondents.⁷⁶ For AI training, content moderation, and biomedical applications, attention checks were effective in discovering mistakes, as each task had a correct answer. These quality-assurance mechanisms, while useful for researchers, exacerbated workers’ stress, as workers reported feeling extreme pressure to maintain utmost attentiveness in otherwise monotonous tasks where mistakes were easily discovered.⁷⁷

Even then, such “checks” were insufficient. In contrast to “mechanical tasks,” which “ha[d] a correct answer” and were “known knowns,” verifying behavioral data was murkier. Whereas AI and biomedical researchers sought Turkers as stand-ins for average human intelligence, social scientists used Turkers to produce new behavioral knowledge through surveys, for which, they explained, there were no “objectively correct answer[s].” In survey research, “genuine responses” were “difficult to parse from insincere ones,” and data quality “nearly impossible to observe.” Even when scholars used attention checks on MTurk, their results were ambiguous. Duplicated GPS locations could be mere coincidence, and nonsense answers were open to interpretation. As researchers knew too well, selecting the same answer repeatedly in a survey “is not conclusive evidence of satisficing.” Responding “nice” or “good” to open-ended questions could indicate anything from auto-fill bots to respondents’ carelessness or poor English skills.⁷⁸

In co-opting MTurk for behavioral research, social scientists used this workforce for unintended purposes. Turkers were not inattentive but rather far too attentive, passing attention checks more frequently than traditional subject pools that built much of contemporary behavioral science (e.g., college students). By filtering out low-performing Turkers, academics were essentially selecting for the highest-performing workers (i.e., the most conscientious survey-takers). Whereas undergraduates may have been lax about surveys due to lack of incentives to perform otherwise, Turkers felt intense pressure to pass attention checks so that they could maintain high ratings, remain eligible for future work, and continue earning subsistence wages.⁷⁹ In using a piecework labor platform to employ research participants, social scientists inadvertently crafted incentives that linked their research subjects’ financial self-interests to their survey responses. Researchers had feared that American undergraduates were aberrant in having too few material concerns. Their new reliance on Turkers skewed their data to the opposite extreme.⁸⁰

The appeal of surveys further augmented MTurk's evolutionary pressures. Some feared that surveys were too attractive—“macrotasks” in a sea of microtasks—taking more time, were less automated, and calling for more worker discretion.⁸¹ Surveys offered Turkers welcomed reprieves from other tedious labor, and Turkers regarded research participation as more meaningful than other work. By bringing comparatively more interesting, and sometimes more lucrative, macrotasks to MTurk, researchers unwittingly lured the highest-performing “Super Turkers” to repeatedly seek them out, worsening their research predicament.⁸² In response, academics advised that there was an “art” to pricing tasks, so as not to skew too high or low relative to the site's norms. Prices that were too low yielded little interest in your task, but prices that were too high attracted profiteers entering multiple fraudulent submissions for triple or quadruple pay.⁸³ Pomona College's IRB waived its $15-per-hour minimum rate for research participation if scholars used MTurk, for fear of contributing to so-called participant coercion through inflated wages.⁸⁴ Revising research regulations to suit digital methods, such guidelines, for the sake of producing good data, encouraged scientists to pay the platform's baseline rate, lest their projects enticed scammers. But MTurk's typical pay was so low that research best practices entrenched exploitative wages.

The situation became problematic for academics as “Super Turkers” morphed into a class of “professional” survey-takers. Behavioral science studies were not meant to be conducted on veteran test subjects, so overexposed to surveys that they were no longer “naive” to experimental manipulations. The cognitive reflection test (CRT)—a hallmark of modern behavioral science that tested participants’ likelihood to give answers based on incorrect “gut” instinct versus critical reflection—was especially vulnerable, as it assessed participants’ automatic, deliberate thought processes, often through trickery or deception. Turkers admitted to no longer being naive to CRTs, as many prememorized answers to the most common prompts.⁸⁵

To glimpse what Turkers felt, one may reference one's own experiences as a consumer steeped in modern advertising. Where psychological manipulation tactics are rampant, nonnaïveté becomes a problem not only for laboratory research but also for marketing. Studies document that average consumers, like Turkers, have become numb to repeated behavioral “nudges,” like indications of scarcity (“Only two rooms left!”) and social proof (“16 other people viewed this room”).⁸⁶

Moreover, although most ethics guidelines suggested that deception be reserved as a last resort, studies involving deception (whether attention checks or CRTs) were ubiquitous on MTurk, even in cases where nondeceptive alternatives were available. As Turkers developed canned responses to behavioral scientists’ recycled deception prompts, researchers raced to invent new ones, breeding a vicious cycle of mutual distrust.⁸⁷

The rise of “opt-in” and nonprobability online research panels had provoked concerns about “professional” survey-takers before MTurk's rise. Scholars had wondered whether the same small pool of people repeatedly self-selecting to participate in surveys (for monetary incentives) was skewing data. One 2006 study, for instance, found that 1 percent of online survey panel participants completed 34 percent of questionnaires.⁸⁸ MTurk replicated this problem. Of the 48 million registered global crowdsourcing workers⁸⁹ and five hundred thousand registered Turkers,⁹⁰ likely only 10 percent and 2 percent, respectively, were active. Saturated by surveys, the typical Turker participated in more studies in a week (twenty studies) than the average undergraduate did in a lifetime (fifteen studies).⁹¹

When a research participant completed three hundred academic studies as the median Turker had, scientists’ assessment of that person's “gut” response—as an approximation for averaged human behavior—ceased to have almost any meaning. Turkers’ nonnaïveté corrupted behavioral data, as participants exposed to identical or similar experimental tasks a second time yielded effect sizes reduced by 25 percent. One Turker shared that whenever she felt “really mechanical” after too many surveys, she stopped to “regain [her] humanity,” telling herself, “I'll start again tomorrow when I . . . feel like a person again.”⁹² In this labor regime for mass-producing human behavioral knowledge, participants furnishing raw data were surveyed until they ceased to feel human. As such, Turkers’ public letter-writing campaign to CEO Bezos repeated, “I Am a Human Being, Not an Algorithm.”⁹³

The rise of “Super Turkers” paralleled the rise of “serial participants” in Phase I pharmaceutical trials, the riskiest, first-in-human clinical tests, where subjects earned up to $5,175 per trial. Those participating serially could earn under $20,000 annually, making a career of clinical research participation. While some Phase I “serial participants” talked of unionizing, others crowdsourced tips for surviving the work with fewest negative consequences (e.g., avoiding lumbar punctures when possible).⁹⁴ When research participation became a profession, repeated exposure compounded risks of being studied.

The Tragedy of the (Commons) Crowd

MTurk's nonnaïveté resembled the canonical “tragedy of the commons,” an economics concept wherein individuals with free access to a shared good, like the environment, depleted it for their self-interest, against the common interests of all users. MTurk, a resource of (previously untapped) online human labor, held in common by all potential online requesters, was exploited to such an extreme that it broke.⁹⁵ Behavioral economist David Rand, an early adopter of MTurk for research (having published the first MTurk study in his field), mourned the site's increasing depletion, stating, “Because it got so popular, it's overexploited, and now it doesn't work for the things that I was originally wanting to use it for.” He lamented, “Man, couldn't we have just kept our mouth shut and kept it as this nice, clean thing for ourselves?”⁹⁶

With this common resource in disrepair, trust plummeted between Turkers and requesters, as each side gamified the system in an arms race against the other. Turkers created workarounds to outsmart researchers’ checks and achieve their “personal best,” as one might in sports or video games.⁹⁷ Researchers saw Turkers’ crowdsourced solutions as threats to data integrity. Turker forums became permanent, searchable databases for circumventing research design.⁹⁸ (Online patient forums have similarly challenged double-blind biomedical research as participants crowdsourced information to discover who was in the placebo or treatment groups.)⁹⁹ Through web-based info-sharing, research participants disrupted the secrecy and information asymmetry built into extant protocols for scientific experiments.

Turkers’ online communities, formed in resistance, sadly also suffered from the “tragedy of the commons.” Silberman, Turkopticon's cocreator, reported that workers, requesters, and others attempted to sabotage the resource by creating fake reviewer accounts to artificially sink or inflate requester scores, blackmail requesters, harass other reviewers, or simply “troll.” Despite the creators’ intention to design a “worker-centered” tool, Silberman received harsh appraisal from Turkers for his stewardship, noting that “Turkopticon itself has been subject to criticism at least as severe as that received by [MTurk] itself.” He admitted that this was warranted, as he and Irani “struggled to respond effectively” to abusive behavior occurring on Turkopticon.¹⁰⁰ Turkers perceived Turkopticon as a contaminated commons, reporting that they distrusted its ratings because they were written by other Turkers competing for the site's best tasks. In their experiences, good requesters were well rated, but the best requesters were rated either poorly or not at all. One worker explained, “Many Turkers won't give away their great ones for fear the work will get scooped up from them.”¹⁰¹

Assisting invisible labor, in this case, also required invisible labor. Silberman served as the system's de facto lead programmer and database administrator since its creation, and he and Irani moderated the forums alongside volunteers.¹⁰² Such arrangements, he noted, could not continue “indefinitely,” as Turkopticon had no funds to pay for staff or operational costs.¹⁰³ Academic free labor, therefore, also had its limits. In 2016, Silberman sought to forge a new commons with professional Turker Rochelle LaPlante called MTurk Crowd, “a worker-owned, democratic forum” allowing openness, transparency, and collaboration.¹⁰⁴ Collective action was possible in a digitally distributed workforce. But as the labor pool was squeezed, fierce competition between fragmented actors constantly threatened common resources to collapse on themselves.

In the void of a collapsed commons, new, for-profit companies emerged to address crowdsourced research's shortcomings. Data consultancies courted scientists to assist in designing and implementing MTurk surveys. Competitors, often founded by behavioral scientists familiar with MTurk's flaws, offered expanded participant pools in Europe; guaranteed research-naive subjects; fraud, location, identity, and VPN detection; participant protection; mediation of researcher-worker disagreements; handling of workplace abuse reports; and a minimum wage of $6.50 per hour, or “fair living wage.” Such companies hoped their platforms could build “trust and virtuous cycles” that MTurk had not.¹⁰⁵ The original, however, has proven difficult to unseat. Some challengers followed it in designing themselves as a hoax within a hoax, seeking to overtake MTurk by using MTurk. One contender, for instance, sought to surpass MTurk but drew 20 percent of its labor force from MTurk.¹⁰⁶

Academics in Capitalism

Social scientists had themselves been economic subjects facing market pressures. Since at least the depressed job markets of 1979, they argued, increased difficulty of academic survival required researchers to produce “low-difficulty, high-volume studies.” The push for shorter studies, more studies, and more participants found its solution in MTurk. In 2005, no studies in top behavioral science journals used MTurk. By 2015, one-third did, and half of all studies occurred online. MTurk studies were, on average, shortest in length (taking least amount of time), even compared to laboratory tests of undergraduates. Pressure for research volume was intense, with the average number of studies in each published article among top journals climbing from 1.27 in 1968, to 1.58 in 1978, 1.78 in 1988, 2.75 in 2005, 3.09 in 2010, and 3.80 in 2015. MTurk met contemporary researchers’ need for speed and volume cheaply. What behavioral science lost in the MTurk revolution, however, were high-difficulty, low-volume studies needed to tackle human behavior's most challenging questions.¹⁰⁷

MTurk's controversies highlighted that research had always been implicated in labor relations. From the golden age of laboratory study of undergraduates and prior, the human sciences had possessed insatiable demand for cheap, plentiful, and naive populations to feed their research. MTurk merely made explicit formerly shrouded labor relations in knowledge production. Just as employers downplayed crowdsourced labor as workers’ supplemental income and recreation “to stop questions about . . . actual working conditions of actual people,” similar frames long obscured the underpaying of undergraduate research participants.¹⁰⁸ Compensating undergraduates was an easy question to skirt in part because their student status seemed to exempt them from consideration as workers. But as more workers relied on survey participation for their entire income, academics became tightly ensnared in questionable labor ethics.

MTurk marked a key shift in the political economy of research participation as the primary arena for recruiting worker-participants migrated from universities to an explicitly profit-maximizing institution like Amazon, where digital piecework was part of the business model. Universities, even neoliberal ones—job market pressures and incentives notwithstanding—still ostensibly functioned differently from for-profit businesses. MTurk's muddying of distinctions between for-profit employment ethics and nonprofit research ethics shed light on worker-participants’ exploitation while also exposing the profit-naive guise of academic research. One scholar observed that her peers “rationalize payment decisions by suggesting that paying low wages to many people is the only way to establish the statistical power necessary to create publishable research.”¹⁰⁹ As MTurk illustrated, within current incentive structures, willingness to exploit workers went hand in glove with productivity and success in research.

Plagued by shrinking research budgets and widespread underemployment, social scientists improvised their own strategies to achieve statistical power needed to publish, or perished.¹¹⁰ When not even deception worked to rein in MTurk survey respondents, scholars went rogue. In more than one-third of studies using MTurk, researchers selectively deleted survey responses during data analysis, sometimes arbitrarily (e.g., to eliminate “suspicious” outlier data). Many failed to disclose in their published articles that they had done so. Researchers mirrored the lawlessness of the MTurk labor terrain by taking their own vigilante liberties, allowing themselves enormous leeway in correcting data that contradicted their hypotheses. Many found it easy to discard data at will because of the low wages they paid respondents, asking, “Who would do studies for such low pay anyway?”¹¹¹ Like capitalist employers, researchers paid MTurk's prevailing wage, extracted surplus labor-value, and, when laborers attested to terrible conditions, claimed that the labor was worthless all along.¹¹² Like cost-cutting practices across other industries, these innovations revealed their final consequences in their shoddy products. The professionalization of research participation threatened knowledge production, so researchers responded. The result was social science that purported to capture truth about human behavior but was itself marred by its distorted processes of production.

Conclusion: Humans in the Human Sciences

This article featured the contradictions underlying AI's tenuous promissory claims, illustrated through the machinations of a labor platform built for tasks that any mundane human could purportedly do more effectively than the best AI. A tongue-in-cheek 2007 New York Times review of MTurk concluded wryly, “We probably have at least another 25 years before computers are more powerful than human brains.” The shortfalls of AI guaranteed people job security because until computers surpass human intelligence, “people will be able to sell their idle brains” on platforms like MTurk.¹¹³ But Turkers were sold, one microtask at a time, not so much as idle brains as idle humans. By being human, workers qualified for most MTurk tasks, no training necessary. Turkers provided not brain power but intuitive, contextual, and innate human processing capabilities—a human essence the site called “human intelligence”—that artificial intelligence could not yet adequately copy. MTurk's disparate work, whether in AI or behavioral research, shared one key similarity: they all sought a humanness—to capture, approximate, and ultimately surpass that human essence. As the almost twenty years since the site's public launch has revealed, human assistance was not a preparatory step for AI's rise, to be discarded once technology was sufficiently powerful; rather, it is an integral part of how AI has been designed, trained, and deployed.¹¹⁴ No matter how advanced technology became, cheap, fungible, and often hidden human labor had to usher it along its way. The platform has been described as anything from “a spooky, but elegant, business idea” to a novel creation that was “at once an opportunity, a sweatshop, and a game.”¹¹⁵ But MTurk's primary genius was in how its very name satirized the labor exchange occurring on the site, exposing the hoax within AI, all while profiting seamlessly from the collective power of the underemployed, exploited, and fragmented twenty-first-century crowd.

Human behavioral data, commentators argued, is the prime currency in this age of surveillance capitalism, the economic surplus from which most profits are and will be generated. Whereas some citizen science participants have been credited as coauthors of research, Turkers have not—because Turkers were not so much coagents in producing scientific knowledge as they were raw materials for its creation. Psychologist David O. Sears, when critiquing his field's sole reliance on undergraduate research participants in 1980, dreamed of a return to the norms of the 1940s and 1950s, when behavioral research was conducted on adults “in their natural habitats with materials drawn from ordinary life.” Troubling as MTurk's labor and research developments were, their next iteration may be bleaker, as researchers bypass even platforms like MTurk to harvest data, in situ and cost free, directly from homes, devices, and bodies of technologies’ users.¹¹⁶

In the final analysis, MTurk is a case study not only in human cognition but also in human nature itself. Driven by perennial labor shortages in social science research participants, academics’ methodological innovation to use MTurk for research led to a vicious cycle in which scholars and Turkers attempted to outmaneuver each other for financial gain, knowledge production, and self-protection. As workers devised bots and strategies for earning more money, researchers engineered new “checks” for verifying participants’ attention, honesty, and humanness (proving they are not bots). In an arms race of exploitation and resistance, Turkers worked feverishly for abysmal pay, and cheapskate professors were burned by their exploited subjects. Behavioral scientists may have exploited online labor to gain insights into human behavior, but the ensuing labor conflict between workers and academics painted just as much a portrait of humanity as survey research did. Scientists seeking to understand abstract or generic human behavior ran up against the unavoidable, human particularities of workers. This drama revealed an essence of human propensities: willingness to exploit others, and human ingenuity in resisting exploitation.¹¹⁷

In the wake of ChatGPT, the large language model (LLM) developed by OpenAI that launched in late 2022, the dynamics highlighted in this article have only intensified. One new study tracked Turkers’ copy-and-paste keystrokes, discovering that likely 33 to 46 percent of Turkers secretly used ChatGPT to complete their work. This trend, if it were to proliferate, would be problematic for the AI industry, as AI training may increasingly, and inadvertently, be outsourced to AI itself. Creating another loop in the recursive pun, the authors titled this new study “artificial artificial artificial intelligence,” remarking that “our results call for . . . new ways to ensure that human data remain human.”¹¹⁸

This project illuminated the role of labor in knowledge creation—and the role of knowledge industries in labor and economic production. Its key tensions implicate not only experimental psychology but also research more broadly. During the “replication crisis” that hit psychology and biomedicine in the 2010s, scholars found themselves unable to reproduce core findings in their disciplines. The Open Science Collaboration in 2015 replicated only 36 percent of results from one hundred studies, the pharmaceutical company AmGen replicated only six out of fifty-three “landmark” basic cancer research studies, and psychology replicated only 1 percent of its research.¹¹⁹ By scientists’ self-reporting, the conditions of modern academia spurred misconduct, as publication pressures that medical scientists felt correlated with their likelihood to commit research fraud.¹²⁰ As markets squeezed academics, costs trickled down not only to research subjects (workers) but also to consumers—those in the public for whose supposed benefit new knowledge was produced. Knowledge production was not exempt from other industries’ lessons. Economic pressures to produce quickly, coupled with limited initial understanding about cost-cutting's consequences, yielded flimsy output to the detriment of all who relied on those products. The durability and quality of knowledge products have come home to roost.

This article began as a term paper in a class taught by Rebecca Lemov. I'm grateful for the generous feedback I received on drafts from Rebecca Lemov, David Jones, Lizabeth Cohen, Elizabeth Lunbeck, Lisa McGirr, Aaron Bekemeyer, Leah Xue, the Harvard Bowdoin Prize Committee, and the Business History Conference (BHC) K. Austin Kerr Prize Committee; and on presentations of this research at the BHC and the European Society for the History of the Human Sciences. Support from the National Science Foundation Graduate Research Fellowship Program made this research possible. My deep gratitude also goes to Labor's reviewers and editorial team for the time and care they put into reviewing this piece. Their close attention and incisive feedback were instrumental in producing a clearer, stronger article.

Notes

References