Panagiotis G. Ipeirotis (also known as Panos Ipeirotis; born May 3, 1976, in Serres, Greece) is a Greek-American computer scientist and the Merchants' Council Professor of Technology and Business at the New York University Stern School of Business. His research focuses on data mining, crowdsourcing, human computation, and the economics of online information systems.
Ipeirotis received the Lagrange Prize in Complex Systems (2015) and the ACM SIGKDD Test of Time Award (2020). His research on Amazon Mechanical Turk underpinned the Pew Research Center's 2016 methodology for estimating the platform's size and has informed reporting on crowdsourced labor in outlets including The New York Times and The Economist.
In addition to his academic career, Ipeirotis co-founded the AI consulting firm Detectica in 2015, which was acquired by Compass, Inc. in 2019. He has also held research positions at Google (2013–2014) and at Meta's Reality Labs division (2024–2025).
Early life Ipeirotis was born on May 3, 1976, in Serres, Greece. In 1994, he won a Gold Medal at the 8th National Greek Competition in Chemistry. Later that year, he represented Greece at the 26th International Chemistry Olympiad in Oslo, Norway.
Education Ipeirotis earned his Diploma in Computer Engineering and Informatics (CEID) from the University of Patras in 1999. He pursued graduate studies at Columbia University under the supervision of Luis Gravano, receiving his M.Sc. in 2001, M.Phil. in 2003, and Ph.D. in Computer Science in 2004.
Academic career Ipeirotis began his academic career as a graduate research assistant at Columbia University (1999–2004). In 2004, he joined the Department of Information, Operations, and Management Sciences at the New York University Stern School of Business as an Assistant Professor. He was promoted to Associate Professor in 2010 and Full Professor in 2016. He holds a courtesy appointment at the Courant Institute of Mathematical Sciences and is an associated faculty member at the NYU Center for Data Science.
Ipeirotis has served as Program Co-Chair for the 13th ACM Conference on Electronic Commerce in 2012, as Technical Program Co-Chair for The Web Conference 2018 (WWW 2018), and as an Area Chair for the AAAI Conference on Artificial Intelligence in 2021 and 2023. He co-organized the Human Computation Workshop series from its founding edition in 2009 (co-located with SIGKDD) through 2012, and served as General Co-Chair of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP) in 2015, after the series had grown into a stand-alone AAAI conference. He has served on the editorial boards of Management Science, IEEE Transactions on Knowledge and Data Engineering, the INFORMS Journal on Computing, and the ACM Journal of Data and Information Quality, and was a founding co-Editor-in-Chief of the journal Collective Intelligence, launched in 2022 as a collaboration between SAGE Publications, the Association for Computing Machinery, and Nesta.
Ipeirotis has been the subject of media profiles in business and Greek-language outlets. Bloomberg Businessweek profiled him in 2011, describing his work on technology that combines human and machine intelligence, and again in 2013, in a feature that called him the "data dude" of business analytics. A 2014 Kathimerini interview discussed his career and conditions in Greek higher education.
Academic lineage and doctoral students Ipeirotis's doctoral advisor Luis Gravano was himself a student of Hector Garcia-Molina at Stanford. According to the Mathematics Genealogy Project, Ipeirotis has advised seven doctoral students at NYU and Columbia.
His doctoral students include Beibei Li (now at Carnegie Mellon University's Heinz College) and Marios Kokkodis, both of whom subsequently received INFORMS Information Systems Society early-career awards — the Sandra A. Slaughter Early Career Award in 2019 and the Gordon B. Davis Young Scholar Award in 2020. Former postdoctoral researcher Djellel Difallah is Assistant Professor at New York University Abu Dhabi.
Research Ipeirotis's research spans text mining, crowdsourcing, and the economics of online information systems. His work has been published in venues such as Management Science, Information Systems Research, and IEEE Transactions on Knowledge and Data Engineering.
Text mining and web databases Ipeirotis's doctoral research at Columbia University, supervised by Luis Gravano, addressed the problem of accessing text databases on the deep web — content behind search interfaces that conventional web crawlers cannot index. An early 2001 paper introduced a query-probing method for categorizing hidden-web databases by topic without retrieving any documents, which the thesis later generalized into the QProber system using machine learning classifiers to derive the probing queries. In parallel, Ipeirotis contributed to research on data integration: a 2001 paper at the International Conference on Very Large Data Bases (VLDB), co-authored with Gravano, H. V. Jagadish, Nick Koudas, S. Muthukrishnan, and Divesh Srivastava, showed how approximate string joins — matching slightly different versions of the same entity across databases — could be implemented entirely within standard relational database systems, without specialized indexes.
Building on this foundation, Ipeirotis published two award-winning papers on modeling and querying text databases. A 2005 study applied survival analysis techniques to model how the content of 152 web databases changed over a 52-week period, winning the Best Paper Award at the 21st IEEE International Conference on Data Engineering. His 2006 paper "To Search or to Crawl?" presented a cost-based framework for choosing between querying and crawling text databases when executing information extraction tasks, and won the Best Paper Award at ACM SIGMOD. In 2007, Ipeirotis received a National Science Foundation CAREER Award for the project "Towards a Text-Centric Database Management System," which funded research on query optimization for information extraction and on the economic analysis of textual content.
Ipeirotis's subsequent research extended in two directions: crowdsourcing quality management, addressing the challenge of obtaining reliable labeled data from non-expert workers; and a collaboration with Anindya Ghose that used economic signals such as prices and sales as implicit indicators of textual content quality, an approach they termed "EconoMining."
Crowdsourcing and human computation Ipeirotis's 2010 paper "Running Experiments on Amazon Mechanical Turk," co-authored with Gabriele Paolacci and Jesse Chandler, provided methodological guidance for using the platform — commonly abbreviated MTurk — in behavioral research.
A 2010 study by Ipeirotis estimated that approximately 40% of tasks posted by new requesters on MTurk involved spam or low-quality content, drawing media coverage.
His 2010 demographic analysis of the MTurk workforce provided early data on worker motivations, geographic distribution, and income levels, and has been cited by researchers studying crowdsourcing platforms. A follow-up study with former postdoctoral researcher Djellel Difallah and Elena Filatova, presented at the ACM International Conference on Web Search and Data Mining in 2018, tracked changes in the MTurk workforce over an eight-year period. Media outlets have continued to draw on this body of work as concerns about the human labor underlying machine learning training data have grown, including coverage in Wired and The New York Times on automated responses on MTurk and a 2026 Marketplace interview in which Ipeirotis discussed CAPTCHA completion and other routine online interactions as unrecognized AI-training-data labor. The Financial Times cited his demographic and wage data on Mechanical Turk workers in a 2016 examination of the human workforce behind commercial artificial intelligence, and The Wall Street Journal quoted him in 2017 on the continued reliance of large companies on human workers within their AI systems. In 2020, Wired quoted him on an apparent influx of new workers onto Mechanical Turk during the COVID-19 pandemic, including workers based in Canada, Italy, and Brazil.
This line of work also led to the development of quality management techniques for crowdsourcing. The 2008 paper "Get Another Label?," co-authored with Foster Provost and Victor S. Sheng, addressed the problem that training machine learning classifiers on labels from non-expert workers degrades model accuracy when those labels are noisy. The paper studied whether repeated labeling of uncertain items could improve classifier performance under noisy labels, and proposed selective repeated-labeling strategies for crowdsourced annotation. It received the Best Paper Runner-Up Award at the 2008 ACM SIGKDD Conference on Knowledge Discovery and Data Mining and the SIGKDD Test of Time Award in 2020. A 2025 survey of learning from crowdsourced noisy labels—the problem of combining unreliable labels from many non-expert workers—noted that large language models such as ChatGPT, Gemini, and Llama are fine-tuned on substantial volumes of crowdsourced labels, including through reinforcement learning from human feedback (RLHF) and direct preference optimization.
ProPublica drew on Ipeirotis's research and blog posts in publishing its "Guide to Mechanical Turk" for investigative reporters, citing his work on data quality through multiple worker verification. Ipeirotis has also commented on the use of crowdsourcing for content moderation; in a 2012 Foreign Policy article on image moderation, he described comparing the judgments of moderators from different regions as a way to reduce cultural bias.
To support this research, Ipeirotis developed "MTurk Tracker," a set of tracking tools for the MTurk platform. The first tool tracked workforce demographics using capture-recapture techniques to estimate the size and composition of the workforce in near real-time. The second tool periodically downloaded all tasks posted on the platform, enabling analysis of the crowdsourcing marketplace's size and activity. The Pew Research Center's 2016 report "Research in the Crowdsourcing Age" drew on data from the task tracker, explicitly crediting "the online tool mturk-tracker, which is run by Dr. Panagiotis G. Ipeirotis." The 2019 book Ghost Work, by Mary L. Gray and Siddharth Suri, drew on Ipeirotis's worker-demographics research to estimate the size of the platform's active workforce.
During a 2013–2014 sabbatical at Google, Ipeirotis worked on crowdsourcing methods for improving the Google Knowledge Graph. He co-developed "Quizz," a gamified system that presented data collection as trivia quizzes targeted by domain expertise, using calibration questions with known answers to assess user reliability in real-time. In a related paper with Chun How Tan, Eugene Agichtein, and Evgeniy Gabrilovich, Ipeirotis examined methods for predicting contribution quality in knowledge-base construction and curation.
Ipeirotis also applied crowdsourcing methods to building training data for computer vision systems. With Pietro Perona and Serge Belongie — co-recipients with Ipeirotis of a 2012 Google Focused Research Award — he co-authored a 2015 paper at the IEEE Conference on Computer Vision and Pattern Recognition describing the construction of a large-scale dataset for fine-grained bird species recognition using citizen-scientist labels.
In a 2015 article in the ACM Journal on Data and Information Quality, co-authored with Josh Attenberg and Foster Provost, Ipeirotis introduced "Beat the Machine," a framework in which crowdworkers were rewarded for finding examples that a trained classifier predicted incorrectly. The approach was proposed as a method for surfacing the "unknown unknowns" — systematic blind spots — of machine learning models.
Data quality and record linkage