The Black Box Society: The Secret Algorithms That Control Money and Information is a 2015 academic book authored by law professor Frank Pasquale that interrogates the use of opaque algorithms—referred to as black boxes—that increasingly control decision-making in the realms of search, finance, and reputation.
Pasquale uses the term "black box" as a metaphor with dual meanings: a black box can describe both a recording device (such as a data-monitoring system), as well as a system whose inner workings are secret or unknown.Pasquale 2016, p. 3 The 319-page academic book, published by Harvard University Press, contains six chapters. Chapter one introduces the challenge of investigating technologies whose functions are overwhelmingly complex and incredibly mysterious.Pasquale 2016, pp. 1-18 Chapter two examines citizens’ digital reputation and the automated decision-making that can perpetuate systemic disadvantage for some while advantaging others.Pasquale 2016, pp. 19-58 Chapter three exposes the hidden mechanisms of profit-driven search engines through a series of disputes over bias and abuse of power in Silicon Valley.Pasquale 2016, pp. 59-100 Chapter four investigates the deeply problematic, unethical use of automated decision-making in the finance industry.Pasquale 2016, pp. 101-139 Chapter five deconstructs the need to open black boxes,Pasquale 2016, pp. 140-188 while chapter six stresses the emergent threat that black boxes pose to democratic societies and capitalist economies, as well as the need for an informed, autonomous citizenry.Pasquale 2016, pp. 189-220
The Black Box Society has been reviewed in several academic journals by experts in the field, who largely praise the book for both its originality and timeliness as well as its vital contributions to the areas of law, technology, and social science. However, the book has received some critical feedback on its conception of transparency as a solution to black boxes—raising questions surrounding privacy protection and ethics—as well as its operationalization of the term "black box."Chander 2016Ranchordás 2016Rubel 2016Spina 2015
Definition of "black box" In academic discourse, the usage of the term “black box” dates back to at least 1963 with Mario Bunge's work on a black box theory in mathematics.
The term “black box,” as used throughout The Black Box Society by author and law professor, Frank Pasquale, is a dual metaphor for a recording device such as a data-monitoring system and for a system whose inner workings are secret or unknown.Pasquale 2016, p. 3 According to Pasquale, in a black box society, individuals encounter these dual meanings every day: as people are pervasively tracked by private firms and government, they come to lack a meaningful understanding of where their data travels, how their data is used, and the potential consequences of data usage and disclosure by these organizations.Pasquale 2016, p. 3
Contents Central thesis Published by Harvard University Press in 2016, The Black Box Society has six chapters, totalling 319 pages. In his review of the book for Business Ethics Quarterly, law professor Alan Rubel identifies Pasquale's central thesis: the algorithms which control and monitor individual reputation, information seeking, and data retrieval in the search, reputation, and finance sectors embody "some combination of too powerful, not subtle enough, and too obscure."Rubel 2016, p. 568
Chapter one: The Need to Know Chapter one of The Black Box Society introduces the epistemological problem that emerges from the increasing adoption of enigmatic technologies: “We cannot understand, or even investigate, a subject about which nothing is known.”Pasquale 2016, p. 1 The author contends that American law has grown silent as questions of personal privacy increasingly arise, instead prioritizing corporate secrecy.Pasquale 2016, p. 3 This incongruity of secrecy and complexity in the information economy is the focus of The Black Box Society.Pasquale 2016, p. 3 Pasquale declares that a black box society is “closely resembles a one-way mirror.”Pasquale 2016, p. 11 In other words, processes of Big Data collection, usage, and disclosure by private and public organizations loom invisibly. Pasquale argues that this is especially true in the areas of reputation, search, and finance—increasingly expressed through algorithms—thereby compromising individual freedoms and market fairness.Pasquale 2016, p. 5
The author's discussion of the power of secrecy is informed by the work of legal scholar Jack M. Balkin, who argues that one of the most important forms of power entails the privilege to interrogate others without being interrogated oneself. Pasquale repurposes this constitutional phenomenon to argue that Internet companies that are resistant to regulation collect increasingly mass quantities of sensitive personal data on their users, with no way for those same users to exercise meaningful consent over their "digital dossiers."Pasquale 2016, p. 4
According to Pasquale, while companies assemble digital dossiers about the intimate lives of private individuals, the decisions these entities make for themselves are influenced as Internet and finance businesses become decision-makers about how they can or should live their lives by way of credit scoring, predictive analytics, and recommendation engines.Pasquale 2016, pp. 4-5
Chapter two: Digital Reputation in an Era of Runaway Data In Digital Reputation in an Era of Runaway Data, chapter two of The Black Box Society, Pasquale declares that private data is a significant source of profit for other people, but often at the expense of the person to whom the data belongs.Pasquale 2016, p. 19
According to the author, data brokers use data mining to analyze private and public records in order to draw inferences about people, largely unrestrained by government laws; it is only leaks, investigations, and complaints that crack open the black boxes of reputation analysis.Pasquale 2016, p. 21 The author labels the exchange of sensitive personal data between commercial and government organizations as a form of “dataveillance."
Pasquale explains the pervasive process of digital scoring, wherein “many of our daily activities are processed as ‘signals’ for rewards or penalties, benefits or burdens” by algorithms.Pasquale 2016, p. 21 Pasquale's main concern here is that original black boxed techniques in credit reporting are now becoming used in other relatively unregulated sectors including medicine, law enforcement, marketing, and employment.Rubel 2016 The author deems this “runaway data” as not only “creepy” but also having substantial costs.Pasquale 2016, p. 26
Pasquale highlights the concern related to the emergent racial bias inherent in automated decision-making alongside other forms of identity-based prejudice and discrimination. These decisions are “subtle manipulations that are nearly impossible to discern for ordinary citizens not privy to the internal computer code.”Chander 2017, p. 1026 Reputation-ranking algorithmic systems are programmed by human beings who cannot easily separate the embedding of their implicit biases and values into the software that they code.Pasquale 2016, p. 38
The author cites a 2012 research study on algorithmic discrimination by computer scientist Latanya Sweeney, former director of the Data Privacy Lab at Harvard University and now senior technologist at the Federal Trade Commission, who found that online advertisements suggestive of arrest records appear more frequently when searching for names that are commonly associated with African American cultures.
Chapter two concludes with a call to action in regard to justice system reform: “It is time to reclaim our right to the presumption of innocence, and to the security of the light.”Pasquale 2016, p. 57
Chapter three: The Hidden Logics of Search Chapter three of The Black Box Society, The Hidden Logics of Search, exposes the new masters of the Internet: search engines that influence, reshape, and answer billions of queries every day.Pasquale 2016, pp. 59-60 Pasquale asserts that search engines have a dark side: opacity. Platforms make decisions daily that impact personal, commercial, and democratic matters by way of automated personalization.Pasquale 2016, p. 60 The author thus asks how far the public can trust the people who make these black boxed systems that possess the social, cultural, and political power to include, exclude, and rank people in the digital economy.Pasquale 2016, p. 81 Furthermore, Pasquale emphasizes that market forces are inadequate in protecting optimal levels of privacy, writing, “In an era where Big Data is the key to maximizing profit, every business has an incentive to be nosy.”Pasquale 2016, p. 81
Socially constructed as neutral and objective, the author highlights how these algorithmic systems regularly make value-laden decisions imbued with bias that come to reflect people's perception of the world.Pasquale 2016, p. 61 This chapter examines four areas wherein the behaviour of major search engines and companies raises concerns over trust—transparency, competition, compensation, and control—using case studies of disputes over bias and abuse of power in Silicon Valley.Pasquale 2016, p. 61
One case study interrogated by Pasquale is the deflection of concerns by microblogging platform X, then named Twitter, over its automated decision-making regarding trending hashtags.Pasquale 2016, p. 75 On X, hashtags can nominate certain terms as “trending”—or interesting and relevant—and tweets containing trending hashtags become recommended and broadcast to all users.Pasquale 2016, p. 76 According to Pasquale, in late September 2011, the Occupy Wall Street protest movement began gaining media traction, and despite the hashtags #OWS and #occupy being used in numerous tweets, these tweets did not appear on Twitter's official Trending Topics list. Supporters and activists accused Twitter of algorithmic suppression, condemning the censoring of a politically controversial movement. Twitter subsequently released a statement, explaining that Trending Topics is dependent on tweet velocity, not popularity, and the contention settled, sparking discussions surrounding media literacy.
Chapter four: Finance’s Algorithms: The Emperor’s New Codes In chapter four of The Black Box Society, Finance's Algorithms: The Emperor's New Codes, Pasquale investigates the increasing computerization and consequent lack of transparency in the finance sector over the past several decades.Pasquale 2016, p. 102 The author contends that the 2008 financial crisis exposed the hidden practices of large banks: bad data, bad apparatuses, and devious corporate structures.Pasquale 2016, p. 120
According to Pasquale, secret algorithms are “obscured by a triple layer of technical complexity, secrecy, and ‘economic espionage’ laws that can land would-be whistle-blowers in prison,” thereby restricting citizens from truly knowing how major financial firms collect, use, and disclose their personal data.Pasquale 2016, p. 103 Citizens also cannot know when complex, trade-secret mathematic calculations including rating formulas are drawing from biased, incomplete, or inaccurate data.Pasquale 2016, p. 103 This obscurity causes instability and conflict, according to the author.Pasquale 2016, p. 103