Synthetic media is digital content in various media formats, including text, image, and video, which has been automatically and artificially produced or manipulated. Although not all synthetic media is generated by artificial intelligence (AI), it often refers to the use of generative AI (GenAI) to produce content, such as deepfakes, through the use of AI within a set of human-prompted parameters.
Synthetic media as a field has grown rapidly since the creation of generative adversarial networks (GANs), primarily through the rise of deepfakes as well as music synthesis, text generation, human image synthesis, speech synthesis, and more. Though experts use the term "synthetic media," individual methods such as deepfakes and text synthesis are sometimes not referred to as such by the media but instead by their respective terminology (and often use "deepfakes" as a euphemism, e.g., "deepfakes for text" for natural-language generation; "deepfakes for voices" for neural voice cloning, etc.) The field of synthetic media gained significant attention starting in 2017, when Motherboard reported on the emergence of AI-altered pornographic videos to insert the faces of famous actresses. Potential hazards of synthetic media include the spread of misinformation, further loss of trust in institutions such as media and government, the mass automation of creative and journalistic jobs, and a retreat into AI-generated fantasy worlds. Synthetic media is an applied form of artificial imagination.
History
Pre-1950s
The idea of automated art dates back to the automata of ancient Greek civilization. Nearly 2,000 years ago, the engineer Hero of Alexandria described statues that could move and mechanical theatrical devices. Over the centuries, mechanical artworks drew crowds throughout Europe, China, and India. Other automated novelties, such as Johann Philipp Kirnberger's "Musikalisches Würfelspiel" (Musical Dice Game) 1757, also amused audiences.Nierhaus, Gerhard (2009). Algorithmic Composition: Paradigms of Automated Music Generation, pp. 36 & 38n7. .
Despite the technical capabilities of these machines, however, none were capable of generating original content and were entirely dependent upon their mechanical designs.
Rise of artificial intelligence
The field of AI research was born at a workshop at Dartmouth College in 1956,Dartmouth conference:
pp. 47–49, who writes "the conference is generally recognized as the official birthdate of the new science." p. 17, who call the conference "the birth of artificial intelligence." begetting the rise of digital computing used as a medium of art as well as the rise of generative art. Initial experiments in AI-generated art included the Illiac Suite, a 1957 composition for string quartet, which is generally agreed to be the first score composed by an electronic computer.Denis L. Baggi, "The Role of Computer Technology in Music and Musicology ", lim.dico.unimi.it (December 9, 1998). Lejaren Hiller, in collaboration with Leonard Issacson, programmed the ILLIAC I computer at the University of Illinois at Urbana–Champaign (where both composers were professors) to generate compositional material for his String Quartet No. 4.
In 1960, Russian researcher R.Kh.Zaripov published the first paper worldwide on algorithmic music composing using the "Ural-1" computer.
In 1965, inventor Ray Kurzweil premiered a piano piece created by a computer that was capable of pattern recognition in various compositions. The computer was then able to analyze and use these patterns to create novel melodies. The computer was debuted on Steve Allen's I've Got a Secret program, and stumped the hosts until film star Harry Morgan guessed Ray's secret.
Before 1989, artificial neural networks (ANNs) were used to model certain aspects of creativity. Peter Todd (1989) first trained a neural network to reproduce musical melodies from a training set of musical pieces. Then he used a change algorithm to modify the network's input parameters. The network was able to randomly generate new music in a highly uncontrolled manner.Todd, P.M., and Loy, D.G. (Eds.) (1991). Music and connectionism. Cambridge, MA: MIT Press.
In 2014, computer scientist Ian Goodfellow and his colleagues developed a new class of machine learning systems: generative adversarial networks (GANs). Two neural networks contest with each other in a game (in the sense of game theory, often but not always in the form of a zero-sum game). Given a training set, this technique learns to generate new data with the same statistics as the training set. For example, a GAN trained on photographs can generate new photographs that look at least superficially authentic to human observers, having many realistic characteristics. Though originally proposed as a form of generative model for unsupervised learning, GANs have also proven useful for semi-supervised learning, fully supervised learning, and reinforcement learning. In a 2016 seminar, computer scientist Yann LeCun described GANs as "the coolest idea in machine learning in the last twenty years".
In 2017, Google unveiled transformers, a new type of neural network architecture specialized for language modeling that enabled for rapid advancements in natural language processing. Transformers proved capable of high levels of generalization, allowing networks such as GPT-3 and Jukebox from OpenAI to synthesize text and music respectively at a level approaching humanlike ability. There have been some attempts to use GPT-3 and GPT-2 for screenplay writing, resulting in both dramatic (the Italian short film Frammenti di Anime Meccaniche, written by GPT-2) and comedic (the short film Solicitors by YouTube creator Calamity AI written by GPT-3) narratives.
Branches of synthetic media
Deepfakes
Deepfakes (a portmanteau of "deep learning" and "fake") are the most prominent form of synthetic media. Deepfakes are media productions that use an existing image or video and replace the subject with someone else's likeness using ANNs. They often combine and superimpose existing media onto source media using machine learning techniques known as autoencoders and GANs. Deepfakes have garnered widespread attention and criticism for their use in celebrity pornographic videos, revenge porn, fake news, hoaxes, and financial fraud. This has elicited responses from both industry and government to detect and limit their use.
The term deepfakes originated around the end of 2017 from a Reddit user named "deepfakes". He, along with others in the Reddit community r/deepfakes, shared deepfakes they created; many videos involved celebrities' faces swapped onto the bodies of actresses in pornographic videos, while non-pornographic content included videos with actor Nicolas Cage's face swapped into various movies. In December 2017, Samantha Cole published an article about r/deepfakes in Vice that drew the first mainstream attention to deepfakes being shared in online communities. Six weeks later, Cole wrote in a follow-up article about the increase in AI-assisted fake pornography. According to a study conducted by Sensity, a company that detects and tracks deepfakes online, 85,047 deepfake videos had been found on online streaming websites by December 2020. This number was expected to double every six months. In September 2019, Sensity revealed that 96% of the fake videos were non-consensual pornography. Most of the victims of these videos were celebrities or high-profile individuals.
In February 2018, r/deepfakes was banned by Reddit for sharing involuntary pornography. Other websites have also banned the use of deepfakes for involuntary pornography, including the social media platform X (formerly Twitter) and the pornography site Pornhub. However, some websites have not banned Deepfake content, including 4chan and 8chan.
Non-pornographic deepfake content continues to grow in popularity with videos from YouTube creators such as Ctrl Shift Face and Shamook. A mobile application, Impressions, was launched for iOS in March 2020. The app provides a platform for users to deepfake celebrity faces into videos in a matter of minutes.
Image synthesis
Image synthesis is the artificial production of visual media, especially through algorithmic means. In the emerging world of synthetic media, the work of digital-image creation—once the domain of highly skilled programmers and Hollywood special-effects artists—can be automated by expert systems capable of producing realism on a vast scale. One subfield of this includes human image synthesis, which is the use of neural networks to make believable and even photorealistic renditionsPhysics-based muscle model for mouth shape control on IEEE Explore (requires membership)Realistic 3D facial animation in virtual space teleconferencing on IEEE Explore (requires membership) of human-likenesses, moving or still. It has effectively existed since the early 2000s. Many films using computer-generated imagery (CGI) have featured synthetic images of human-like characters digitally composited onto the real or other simulated film material. Since the end of the 2010s, deep learning AI has been applied to synthesize images and video that look like humans, without need for human assistance, once the training phase has been completed—whereas the old school 7D-route required massive amounts of human work. The website This Person Does Not Exist showcases fully automated human image synthesis by generating images that look like facial portraits of human faces.
Audio synthesis
Beyond deepfakes and image synthesis, audio is another area where AI is used to create synthetic media. Synthesized audio will be capable of generating any conceivable sound that can be achieved through audio waveform manipulation, which might conceivably be used to generate stock audio of sound effects or simulate audio of currently imaginary things.
AI art
Artificial intelligence visual art, or AI art, is artistic content generated or assisted by AI programs. The classification of AI output as "art" remains controversial, and AI-created works have been variously rejected or recognized by existing artistic institutions.
The application of robotics or primitive algorithms to art has been discussed or practiced since antiquity; however, the term primarily refers to visual works created by modern GenAI trained on preexisting content. During the AI boom of the 2020s, text-to-image models such as Midjourney, DALL-E and Stable Diffusion became widely available to the public, allowing users to quickly generate imagery with little effort.
The history of AI has raised many questions on the nature of art in human-AI collaboration, while commentary about AI art in the 2020s has often focused on issues related to copyright, deception, defamation, and its impact on more traditional artists, including technological unemployment. AI art frequently constitutes AI slop, mass-produced content for online engagement at the expense of quality.
In August 2023, the U.S. Supreme Court ruled that AI art is ineligible for copyright due to failure to meet human authorship. In March 2026, it declined to hear a case over whether AI-generated art can be subject to copyright.
Many tools are available to artists working with diffusion models. They can define both positive and negative prompts, but they are also afforded a choice in using (or omitting the use of) VAEs, LoRAs, hypernetworks, IP-adapter, and embedding/textual inversions. Artists can tweak settings like guidance scale (which balances creativity and accuracy), seed (to control randomness), and upscalers (to enhance image resolution), among others. Additional influence can be exerted during pre-inference by means of noise manipulation, while traditional post-processing techniques are frequently used post-inference. People can also train their own models.