Vocaloid is a singing voice synthesizer software product. Its signal processing part was developed through a joint research project between Yamaha Corporation and the Music Technology Group at Pompeu Fabra University, Barcelona. The software was ultimately developed into the commercial product "Vocaloid" that was released in 2004.
The software enables users to synthesize "singing" by typing in lyrics and melody and also "speech" by typing in the script of the required words. It uses synthesizing technology with specially recorded vocals of voice actors or singers. To create a song, the user must input the melody and lyrics. A piano roll type interface is used to input the melody and the lyrics can be entered on each note. The software can change the stress of the pronunciations, add effects such as vibrato, or change the dynamics and tone of the voice.
Various voice banks have been released for use with the Vocaloid synthesizer technology. Each is sold as "a singer in a box" designed to act as a replacement for an actual singer. As such, they are often released under a moe anthropomorph avatar, however, there are also voice banks released without an assigned avatar. These avatars are also referred to as Vocaloids, and are often marketed as virtual idols; some have gone on to perform at live concerts as an on-stage projection.
The software was originally only available in English starting with the first Vocaloids Leon, Lola and Miriam by Zero-G, and Japanese with Meiko and Kaito made by Yamaha and sold by Crypton Future Media. Vocaloid 3 has added support for Spanish for the Vocaloids Bruno, Clara and Maika; Chinese for Luo Tianyi, Yuezheng Ling, Xin Hua, and Yanhe; and Korean for SeeU.
The software is intended for professional musicians as well as casual computer music users. Japanese musical groups such as Livetune of Toy's Factory and Supercell of Sony Music Entertainment Japan have released their songs featuring Vocaloid as vocals. Japanese record label Exit Tunes of Quake Inc. also have released compilation albums featuring Vocaloids.
Technology
Vocaloid's technology is generally categorized into the concatenative synthesis in the frequency domain, which splices and processes the vocal fragments extracted from human singing voices, in the forms of time-frequency representation. The Vocaloid system can produce the realistic voices by adding vocal expressions like the vibrato on the score information. Initially, Vocaloid's synthesis technology was called on the release of Vocaloid in 2004, although this name is no longer used since the release of Vocaloid 2 in 2007. "Singing articulation" is explained as "vocal expressions" such as vibrato and vocal fragments necessary for singing. The Vocaloid and Vocaloid 2 synthesis engines are designed for singing, not reading text aloud, though software such as Vocaloid-flex and Voiceroid have been developed for that. They cannot naturally replicate singing expressions like hoarse voices or shouts.
System architecture
The main parts of the Vocaloid 2 system are the score editor (Vocaloid 2 editor), the singer library, and the synthesis engine. The synthesis engine receives score information from the score editor, selects appropriate samples from the singer library, and concatenates them to output synthesized voices. There is basically no difference in the score editor and the synthesis engine provided by Yamaha among different Vocaloid 2 products. If a Vocaloid 2 product is already installed, the user can enable another Vocaloid 2 product by adding its library. The system supports three languages, Japanese, Korean, and English, although other languages may be optional in the future. It works standalone (playback and export to WAV) and as a ReWire application or a Virtual Studio Technology instrument (VSTi) accessible from a digital audio workstation (DAW).
Score Editor
The score editor is a piano roll-style editor to input notes, lyrics, and some expressions. When entering lyrics, the editor automatically converts them into Vocaloid phonetic symbols using the built-in pronunciation dictionary. The user can directly edit the phonetic symbols of unregistered words. The score editor offers various parameters to add expressions to singing voices. The user is supposed to optimize these parameters that best fit the synthesized tune when creating voices. This editor supports ReWire and can be synchronized with DAW. Real-time "playback" of songs with predefined lyrics using a MIDI keyboard is also supported.
Singer library Each Vocaloid license develops the singer library, or a database of vocal fragments sampled from real people. The database must have all possible combinations of phonemes of the target language, including diphones (a chain of two different phonemes) and sustained vowels, as well as polyphones with more than two phonemes if necessary. For example, the voice corresponding to the word "sing" ([sIN]) can be synthesized by concatenating the sequence of diphones "#-s, s-I, I-N, N-#" (# indicating a voiceless phoneme) with the sustained vowel ī. The Vocaloid system changes the pitch of these fragments so that it fits the melody. In order to get more natural sounds, three or four different pitch ranges are required to be stored into the library. Japanese requires 500 diphones per pitch, whereas English requires 2,500. Japanese has fewer diphones because it has fewer phonemes and most syllabic sounds are open syllables ending in a vowel. In Japanese, there are three patterns of diphones containing a consonant: voiceless-consonant, vowel-consonant, and consonant-vowel. On the other hand, English has many closed syllables ending in a consonant, and consonant-consonant and consonant-voiceless diphones as well. Thus, more diphones need to be recorded into an English library than into a Japanese one. Due to this linguistic difference, a Japanese library is not suitable for singing in eloquent English.
Synthesis engine
The synthesis engine receives score information contained in dedicated MIDI messages called Vocaloid MIDI sent by the score editor, adjusts pitch and timbre of the selected samples in frequency domain, and splices them to synthesize singing voices. When Vocaloid runs as VSTi accessible from DAW, the bundled VST plug-in bypasses the score editor and directly sends these messages to the synthesis engine.
Pitch conversion Since the samples are recorded in different pitches, pitch conversion is required when concatenating the samples. The engine calculates a desired pitch from the notes, attack time, and vibrato parameters, and then selects the necessary samples from the library.
Timing adjustment In singing voices, the consonant onset of a syllable is uttered before the vowel onset is uttered. The starting position of a note ("note-on") must be the same as that of the vowel onset, not the start of the syllable. Vocaloid keeps the "synthesized score" in memory to adjust sample timing so that the vowel onset should be strictly on the "note-on" position. No timing adjustment would result in delay.
Sample concatenation
When concatenating the processed samples, discontinuities are reduced by spreading the phase between samples via phase correction and estimating spectral shape using a source–filter model called the excitation plus resonances model.
Timbre manipulation The engine smooths the timbre around the junction of the samples. The timbre of a sustained vowel is generated by interpolating spectral envelopes of the surrounding samples. For example, when concatenating a sequence of diphones "s-e, e, e-t" of the English word "set", the spectral envelope of a sustained ē at each frame is generated by interpolating ē in the end of "s-e" and ē in the beginning of "e-t".
Transforms After pitch conversion and timbre manipulation, the engine does transforms such as inverse fast Fourier transforms to output synthesized voices.
Software history
Vocaloid
Yamaha started development of Vocaloid in March 2000 and announced it for the first time at the German fair Musikmesse on March 5–9, 2003. It was created under the name "Daisy", in reference to the song "Daisy Bell", but for copyright reasons this name was dropped in favor of "Vocaloid".
Vocaloid 2
Vocaloid 2 was announced in 2007. Unlike the first engine, Vocaloid 2 based its results on vocal samples, rather than analysis of the human voice. The synthesis engine and the user interface were completely revamped, with Japanese Vocaloids possessing a Japanese interface.
Vocaloid 3
Vocaloid 3 launched on October 21, 2011, along with several products in Japanese, the first of its kind. Several studios updated their Vocaloid 2 products for use with the new engine with improved voice samples.
Vocaloid 4
In October 2014, the first product confirmed for the Vocaloid 4 engine was the English vocal Ruby, whose release was delayed so she could be released on the newer engine. In 2015, several V4 versions of Vocaloids were released. The Vocaloid 5 engine was then announced soon afterwards.
Vocaloid 5
Vocaloid 5 was released on July 12, 2018, with an overhauled user interface and substantial engine improvements. The product is only available as a bundle; the standard version includes four voices and the premium version includes eight. This is the first time since Vocaloid 2 that a Vocaloid engine has been sold with vocals, as they were previously sold separately starting with Vocaloid 3.
Vocaloid 6
Vocaloid 6 was released on October 13, 2022, with support for previous voices from Vocaloid 3 and later, and a new line of Vocaloid voices on their own engine within Vocaloid 6 known as Vocaloid:AI. The product is only sold as a bundle, and the standard version includes the 4 voices included with Vocaloid 5, as well as 4 new voices from the Vocaloid:AI line. Vocaloid 6's AI voicebanks support English and Japanese by default, though Yamaha announced they intended to add support for Chinese. Vocaloid 6 also includes a feature where a user can import audio of themselves singing and have Vocaloid:AI recreate that audio with one of its vocals.
Derivative products Software
Yamaha developed Vocaloid-flex, a software application which contained a speech synthesizer. According to the official announcement, users could edit its phonological system more delicately than those of other Vocaloid series to get closer to the actual speech-language; for example, it enabled final devoicing, unvoicing vowel sounds, or weakening/strengthening consonant sounds. It was used in a video game Metal Gear Solid: Peace Walker released on April 28, 2010. It was mainly a corporate product with a consumer version never seeing a full release. This software was also used for the robot model HRP-4C at CEATEC Japan 2009. Gachapoid was the only commercially released software that had access to this engine under the name of V-Talk. Users could use the software for six months, free of charge from the date of installation before its retirement on February 13, 2015.
Another Vocaloid tool that was developed was VocaListener, a software package that allows for realistic Vocaloid songs to be produced by analyzing an audio recording of a singing performance (a cappella) and imitating it to generate Vocaloid singing parameters automatically.
To aid in the production of 3D Vocaloid animations, the program MikuMikuDance was developed. This freeware allowed a boom in the birth of fan-made and derivative characters, as well as a boost in the promotions of Vocaloid songs. MikuMikuDance's developer went on a hiatus in May 2011 (initially announced as a retirement from development), but started updating the software again in June 2013.