The Internet of Musical Things (also known as IoMusT) is a research area that aims to bring Internet of Things connectivity to musical and artistic practices. Moreover, it encompasses concepts coming from music computing, ubiquitous music, human-computer interaction, artificial intelligence, augmented reality, virtual reality, gaming, participative art, and new interfaces for musical expression. From a computational perspective, IoMusT refers to local or remote networks embedded with devices capable of generating and/or playing musical content.
Introduction The term "Internet of Things" (IoT) is extensible to any everyday object connected to the internet, having its capabilities increased by exchanging information with other elements present in the network to achieve a common goal. Thanks to the technological advances that have occurred in the last decades, its use has spread to several areas of performance, assisting in medical analysis, traffic control and home security. When its concepts meet music, the Internet of Music Things (IoMusT) emerges.
The term "Internet of Musical Things" also receives numerous classifications, according to the use of certain authors. Hazzard et al., for example, uses it in the context of musical instruments that have QR code that directs the user to a page with information about this instrument, such as manufacturing date and history. Keller and Lazzarini, use this term in ubiquitous music (ubimus) research, while Turchet et al. define IoMusT as a subfield of the Internet of Things, where interoperable devices can connect to each other, aiding the interaction between musicians and the audience.
Like the IoT, the Internet of Music Things can encompass a variety of ecosystems. But generally, it is marked by being employed in musical activities (rehearsals, concerts, recordings, and music teaching) and relying on service and information providers.
The field has been described as enabling new services and applications for the music industry that make use of the interconnection between logical and physical devices.
Musical things A musical thing is formally defined as a "computational device capable of acquiring, processing, acting, or exchanging data that serves a musical purpose." In short, these objects are entities that can be used for musical practice, can be connected in local and/or remote networks, and act as senders or receivers of messages. They can be, for example, a smart instrument (instruments that use sensors, actuators and wireless connection for audio processing), wearable devices or any other capable of controlling, generating or executing musical content over the network.
Unlike traditional audio devices, such as microphones and speakers, musical things are not useful by themselves, thus the need arises to insert them into a chain of equipment. Thus, the need arises to think about standards, protocols and means of communication between them. These challenges will be analyzed below.
The challenges of creating musical things The first challenge concerns the hardware used in musical things. First, one should keep in mind that these devices are not analog. Because of this, they can be reprogrammed and must have internet connectivity and/or another possibility to communicate with other equipment. Secondly, they are not traditional computing devices. This means that they are programmed for a general purpose, not only to perform certain tasks, as is the case with smartphones and personal computers. Finally, they will be employed in an artistic and musical context. In this way, the aesthetic characteristics are as important as the computational ones.
That said, the hardware challenges are clear, and these include the processing capacity, as well as the storage and power consumption of musical things, which must be good enough to withstand artistic performances, while not making these objects expensive or unergonomic and unwieldy. In addition, they should be able to take on different roles in different scenarios. Thus, they should allow users to add or remove components (such as sensors and actuators) aiming to be adaptable, expressive and versatile.
The second challenge deals with the behavior of musical things. They must primarily exchange sound data, but it is desirable that they also exchange control data and processing parameters. In this sense, they must adapt their operating mode in order to be able to cooperate with the other elements present in the network, and also have their software and operating logics updated remotely.
The third adversity is possibly the most sensitive and most difficult topic to deal with. You have to think about what data is possible to share and how to do it. For audio formats, one can think of Pulse Code Moduation (PCM) formats, like WAV, because it is the most common in real-time audio processing systems. However, issues such as latency and quality are not guaranteed. Files in MP3, FLAC or OGG format, on the other hand, require more processing and the latency arising from this can make the environment impractical.
Possible solutions for creating musical things Possible solutions to these problems include the use of common IoT elements in music practice or the assignment of networking capabilities on behalf of traditional audio objects. Effects units (such as guitar pedals) should be built so that the user is free to remove or insert buttons and sensors, while in logic units the software is modifiable. Audio equipment should send and receive data over the network, and also be remotely controlled. This can be useful for adapting these elements to the different types of data circulating on the network.
The musical instruments, on the other hand, will function similarly to smart musical instruments, where they will be equipped with sensors and actuators capable of capturing stimuli from the environment and from the musicians themselves. Musical aids such as metronomes and tuners can be transposed to digital media, while performance aids such as light and smoke cannons can be controlled and synchronized over the network.
However, IoMusT is not only about making adaptations of what already exists, but also by creating devices, capable of generating new perspectives for musical practices.
Related fields This section reviews some of the various application domains that aid an IoMusT environment. The review is not intended to be exhaustive, aiming to describe the main features and functionalities of each area.
Network musical performance A networked performance is a real-time interaction via machine, which allows artists dispersed across the globe to interact with each other as if they were in the same environment. While not intended to replace the traditional model, it contributes to music creation and its social interactions, promoting creativity and the exchange of cultures.
Among its main characteristics are: low latency, where the sounds produced should be heard almost instantaneously; synchronization, to prevent long delays from hindering interaction in the environment; interoperability through standardization, which allows different devices to communicate over the network; scalability, which makes the system comprehensive and allows distributed participation among users; and easy integration and participation, aspects that ensure that users have no difficulty in finding devices on the network, and can connect or disconnect from it whenever they want. As for the challenges in this area, they can be illustrated by the requirement for high bandwidth and ordering in the transmitted stream, and sensitivity to delay in the delivery of data packets.
Interactive art
Art has always had its interaction marked by the relationship between the artist and the medium he uses to materialize the work, while the audience had only the role of passively observing everything. This began to change when artistic movements led by Allan Kaprow and the Fluxus and Gutai groups began to allow for more active audience participation. In this context, interactive art emerged, characterized by allowing the viewer a degree of active involvement in the show, either by walking among the installations and sculptures, or by producing sounds, , and movements.
The architecture of these environments is designed to handle different types of signals, ranging from audio and video to those produced by the human body, such as heartbeats. As such, they also require functionality that ensures interoperability and handles data delivery delays.
Ubiquitous music Ubiquitous music, usually abbreviated to ubimus, is a research field that combines music, technology, and creative processes with strong social and community engagement. Although its original proposal is focused on music production, current technological developments have opened new social and cognitive dimensions to this field, leading it to become increasingly interested in educational and artistic topics. Thus, current perspectives encompass a wide diversity of subjects and actors, ranging from casual participants to highly trained musicians.
The ubimus ecosystem supports the integration of audio tools and audience interaction, and can be reconfigured to meet the needs of users. Consequently, the desired concepts are not dependent on specific implementations. Other important features are conceptual approaches and reliance on empirical methods. These aspects encourage the development of technologies for music creation, especially those that make use of common objects and spaces in the daily lives of those involved in the process.
Web Audio, cloud computing and edge computing
Web Audio is a JavaScript API for audio processing and synthesizing in web applications, representing a technological evolution in this segment. It presents some features common to DAWs, such as audio signal routing, low latency, and effects application. It also allows participative networked performances and expands the capabilities of using smartphones in these media.
Its environment uses audio nodes for manipulating sound in a musical context. They are connected by their inputs and outputs to create paths for routing audio, which is modified by effect nodes along the way. In this way, it is able to support numerous sources with different layouts, as well as being flexible and creating complex functions with dynamic effects.
Web Audio paves the way for using web browsers for musical purposes. Among the advantages observed from this are easy distribution (no installation required) and maintenance, platform and architecture independence, security (the browser can prevent plugins with incorrect behavior from affecting the system), and emergence of new types of collaboration.
Cloud computing, on the other hand, is a structure composed of distributed servers that simulate a centralized network, allowing load balancing and resource replication, minimizing the amount of network consumption and improving its scalability and reliability. It aims to provide numerous services, ranging from file storage to intercommunication between music applications, offering an unprecedented level of participation and performance.
Its main feature is to allow users to access the services without the need for knowledge about the technology used. Thus, they can access them on demand and regardless of location. Other points worth highlighting in this network are: broad access, elasticity, and resource management.