“Now I can feel the music not only by my ears, but also with my eyes”

ALIVE – Augmented Live music performance using Immersive Visualisation and Emotion

On 13th December 2018 I started a fabulous 3-month part-time postdoc to carry out some user tests and research into how live music performers and audience members interact with mixed / augmented reality. In addition to MR visuals, an emotion sensor was present giving feedback on participants perceived emotion. We managed to get through 2 user tests in this time. A short video taken from the second experiment here which gives an excellent account of the music, visuals and experience for participants.

To perform the music, we required what are known as SMART Instruments. These are defined as instruments which can include embedded sensors, actuators as well as wireless connectivity. The SMART instruments were connected to the visual devices by WiFi and messages sent via OSC.

The MR visuals were developed by the augmented and mixed reality developers, Fracture Reality. Participants were able to view the MR visuals through a HoloLens and the augmented reality through a mobile device. These devices allowed participants to walk around and through the visuals during performances as well as being within the same space as other participants and musicians, (unlike virtual reality that removes participants from their current space).


The emotion sensor was supplied by Sensing Feeling who provide “accurate, real-time and simultaneous emotion detection of many users undertaking their normal and natural behaviour in physical spaces.” The sensor monitors facial expressions and the number of individuals in its range.

Test 1

In the first test the SMART instrument used was a laptop. This was used to trigger samples which were held in the apps on the HoloLens and Mobile Apps. Visuals in this case was a large sphere of a number of smaller spheres which were manipulated by the audio signal. The samples used were that of a handpan drum, 8 notes in the key of F minor.

The room set up was as shown below:

You can see that the emotion sensor was at the front of the projection screen showing the emoticon representation of the participants. The emoticons had 2 states representing individual participants which were a happy face and a neutral face.

Participants were asked to identify themselves as either audience members or performers (although no participants performed in this experiment) and the test lasted approximately 15 minutes, with 2 participants at a time. An introduction for the mobile device and HoloLens was given and each participant used both devices during the test. Once completed, participants were asked to complete a questionnaire consisting of a number of free text questions and rating their agreement to statements on a 7-point Likert scale.

Main conclusions from the first test:

  • Everyone enjoyed the test
  • The HoloLens field of view is restrictive but generally everyone preferred this device
  • The majority of people ignored the emoticons
  • The majority of people did not like the idea of a sensor reading and displaying their perceived emotion
  • the majority of participants felt more connected to the music than usual, especially performers
  • For audience members, it is hypothesised that those who perceived the musical expression more clearly were less distracted by the novel technology and saw the visuals as a complement to the music while others may have been distracted by the visuals.
  • lack of expression from the laptop instrument was also felt to constrain the performance (“the instrument does not allow much expressiveness”)

  • Some participants wished to play with the visuals themselves, to reach out and touch them and hear their interactions represented in some sonic form
  • One participant stated “I really like that I am in the same room and see the people that are inside of it rather than being catapulted into a whole different dimension like VR”

  • Some participants were distracted from listening to the music by the visuals
  • Performers did not want to have their current emotional state projected on the screen and would rather the emotion of the music be expressed. In fairness, on numerous occasions when I have performed my current emotional state is generally stressed that I will remember the words, chords and all the gear works!
  • Some felt that the visuals decreased the separation between the performer and the audience members
  • One stated that they would like to conduct the visuals, musically and emotionally

Word Clouds from both audience and performer participants are shown below:

Audience Participants
Performer Participants

Test 2

About 6 weeks following the completion of the first test, we organised a second one. This time we implemented a number of changes based on the feedback given from participants in the first test. One of the main complaints was that the field of view of the HoloLens was too small. This was out of our control but the next generation HoloLens has addressed this issue.

The room setup for test 2 is given below:

One of the main differences with the second test is how the music was developed and presented. We decided to keep the handpan drum samples but used a midi controller to trigger these, giving a range in volume from 0 – 127 and hence more expressive than the laptop triggering the samples.

We wanted participants to perform 2 pieces of music, one of high energy and one of low energy but realised that it was asking a lot of performers just to drop them into a live gig with very little practice and expect them to create these on an instrument they had never played before. We also wanted a degree of similarity in the music across performers so that we have a uniform experience for the audience members to research their responses.

To solve this, I wrote two pieces of interactive music; one of high tempo and energy and one of low tempo and calmer. For each piece I recorded 4 individual loops for a drum track and 7 loops for a chord track. Therefore, the drum track had individual loops A, B, C & D with different patterns and the chord track had loops representing chords I, II, III, IV, V, VI & VII in the key of F minor (matching the key of the handpan drum samples). The drum and chord tracks both had a separate gain, controlled by sliders.

All tracks were held in Ableton Live and pure data used to communicate the current chord loop to a GUI, which also represented the handpan drum and which note was currently being played on it. The chords and drum patterns were all one bar in length and could only change once the bar had completed.

This meant that the music required two performers – one performer controlling the drum and chord patterns on midi launchpad while the handpan drum was controlled from a separate midi controller. Although a performer could control which drum and chord pattern played and the volume of each, the structure of both tracks was semi-defined to ensure some uniformity across a number of tests. The handpan drum performances were improvisation along with the other tracks.

A further improvement was in the mixed reality visuals. In test one there was one visual representing the handpan drum and in the second experiment this was increased to three separate visual components, one for the handpan drum, one for the drum track and one for the chord pattern. One of the visual components was mapped to two physical pillars within the performance space.

We provided the performers with a VJ controller, giving them control over aspects of the visuals, like the size, so that they could adjust this during a performance. The feedback from the emotion sensor was changed from individual emoticons representing everyone present in the room to an ambient colour based on an aggregate reading from everyone in the space. The experiment was also setup to be more ‘gig’ like, where the music was output over a PA system, with food and drink provided.

Like the first experiment, participants were invited to give feedback via a questionnaire. Results from the second experiment are yet to be published and I am currently looking for a suitable journal. It can be drawn from the results that everyone enjoyed the experience and would like to see more of this technology used in live performances. One quote from a participant was:

if it could be adapted for all live performances – theatre, music, concerts etc – added value could be huge”

Full results for the second experiment will be published soon…..


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s