VR Instrument
December 2020
We leverage breath as an input for playing Virtual Reality (VR) instruments. Breath is an integral part of our life but rarely used in today’s interactive device. We explore the potential of breath input for instrument playing in VR. Specifically, our device uses a microphone to detect exhales, which are then mapped to the instrument audio in VR. As audio latency is an important factor that influences the VR instrument playing experience, we perform a technical evaluation of our current system’s latency and a Just Noticeable Difference (JND) test with three human participants. We also conduct post-session interview to understand participant’s experience in playing the VR instrument.
This is a collaboration work with Alex Mazursky, advised by Prof. Ben Zhao. I was in charge of the VR design and device communication. I implemented a music studio scene, where the user is placed into a training session and asked to practice the instrument. For the instrument selection, I leveraged the harmonica, as harmonica playing is purely based on breath input and require minimal haptic feedback. For the device communication side, I led the team to try two different protocols, BLE-OSC and BLE. We chose Bluetooth Low Energy at last for its lower latency and power consumption.