cancel
Showing results for 
Search instead for 
Did you mean: 

Pensato - A VR interface for music performance

Mystfit
Protege
Hi everyone, I wanted to share what I've been working on for my Masters degree over the last year; a VR interface that lets you perform music by interacting with components representing elements from music software or devices. This video is a cut together version of a performance I've made showcasing some of what the interface can do.




Technology rundown:

Gloves:
The input is controlled using dissected Razer Hydra game controllers mounted on the back of the gloves, paired with two Teensy microcontrollers equipped with bend sensors for detecting hand gestures.

Environment:
This performance features a master version of Pensato running in Unity on the computer monitor to my right, which creates the visuals for the Rift and controls Ableton Live. The projector computer runs another copy of Pensato that listens to the position and orientation of the gloves, as well as the hand gesture and displays and displays a copy of my VR interface on the 3 projectors.

Sound:
The laptop runs Ableton Live 9 and is connected to each copy of Pensato using Showtime, a framework that I developed for connecting together realtime performance software and hardware. The glove data is transmitted over the same system.

This work is the final compositional output for my Masters of Design Innovation: Computer Graphics at Victoria University of Wellington. All audio was captured from the performance and used to create the edited version of the video.
3 REPLIES 3

aerosteak
Protege
:shock:

LOVE IT!!! I would love to have the same control over my rig in my studio!

IsoMacintosh
Explorer
I dont know how to make music, but add support for perception neuron and i will throw money at you. 😄
BTW: Did you start the kick at wrong time? Sounded a bit weird.

Mystfit
Protege
"IsoMacintosh" wrote:
I dont know how to make music, but add support for perception neuron and i will throw money at you. 😄
BTW: Did you start the kick at wrong time? Sounded a bit weird.

I'm planning on moving to ControlVR as my input method in the future, but I can take a look at perception neuron as well. I'm hoping that I can make the input system generic enough that it will work with a variety of tracking hardware whether they track down to the finger level or not, so you could use a STEM and hit buttons in order to trigger the intended gestures.
Also, I started the kick on an offbeat in the middle of the arpeggiator to add a bit of variety. Since the kick goes off every beat, it quickly evens out but can sound a bit off at the start.