Graphic designer Superdeux came to Secret Feature with an interactive project to be featured as part of his gallery show “I Wish I Could Talk” at The Summit SF. Shapes would be projected on a wall and users can interact with the shapes by moving in front of an XBox Kinect. Fun and intuitive. During the show’s opening reception the space in front of the projection would be clear and people would be free to play. During the exhibition interaction would be limited to people passing in front of the projection, colliding with the shapes.
ActionScript to C++
I have been a Flash developer since the early days and my programming skills have grown with the platform. I was hungry to try out a new language and this seemed like a great opportunity. Fascinated by the pieces Robert Hodgin had been creating in Cinder, I installed Xcode and began working through the Cinder tutorials and examples.
I spent a couple weeks haunting the Cinder Forums, watching videos, playing with example code, and reading a C++ book. I looked into 2D physics libraries in C++ and Box2D seemed like a good start since I had used the ActionScript version and Cinder Forum user Sansumbrella had worked out the integration with Cinder. Superdeux’s shapes needed to be circular and textured so I created a fork of Sansumbrella’s Box2D Sandbox and added circular and textured elements based off the existing BoxElement class-
Kinect, OpenNI and OpenCV
Next I had to figure out the Kinect interaction. I was really impressed with the work people were doing using OpenNI, which matches body motion accurately using skeleton detection, but it requires users to pose with their arms in the air while it calibrates. This won’t work for the casual interaction we are looking for. After more digging around in the Cinder forum I found an example of hand tracking using the Kinect by Andy Berg-
This method uses the OpenCV library. Andy’s example is for a single user but it seemed possible to reuse as a multiuser/multipoint control system. I bridged the Kinect/Box2D worlds by dynamically adding and removing invisible Box2D elements that would be pulled toward the Kinect touch points, colliding with the onscreen shapes. This would allow multiple people to use the app and it would also provide the effect of collision as Summit customers walked by the projection. A debug shot here shows the control points in turquoise-
At The Summit
With a couple days to spare, I added a feature to take snapshots of people in front of the kinect along with a screenshot.
This being my first C++ app, was very rewarding. I am more than happy with the performance, the interaction is good and Superdeux’s graphics make it fun to watch. I’m excited to start my next Cinder project. Please go see Superdeux’s exhibit this month at Summit if you are in San Francisco. The interactive piece is on display in the evening.
Jeremy Brautman has a nice review of the exhibit- http://www.jeremyriad.com/blog/events/superdeux-at-the-summit/
App and Source
App (OS X)- http://secretfeature.com/apps/secretfeature-and-superdeux-app.zip
My fork of the Box2D Cinder wrapper- https://github.com/secretfeature/sc-Box2D