3d Sensing and Visualization: Holographic Warpaint

Last spring I worked on a play. It was an adaptation of Samuel Delany's epic science fiction novel, Dhalgren. This book is crazy, and the play was crazy - I did the sound design and worked on the video as well. This is the kind of book that sticks with you (the wild dense prose, the imagery, the....extremely detailed pornographic sex), and there were many things from the book that weren't realized in the play and I'm holding onto them. The book is set here - well, in a city somewhere in America - after an unnamed disaster has taken place. The city is a wasteland, but people are still living there. They live for free in parks, or squat in apartments where nothing works.

Gangs, known as the Scorpions, run the streets. This is the element I'm thinking about. Members of the Scorpions wear projector necklaces. When they press a button on the projector a holographic animal surrounds their bodies. Like holographic warpaint. One of the characters is known as Dragon Lady, because her projection is a dragon. One of them is a baby dinosaur - which I love. One of them doesn't work correctly and appears as an amorphous blob. I think it's weird that I can't find an image of this somewhere. I feel like it's one of the most memorable images from the book - gangs of fierce, oversized, holographic animals walking through the streets.

So, I made a failed attempt at this last semester in ICM using color tracking with lame colored LEDs strapped to my body. When the Kinect came out, I knew it was a solution, which is why I'm in this class.

I had previously envisioned a solid, neon colored animal shape for these shields, and thought of using skeleton tracking with OpenNI to animate a 3D character. I was nervous about the animated character, though, and pretty sure it would look dumb.

A simple, and I think effective, solution occurred to me late in the game. I reimagined the design of the holograms - they could be skinned as the creatures rather than shaped like them. I modified an example from class to remove background information, then map pixels from existing images to the depth image from the Kinect. I projected this onto two layers of mesh that I stood behind, producing a faux 3D projection effect. I tried a couple images - two dinosaurs and a lizard.

Here's a diagram of my setup: