Where better in the world to introduce elegant moving screens than a country that made narrative on flat surfaces come alive, from painted screens to manga?

Magician, visualist, and technologist Marco Tempest sends this brilliant video documentation of the work he’s been doing with what he calls “Magic Projection.” The technique is simple – and extraordinarily effective. Infrared tracking points in the screen, coupled with extremely efficient vision analysis software on the computer, produce a perfectly-scaled image. Beyond that, everything is Marco’s own ingenuity. (One reason I think we all have a lot to learn from Marco is that his sense of how to do things as a magician is different from how a lot of us with arts backgrounds approach developing our techniques.)

This is, of course, markedly different from manual projection mapping, which requires that you scale your image by hand to whatever surface you’re using.

The tools are all free and open source. Our friend Zach Lieberman, a fantastically-skilled coder and originator of OpenFrameworks, worked to develop the project with OF, Intel’s free vision library OpenCV, free hardware platform Arduino, and Sony PS 3 Eye drivers MacCam. (OpenFrameworks, for those of you just joining us, is the Processing-inspired, artist-friendly C++ coding platform.)

Description from Marco:

Here is my “Magic Projection” system out on the streets in Tokyo. “Magic Projection” is my new Augmented Reality Projection Tracking system created for use in my magic stage performances. Have a look and let me know what you think.

The system works by tracking embedded infrared LED tracking markers in lightweight screens with a modified PS3 EyeToy camera and then fits projected video images onto moving screens at 120 fps.

In addition it features a virtual spotlight to light the performer while holding the screen without spilling light onto the projection surface, real-time 2D particle physics, an electronic whiteboard and a 3D function that rotates 3D objects in real time in relationship to the screen angle relative to the projector.

And yes, I was a bit lazy and didn’t link to Johnny Lee’s work, which inspired this (and is credited accordingly):

Foldable Displays (tracked with the Wiimote)

The Wiimote also works effectively; Marco is instead using the PS3 Eye, which will also work as a camera feed if that’s important. Lee’s creation plays with the idea of folding, but as you can see, the idea is familiar. (Thanks, John Holdun!)

  • Best tracking setup I've seen yet, my hat is off.

  • Johnny Chung Lee's projects are more technically complex and exciting but this is the first item I've seen a setup like this in a public environment, and for that I like it. Plus, that effect with the balls was great.

    And I love that he's using a canvas. Very functional, but it has a great metaphoric kick to it as well.

  • @John: I'm curious, what makes Johnny Chung Lee's project more sophisticated to you? (I guess the obvious difference is the folding, but beyond that?) I agree, it's also brilliant work!

  • Peter: What makes Johnny Chung Lee’s project more sophisticated to you?

    It's mostly the folding, yeah, or to be a little less specific, the fact that Lee's implementation is extremely extensible and (whether by cause or effect) more imaginative. I felt like I had seen everything Marco had up his sleeves by the end of his opening thirty-second montage while Johnny's three minute demonstration left me wishing there was more.

    Then again, it's perhaps unfair to compare the two, as one is an implementation while the other is more of a study—and as I mentioned, I do like the fact that Mr. Tempest is using this live.

  • Well, in a way, doing the mapping just creates the canvas — then you take it from there. I'll be interested to see what people might do with it. I mean, you never really run out of possibilities with a 4:3 rectangle on a wall, either. Marco's implementation is fairly new, so I expect we've only seen the beginning of it!

  • Gabriel

    I made with a fiend a similar but very very primitive system, without programing, only a camera and a proyector


  • Cort3x

    kawaai! kawai! kawaaii!

  • Amazing implementation. I would agree that this is a rehashing of the Johnny Chung Lee Foldable interactive displays, but a great creative one at that. The hard part here is going to be getting rid of that .25 second lag. Typically cameras (even in the $1000 range) and projectors are not optimized for latency, they are optimized for frame rate. This means that getting the image to stick on the canvas is going to be a difficult problem. Johnny Chung Lee's work in Hybrid Infrared and Visible Light Projection shows promise for high frame rate and low latency. Also some work at MERL.

    I would also love to know more about the virtual spotlight. Sounds like background subtraction in IR?

    Looking forward to the future in this area…

  • @Brett Jones: The virtual follow-spot uses the centroid of the board for tracking. The follow-spot moves relative to preset offset positions that I define in a cue-list playlist.xml file. I also draw a soft adjustable border to separate the follow-spot from the screen to compensate for the projector lag when moving fast.

    I am really excited about Zach's last minute 3D implementation that rotates relative to the rotation angle of the board (did not have time to explain that properly in the video – the folding chair model in the video is rendered in real time) and the 2D particle physics system that uses sound events (that I failed to record).

    There is also an integrated mask painting system (red on the video) that enables me to paint out spill from other IR light sources in environments where it is difficult to control ambient lights and reflections that might confuse the sorting logic of the tracked markers.

    The systems architecture is designed to be very quick to setup and to calibrate and to enable the performer to control all aspects of his performance. It utilizes an easy to use cue list system that provides visual feedback on the diagnostics prompter screen of what is happening and what is a about to happen next.

    Cheers, Marco

  • Pingback: Magic Projection 1.0 - AR.m-ato.me()

  • Bob Mackey

    WOW! Very interesting and a very amazing concept.

    Well done Marco!

    Wondering if you are going to make your Magic Projection software available to all of us.

  • infectedspleen

    fantastic work. As bob mackey suggested, having this software available would be great. i work in a theatre and we are working on a play about galileo's telescope. this sort of technology would really drive the play.

    but again, wow, fantastic work.

  • Pingback: The Worlds First AR Magician? | hellokinsella()

  • Pingback: Playfool® - Darren Richardson » Blog Archive » Magic Projector()

  • Mowgli

    Nice work!
    Is there any 'ready to go' software like this available somewhere or am I going to have to write it myself?

  • I need this kind of software, but it would be projected onto a person, not to a board. The person would have the necc. IR points.

    Anyone can write this kind of software?
    Got a project for ya!
    get in touch with me ASAP please: fotosafari@yahoo.com

  • simon

    Great work guys keep the tracking going,
    We have worked on similar for years, anyone can build similar with Max and jitter, there are amny tracking plugins to get or make.
    Once you have an IR reflective surface and enough ir light then you can have the video display on anything, its a bit like the green screen days.
    When you use your camera, or webcam just use an old colour negative one of the end black frames and cover the lens this acts as an IR filter and only IR light will be displayed. This can be reflected as in the folding projection or independent IR lights as in the case of projection majic.
    Any way of to project on something have fun and keep tracking

  • Pingback: Camera calibration – inimArt()

  • Pingback: Create Digital Motion » Augmented Projection with Magician Marco Tempest, Big in Japan()

  • It is a fantastic technology.

    Is there an any software available for this technology?
    If anyone knows, I will appreciate if you share it.