At the bleeding edge of advanced projection mapping, new techniques are aligning the virtual world ever closer to the real one. And whereas VR technology is all about putting your head inside a virtual world, mapping processes allow the world around us to become more fluid as a canvas.
If you’re friends on Facebook with anyone who cares about media art, you probably already saw this video making the rounds the past few days. Not only is it aesthetically impressive, but it also reveals some of the direction people are going with custom-programmed, mapped visuals.
INORI (prayer) is a massive collaboration, bringing together choreographer Aya Soto, the visual design studio WOW inc. and its creative and technical director Nobumichi Asai, and film producers for TOKYO.
It also represents the merging of two significant bodies of research into visual effects. The Ishikawa Watanabe Laboratory of The University of Tokyo developed the dynamic projection mapping technology, including a 1000 fps projection system called DynaFlash. (The projection system itself is made available commercially by Tokyo Electron Device Limited – though it’s just one ingredient here.) Then, there’s the face mapping technology, involving software developed in-house by WOW.
The trick here is speed: high-speed sensing of faces (WOW) gets mapped to high-speed projection (DynaFlash) which is mapped dynamically onto the face (Watanabe Laboratory). Latency matters, too — here kept to 10ms maximum.
There are actually even two tracking systems – hands from Watanabe and faces from WOW. Then, the team had to bring all these elements together.
And that’s really always been the challenge in dance technology. Dance is beautiful as a medium precisely because our perception and emotions are so closely tied to every nuance of movement in time. Tossing some 30 fps imagery on top of that is problematic.
Of course, it matters just as much that the dance and dancers here are so expressive. (The narration in the making-of video is quick to point out that this isn’t just a demo – and I think you’re right, smug narration voice, I agree.)
This is truly an epic effort, I think not only as a showcase for technology, but an impressive demonstration of artists and coders and different research projects working in close harmony. And so even if you don’t have this kind of technology at your disposal, the model in collaboration might be just as meaningful.
The studios involved:
(One note – there are confusingly three different things called TOKYO here. There’s TOKYO, the creative team – link below. There’s TOKYO ELECTRON DEVICE LIMITED, the folks marketing the high-speed projector. And then there’s the University of Tokyo. If that weren’t confusing enough, the lab at the University of Tokyo then made the projection system marketed by Tokyo Electron Device Limited who then worked with TOKYO. In Tokyo. Tokyo Tokyo Tokyo, Tokyo? Okay, enough.)
WOW: wow.co.jp
TOKYO: http://lab.tokyo.jp/
Details:
INORI (prayer)
Super High Speed Face Mapping (1000fps)
Making
vimeo.com/210565827
This project is achieved as collaborative work between AYABAMBI, Ishikawa Watanabe Lab, The University of Tokyo,TOKYO, responding the call for collaboration by Nobumichi Asai (WOW).
Staff
Nobumichi Asai [Creative Director | Technical Director – WOW]
Shingo Abe [CG Director – WOW]
Atsushi Yoshimura [Programmer – WOW]
Ayaka Motoyoshi [Assistant Producer – WOW]
Eiji Tanigawa [Director | Editor – TOKYO]
Senzo Ueno [Director of Photography – TOKYO]
Toshiyuki Takei / Shinya Masuda [Producer – TOKYO]
Minami Chiwaki / Yuma Yoshimura / Kohei Takayama [Production Manager – TOKYO]
Suzuko Ohgaki [Making Director – TOKYO]
Aya Sato [Choreographer]
AyaBambi [Cast]
Oi-chan[Management]
Shootings
Tomorio Takahashi [Gaffer]
Hisashi Morikawa / Mie Inaba / Akihiko Imai [Light Assistant]
Takashi Ideguchi [Camera 1st Assistant]
Shibuya Hiromi / Fujii Ryosuke [Camera 2nd Assistant]
Setsu Fukushima / Ryosuke Taniguchi [Music – Ongakushitsu Inc.]
Yosuke Nagao [Music Composer]
Yasuo Fukuda [Colorist]
Ryota Abe [Online Editor]
Mizuki Kawano [Mixer – TAIYO KIKAKU co.,ltd]
Special Thanks
Yoshihiro Watanabe [Ishikawa Watanabe Laboratory,University of Tokyo]
Masatoshi Ishikawa [Ishikawa Watanabe Laboratory,University of Tokyo]
Tomoaki Teshima [Exvision]
TOKYO ELECTRON DEVICE LIMITED
Takeshi Yuasa / Kiwamu Sumino / Hiroshi Watase /Toru Yamashita
Atsuko Kushima / Tomoaki Kiguchi
nobumichiasai.com
– – – – – –
First, the soundtrack was composed according along the theme “Life”. Nobumichi Asai and Shingo Abe was inspired by the music and then made face mapping work. Aya Sato designed choreography. TOKYO completed the project by making the video work.
The music brought Asai the image of “the radioactive.” The destructive force of the radioactive could cause “death,” “suffering,” and “sorrow.” And “prayers” could overcome them. These subjects infuse AYABAMBI’s powerful performance. Their performance crushes and conquer black tears, skulls and the Heart Sutra. We built up the concept during the production and Abe designed animation for face mapping.
There was a big challenge in technical aspects. We realized the new mapping system that allows us to follow intense performances by using the latest 1,000 fps projector, DynaFlash(*1) and a super speed sensor. It is very new and it had not been done before. Projected images become part of their skin and they transform their faces.
At the beginning of our development, there was an issue, AYABAMBI would loose freedom of performances if we accelerated the tracking speed. Asai, Yoshimura (WOW), Lecturer Watanabe and Teshima (EXVISION) explored how we could keep the tracking speed securing the freedom of performances, taking three months of trial and error to reduce a few milliseconds. And we finally developed this system(*2).
*1 “DynaFlash” is developed jointly by Ishikawa Watanabe Laboratory, the University of Tokyo and is commercialized by Tokyo Electron Device Limited.
*2 For tracking hands, we use the dynamic projection mapping technology developed by Ishikawa Watanabe Laboratory, the University of Tokyo. For tracking faces, we use the face mapping technology developed by the visual design studio, WOW inc.
You get a feel for the underlying tech via some other examples. Here’s a recent work by WOW, which reveals not only how fast the mapping can be, but just how detailed, as well:
And you may have seen this beautiful work they did with Lady Gaga for the GRAMMY Awards:
Looking at the other side of this, here’s a recent demonstration of how the Ishikawa Watanabe Laboratory is developing mapping, with the sexy title “Dynamic projection mapping onto deforming non-rigid surface” (hey, they’re academics):
Of course, to be honest, all this makes me want to return to just not mapping anything and playing with projection on surfaces organically – because if you can’t beat them, do the opposite. (And yeah, I did that in another life, back when we were using, like, DV camcorders!)
But looking the other direction, it’ll be interesting to watch whether these kinds of advanced techniques remain a specialist process, or whether they will find their way into more common usage. The message for movement and dance is clear, though: speed is as much the frontier as (pixel) resolution.
Also from Japan — an effort a couple years back (though notice that here, what was lacking was speed):