Illusion machines with Alex Davies

Last week Alex Davies was here for a visit. After a few days in Madiera, we ended up back in Linz, looking at various ideas and collaborations, possibilities and projects. Alex has been working with illusion as a prime motivator for his work, based upon the premise that media art is, when it works, a form of illusionist conjuring. A story is being told that is, in some sense, not real, but the media attempts to make it believable.

This is, of course, a bad interpretation of his work, but I will let him explain it better. Ask him. But it helps us think about ways of building systems so that, rather than visitors concentrating on how things happen, they can think about why they happen, and imagine causality within the storyworld rather than as a technical process. If we see muddy footprints leading to a closed bathroom door and hear splashing and the sound of someone on the phone, their voice echoing in the tiled bathroom, we can imagine that the character is washing themselves and discussing why they are dirty. How the footprints are made and what device is playing the audio are essentially irrelevant.

A camera captures the visitor on the right (who sees themself) on screen, mixing pre-recorded acting on the other side of the wall to provide the illusion of presence in real time.

A camera captures the visitor on the right (who sees themself) on screen, mixing pre-recorded acting on the other side of the wall to provide the illusion of presence in real time.

In order to create illusions, it seems to help that the illusions are not only visually compelling, but things like movement and even free movement can add to this. Alex’ work on The Very Near Future, presented at ISEA13 (interview) and then Sydney Artspace in a larger version (video and interview), explored this, developing from collaborations with us including Domestic Bliss and The Black Box Sessions. So Alex used illusions such as mixing pre-recorded imagery and live imagery from a camera mounted over a wall, in order to offer the illusion that the pre-recorded action was taking place in real time. This effect was supported by voices heard over the wall and the shadows of characters on the smoked glass door separating the rooms. How can we carry on these developments? What sort of illusions can occur? Which ones are narratively useful and which ones are just decoration? Sound placement has been used extensively to create illusions or experiences of presence, technologies like sound field synthesis seem to offer higher levels of acoustic immersion. What else can be done?

One area of research that has emerged over the past years goes by the name of computational photography. This has entered mainstream awareness with the film effects in the Matrix as well as the integration of panorama stitching options into cheap point-and-shoot cameras. One other area has developed from early 20th century investigations into light fields – what a sound field is to acoustic perception, Gershun’s light fields are to visual perception.

The capacity that light fields seems to offer is that given a two dimensional collection of two dimensional images of an object, a 4D representation of a light field, a computational system can prepare a view of the object in question from any point within the surface enclosed by the collection. So this would mean that the illusion of a visitor manipulating a camera on a boom, peering over a wall and seeing pre-recorded actors as well as themself, could be extended to allow the visitor to move that camera and the pre-recorded action would remain visible as if the camera was filming them.

A mixture of geometry and software might make it possible: computational imaging for immersive illusions.

A mixture of geometry and software might make it possible: computational imaging for immersive illusions.

This seems too good to be true, so we began exploring these possibilities. Not to say we got far. It seems that in order to get interesting results, we would need to have arrays of dozens of cameras. Not to mention software to stitch these images together in realtime based upon dynamic camera movement. This is slowly becoming possible. Let’s see if it makes sense to carry it forward.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s