Jaron Lanier Interview: Conversations with History; Institute of International Studies, UC Berkeley

Culture and Technology: Conversation with Jaron Lanier, computer scientist and artist, October 3, 2005, by Harry Kreisler

Page 2 of 6

Technology

In your work you were a pioneer in what is called virtual reality. Talk about how you go into that, because in the body of your work there is a real interest in the development of technology, but also an effort to relate it to human communication.

My initial motivation for virtual reality is a little different from what's come to be my primary motivation over the years. My initial motivation had to do with trying to come up with a vernacular access path to mathematics.

My feeling is that mathematics is a beautiful thing, it's a beautiful set of ideas that most people are excluded from and feel terrified of. And my guess as to why there is that gap, that failure of communication, is that the language of mathematics is almost cruel in its obscurity, and there's a little bit of a deliberate history to that. Academics have often tried to exclude the public from what they do, but in particular with mathematics, because it had to be written down by pencil, a lot of complicated ideas had to be compressed into little symbols you could write on a page. Now, with computer graphics and all, [I felt that] maybe we could animate and turn these things into some other expression. As I was trying to do this on conventional computer screens I ran into the problem of running out at the end of the screen, so I wanted to be able to be immersed inside it.

The idea of being immersed in a virtual world dates back well before my time to Ivan Sutherland's work in the sixties. What I did that was an extension of Ivan's work was to take the virtual world and put the human body into it by measuring the hands and the whole body so that you were physically in there, but then more importantly perhaps, create a social environment where multiple people were in the virtual world and could look at each other, and there would be a version of you -- this is the idea of the avatar --and then you would become a sort of a puppet of yourself.

But the interesting thing is that there is a form of expression which still needs to be explored much further, where you could turn into an idea, so for instance, your body could become a triangle and you could become almost like a dance expressing trigonometry. This route into mathematics, I believe, will someday be as common as singing a tune or just speaking. I believe this will be the future of the expression of mathematics, but it'll take quite a while to formulate. So, this was my initial interest.

You coined the term "virtual reality." You were the first one to build an avatar. Do you say "build?"

The first actual avatar design was done by Ann Lasko in the lab. All of these things are actually team efforts, and one shouldn't credit me without mentioning a whole slew of other people, Tom Zimmerman and many others. So, it's a bit tricky, and I often end up writing letters to journals saying, "You can't just credit me. You must credit all these other people." But at any rate, yeah, I did make up the term "virtual reality," which I don't think is a great term but it seems to have resonance and it's stuck around.

You were young when this happened, and you were very idealistic and hopeful. In what ways did virtual reality live up, or not live up, to your expectations?

We're just entering the era in which we'll start to see what happens with it, because from the very beginning we knew it would take this long for the technology components to catch up. If you think of virtual reality as an industrial technology, it's easily met all expectations we could've had. You cannot drive a car that wasn't designed in virtual reality. If you have surgery, your surgical procedure was probably improved in a virtual reality simulation. There are a whole host of ways in which virtual reality is behind the scenes doing good work these days, which we're extremely pleased with.

The idea of virtual reality as something that people have direct contact with is another matter, and right now perhaps the closest thing we have is the video game market, which for the most part I find very disappointing. It's mostly not very creative, it's mostly a slightly interactive version of a movie with very, very, very repetitive and mundane gaming principles and very little opportunity for creativity. But I'm still hopeful, I still have my idealism.

What I'm hoping will happen -- and this is my crazy idealism, this might be where I'm a nut case, I don't know, but if I'm a nut case I choose it, I celebrate my nuttiness and I hope others might join me in it -- what I believe is that by having access to good quality virtual reality where you can be creative, where you can make up the world, we're going to start to see an emergent new form of language, which I call "post-symbolic communication," which means a way to share experience by directly making it up instead of talking about it with symbols. The reason I like that is because I think there will be a kind of intimacy to it that will be charming and that will extend some of the energy of childhood into the adult experience. I think it'll be a beautiful extension of culture. So, that's what I'm really going for; that's my new interest in virtual reality.

We can't cover all that you've done, and we want to talk about the music, but the next big project was tele-immersion. Talk a little about that, because again, you're trying to create a world which replicates or comes very near to human experience and human interaction and communication.

This was during the nineties. I had a job as the chief scientist of the engineering lab of Internet 2. Internet 2 was the collaborative lab of most of the American research universities to study Internet-related things.

My main project was this thing called tele-immersion, which is to try to make what you might call a fully realistic avatar, or a version of a person in virtual reality that looks like the person, so that you simulate physical co-presence. There are two reasons we tried to do that. One was just for the bravura joy of it, to say, here we've pushed virtual reality to an extreme, even if realism isn't what you ultimately want, to prove you can do it. It gives you other options as well.

The other reason is the scientific agenda, which is understanding human perception. All along, from the very beginning, there have been two sides to virtual reality. On the one hand, it's a technology that you can make and then do things with, but on the other hand, it's a scientific instrument that you can use to study how people holistically perceive and understand the world. Prior to virtual reality instruments, you could measure a little thing about vision, a little thing about touch; but with virtual reality you have this whole simulation so you can study whether you really understand how the human sensorum works. This was an extension of that research which continues, which I'm very, very devoted to.

A company spun off of this that sounded very interesting, Eyematic. Is that what it's called?

Well, Eyematic is no more. One thread of research, which was one of my very favorite ones, was a collaboration with a wonderful neuroscientist at USC named [Christoph] von der Malsburg. What we were doing there is looking at mathematical models of how the visual cortex, which is located in the back of your head and is the portion of your brain tissue that immediately receives signals from your eyes, is able to represent visual objects that you see and what kind of pattern they turn into once they enter the brain. We tested a hypothesis of how that works and we were able to make some wonderful little technological toys out of it, including a magic mirror where a camera would look at your face and then on a screen you could see your instant facial expressions, smiles and frowns, and so forth, transferred to either a fantasy character, or to yourself, or to another person.

We were able to abstract the facial expression component of our perception of other people and apply it in different ways, which is one aspect of doing this trick of tele-immersion. That's one of two major collaborations I've had with neuroscientists, so in trying study the sensorum you inevitably you go a layer deeper and try to understand a little bit of how the brain connects to this world of the senses.

Does your current work continue this kind of study, the Cocodex work here?

No, the Cocodex is the new consortium of researchers I'm working with in the 2000s, and this involves yet another instrumentation strategy. Whereas before we've had either helmets or special rooms that create the virtual reality illusion, now we have a device which is sort of like a quick robot arm that holds various sensors and displays in front of your face as you move around in order to achieve results that we couldn't achieve with any of the other devices. We haven't tested this instrumentation yet, it's still being built, but we believe this instrumentation will get us one round closer to being able to test even more sophisticated hypotheses of how people perceive each other. From a scientific point of view, this is the agenda of virtual reality, to understand human perception holistically and in particular of other people.

One of the things I often tell people about this research is that if you go back a century, almost exactly a century ago in about 1905, you have E.M. Forster's short story, "The Machine Stops," which was the most impressive act of prediction of media technology. It predicts the Web, and e-mail, and all these things. Forster, along with everyone else in his era, predicted that video conferencing or a simulation of presence at a distance would be commonplace in the era of electricity. It's the one gadget (aside from the flying car, of course) that's never appeared. We know why flying cars don't appear, but we don't understand yet why the simulation of presence at a distance, visual telecommunications -- we still don't understand why this thing hasn't worked. I'm determined to answer that question, and if possible, solve it.

Next page: Futuristic Visions

© Copyright 2006, Regents of the University of California