Valley of Genius Read online

  Young Harvill: VPL originally stood for “Visual Programming Language.”

  Tom Zimmerman: Jaron was developing a programming language. He really wanted to go for it.

  Jaron Lanier: Back then we had the freedom to think in the big picture, which very few people have now. So the way people thought is they started with a general philosophy of life and tried to turn that into a computational paradigm and then thought about software architecture and hardware interfaces as part of the same unified concept. So that’s exactly what Doug Engelbart did with his Augment system, and it’s exactly what Alan Kay was doing with his systems: Smalltalk and the familiar interface and computer came about as one thing. And there were many other examples. So this is the way we all did it back then.

  Young Harvill: Jaron’s idea was that you could have a visual language. Kids could choose a series of actions that were represented by pictures, watch what they did by individually triggering them, and then collect them in a little order. And then that would become essentially a subroutine represented by this new picture that you made. Then that would become another one of these items that you could then demonstrate how it worked.

  Tom Zimmerman: Jaron saw the glove as an input device for his programming language.

  Young Harvill: What Jaron was pushing for, and Tom I think also, was for gestural interface rather than just click and drag and place. The visual language that you get with a mouse and a keyboard is limited. If you wanted to have a visual-type input, or say you wanted to have more analog-like input to trigger music or bend notes, then the gestural interface was really important. And so that was his whole point with the glove and gestures.

  David Levitt: It’s a direct manipulation interface.

  Young Harvill: Jaron was very interested in Koko the guerrilla and the whole gestural thing. We were interested in sign language as just a beautiful gestural language. So I think at some point Jaron felt that this was a way to go after the whole Engelbart thing in terms of just pure expression. And everyone knew what the guys at Xerox PARC did and what they put together and how far they pushed that system—it was a touchstone.

  Jaron Lanier: I was really interested in systems where you could change them while you’re using them without limitation.

  Young Harvill: So the first thing I ended up doing for VPL was the output for the optical glove that Tom was working on at the time.

  Jaron Lanier: We very quickly added a tracker to it so we knew where it was in space, and integrated it with some of the software we already had so you could pick up stuff. And it clicked beautifully. I know it ran, I mean I have a picture of it running on a really early Macintosh, so the glove was already running and interfaced in ’84. Glove-based interactions were very hard, because the machines were so slow back then. So it required heroic programming, which was in those days mostly supplied by Andy Hertzfeld on the sly.

  Andy Hertzfeld: I have zero interest in talking about VPL or Jaron.

  Alan Kay: Jaron’s first version of VPL was certainly the best demo of anything I’d ever seen on a micro. It was just fantastic in every way, a programming tour de force with four or five really great ideas in it about how you might think about programming—particularly for younger kids.

  Dan Ingalls: It definitely had this feeling of being able to do drag-and-drop in a 3-D virtual world. There is a lot that can be done in programming using essentially drag-and-drop: You can drag properties out of a parts bin and drop them on objects, and then the objects take on those properties.

  David Levitt: The glory of programming this way was that by linking the graphic objects together and saying, “This one has that strength, this one has this weight,” you could make gravity collisions. You could make a whole game. And if you drew it in front of a kid they would get every step. It was just miraculous: They understood this as programming inside a video game, rather than about a video game in some external text language.

  Alan Kay: And it was just great. And of course it used a very, very simple glove he had. And I loved it.

  So, as a by-product of building this novel programming language, Lanier had the beginnings of a virtual world. It was, essentially, a 3-D version of the graphical user interface pioneered at PARC. Instead of a mouse, there was a glove. Instead of a two-dimensional desktop, there was a three-dimensional workspace. Yet Lanier’s virtual world was still trapped on the other side of a screen. It was another Atari Research alumnus, Scott Fisher, who took the final step—the leap into cyberspace itself.

  Brenda Laurel: Scott Fisher, who was part of the proto-VR scene at Atari, went on to NASA Ames and really put the first systems together.

  Scott Fisher: God, after I was fired from Atari it must have taken like nine months to do the background checks, but by the beginning of ’85 I’m at NASA. I was also really passionate and obsessed with this idea of getting these head-mounted displays put together. And within the first week I discover that they have already put together this really great display.

  Tom Zimmerman: So Scott had a head-mounted display at NASA.

  Jaron Lanier: It was a very, very low-res display. So basically each pixel was as big as like a kitchen tile or something. But it had a head tracker and it was reasonably fast. And you could see fairly detailed things in it.

  Scott Fisher: It had like, I think, one-hundred-by-one-hundred resolution.

  Brenda Laurel: It was just green vector graphics on a black background.

  Tom Zimmerman: Scott’s vision was an astronaut could manipulate things outside of the capsule while being inside the capsule. Sort of like what they now have with the shuttle arm.

  Scott Fisher: So I tracked down Tom and he by then had hooked up with Jaron to do VPL, and they were trying to figure out how to use the glove for this visual programming language that Jaron wanted to do. And I’m like, “Well, can you guys build a glove for us?” Because I needed a way to reach into that space and touch these virtual objects.

  Tom Zimmerman: So Scott Fisher did telerobotics, while Jaron was talking about synthesized worlds. And he came to us with that vision and then contracted us to make a glove to do that.

  Scott Fisher: We spent a lot of time with Tom and Chuck Blanchard, who did most of the software. He was from VPL. He helped us interface it with our system and wrote some great code to make that work. We ended up getting another glove so we could have two hands. And Tom continued to work on the glove technology. So it was getting better and better.

  Tom Zimmerman: It was kind of the industrial use of virtual reality.

  Brenda Laurel: They were doing training. I can remember laughing because one of their demos had a pull-down menu and it was like, “Why the fuck would you go to so much trouble to model a real world and then put a pull-down menu in it?” It’s just like using a speech interface to say, “Move arrow on my screen.” It was dumb. But I could immediately see the possibilities.

  Jaron Lanier: It taught me how VR worked. It made a huge impact on me.

  Tom Zimmerman: So Scott gave us entrée into matching the gloves with the head-mounted display. So that was a crucial element.

  Jaron Lanier: Then we developed our own head-mounted display.

  David Levitt: Our hardware engineer took apart a Sharper Image TV, a little Sony TV that was relatively uncommon in those days, and reverse-engineered until we found the video signal and said, “Yes, we can make a Silicon Graphics machine drive that.”

  Jim Clark: Silicon Graphics was a company that primarily made what would now be called GPUs—special purpose graphics processing units—to accelerate graphics, so that people who used our equipment could visualize the models that they made.

  David Levitt: We used two Silicon Graphics machines, one for each eye, of course. So that became the first EyePhone.

  Jaron Lanier: EyePhone, E-Y-E, obviously.

  Mitch Altman: Our resolution was superlow: 480 by 680.

  Jaron Lanier: And the EyePhone was the first commercially available head-mounted display. We sold a lot to labs a
ll over the world.

  Scott Fisher: I made it very clear in talks that I gave and talking with our collaborators like VPL that our next big push at NASA was for a multiuser system—having multiple people in the shared virtual space.

  Jaron Lanier: The whole thing of being in there with other people where each person becomes an avatar was superimportant. That was the whole point.

  Tom Zimmerman: And so what Jaron did was he took the head-mounted display, which was done by Scott Fisher at NASA, and he now created this whole interactive virtual world where he was really emphasizing interaction between humans in this synthetic world.

  Scott Fisher: And next thing I know VPL has put out press releases saying that they’ve done it! It was competitive, and I hadn’t been thinking of it as competitive. So again I guess I was kind of naïve about that.

  Tom Zimmerman: There might be some jealousy. Because Jaron definitely got 95 percent of the media attention.

  David Levitt: Our lead product at VPL was called “Reality Built for Two.” RB2, that was an SKU at VPL.

  Jaron Lanier: This is actually a bit of an inside joke and it goes back to Alan Kay. So Alan Kay had once called a computer “a bicycle for the mind.” And in fact Steve Jobs put that slogan on some of the early Apple literature. So I thought the idea of it being a “reality built for two” recalled bicycles.

  Alan Kay: A lot of this stuff is just modern engineering versions of old ideas. Ivan Sutherland did it first.

  Jim Clark: Ivan had created a head-mounted display in 1968. I had the idea that one ought to be able to work in a three-dimensional environment doing three-dimensional design. Making all that work and putting it together as a system, and doing the mathematics to make the curves and all of that, was essentially my PhD thesis. It was the first early beginnings of being in 3-D.

  Tom Zimmerman: Jaron really took the idea and ran with it and did a great job of marketing. Jaron used to say we were “the haberdashers of virtual reality.”

  Jim Clark: I kind of resented Jaron Lanier for coining that term “virtual reality,” but he is a very smart guy. He is not a technology guy particularly.

  Jaron Lanier: See, we didn’t make the first head-mounted display, but we did make the first commercial one. And the requirements for a reproducible one were totally different. So we had to be able to make them on an assembly line. That was all us.

  Tom Zimmerman: So when we started Jaron saw the glove as an input device for his programming language. And the message we got pretty quickly was no one wants to program but the glove is cool. So we had a glove. We had a pretty powerful new paradigm of being in the simulation. So we were a hammer looking for nails. We were capabilities looking for applications. And it turns out entertainment was the killer app and probably still is.

  Kevin Kelly: I don’t remember how I connected with Jaron—maybe I was the first—but I somehow wound up at VPL with Jaron in ’88. I came down to try out his goggles and gloves setup. This was pretty early on, so there wasn’t a lot of other people interested in this. I got to try on this thing and entered in a world that Jaron had just made that afternoon that he hadn’t even explored. And it was pretty amazing. It was just like, Wow, this is amazing! I thought that it was the beginning of something really big immediately. I could see it.

  Jamie Zawinski: I was working at Berkeley in the AI group there, and somehow we got an invite. Yeah, waited in line, put the helmet on, got to play around, it was running on SGI Indigos or something. It’s like Oh, I can reach out and I can grab the diamond, and then it turns into a rose! Wow, that’s so cool! I probably got to wear the thing for maybe a couple of minutes, but it was amazing! It was so cool.

  Jamis MacNiven: You’d put the headgear on and a glove, and then you sort of walk out into virtual space and you just drift through the room. It was all very geometric, but you felt as if you went out of your body, drifting, and the glove would be the way you would direct it. I think you saw a little icon of a glove that was your hand.

  David Levitt: To move in the world you needed navigation. The standard way was to let the dataglove be your controller and to point in the direction that you wanted to go and click your thumb to accelerate.

  Jamis MacNiven: And you’d drift around corners and down streets, and you felt like you had gone into the Matrix!

  David Levitt: I wound up being one of the big people demoing people in it and expanding and improving the demos over time. I experimented with gravity, and I made it sideways. I set it up so that any time I closed my hand, a ball would appear, and when I released it would be pulled by gravity, but horizontally so it would bounce off a wall and then come back to me.

  Jaron Lanier: David Levitt was the first person to fall asleep in VR and then woke up inside VR which is this amazing thing.

  David Levitt: And waking up to that could not have felt more alien. “Oh my God, gravity is sideways!” It was just too much.

  Kevin Kelly: And it was very fun, kind of surprisingly fun to just explore a fantastical world with nonsense entities and creatures and stuff. And so the thrill is probably closest to visiting a really weird temple in Kathmandu. It’s sort of exotic in that sense. It’s strange. And you are also aware that this is all happening in your living room or whatever and that’s kind of exciting, too. And of course Jaron has got his whole philosophy and he’s going on and on.

  Jaron Lanier: Postsymbolic communication, yeah. I used to go on and on about that stuff.

  Kevin Kelly: I didn’t understand anything he was saying.

  Tom Zimmerman: Then Young Harvill and Young’s wife, Ann, started working on these bodysuits instrumenting the whole body.

  David Levitt: People, of course, like new technology and they quickly said, “What are the sexual possibilities?”

  Howard Rheingold: A lot of it was, “Okay, we’re going to have sex at a distance through computers somehow.”

  David Levitt: The term teledildonics quickly became common.

  Howard Rheingold: I got so much weird attention for that one word. There were, I’ll tell you, journalists from all over the world wanted to talk about teledildonics for a little while. It was part of this cyberculture vision, the Mondo 2000 vision.

  Jaron Lanier: There was a tremendous pressure from the cultural underground: “Oh, I am supercool. I publish the underground magazine from Amsterdam and I know Tim Leary.” It was, “I did this and that and I’m the coolest and you have to give me a demo.” There were a lot of people like that.

  Scott Fisher: When VR got into popular culture and the media went crazy, I felt that it was kind of over at that point.

  Jaron Lanier: They were really wild times back then. I mean there was a flow of celebrities into the place. My personal assistant would be like, “The Dalai Lama is stuck in traffic. Go ahead with Leonard Bernstein, it’s okay.”

  David Levitt: We demoed for Spinal Tap.

  Jaron Lanier: I remember Spinal Tap showing up. There was this big fuss. I was like, “Why do we want to show it to Spinal Tap?” Then some of the engineers said, “Oh, come on! You have to let Spinal Tap in.”

  David Levitt: We made a tiny virtual Stonehenge in honor of their movie!

  Jaron Lanier: I said, “You know, if you use the wigs they’ll just get messed up by the head mounts.” And they said, “Fine.”

  Jaron Lanier: And that was like a day at work at VPL—just kind of completely insane.

  Young Harvill: I remember being worried about the hype. You know that inevitably in pretty much every technology it reaches this point where people really get what it could possibly do and then they just kind of spin creatively on. Then they kind of convince themselves it’s kind of already there. That’s a really dangerous phase for any technology.

  Scott Fisher: People started saying that VR was dead or it was never going to go anywhere.

  Kevin Kelly: Why did it not take off? I certainly expected it to take off. I think it’s not just that it was the collapse of the hype.

  Jaron Lanier: My wor
st mistake with VPL was the business model. The problem was that there’s not that many labs ultimately: After maybe a couple thousand customers in the world you run out. And it gets hard to come up with something so dramatic for the next-generation machine that you get them to spend the money again. And so at that point all you can do is either try to go down in price point to sell something to larger numbers of customers—or turn into a military contractor.

  David Levitt: So the RB2 system, the Reality Built for Two, included a Mac as the main control, and one or more Silicon Graphics machines. In those days it was hard to get the performance you needed without a separate Silicon Graphics machine for each eye, so that got expensive.

  Mitch Altman: RB2 was a quarter-million-dollar machine, not many people could afford it.

  Jaron Lanier: For the full-on thing, it was about a million bucks a person. And there were only a couple thousand customers in the world who could pay a million dollars for a VR system.

  Kevin Kelly: Without Moore’s law you couldn’t take the next step. You could say the expectation ran ahead of VR and that was the fault of the expectation, but actually I think the expectation was valid. It’s just that there was a gap. There was some evolution needed to take place. We went to the moon and then didn’t follow up.

  Jaron Lanier: VPL started to hit a ceiling around ’92, which forced the question of what to do next. I wanted to put all available resources into selling Microcosm, which was this amazing product that we developed that never shipped. It was this beautifully sculpted one-piece virtual reality unit that had all the tracking equipment and the computers and everything. There was a really innovative head-mounted display—sort of like opera glasses where there was a handle so you can get in and out of it really fast. And then you could wear a glove on the other hand—and it worked beautifully. I thought that we might be able to get that to a price point where we could sell enough of them to start to follow Moore’s law down and sell it to larger and larger volumes and make it to the big time. So that’s the path I wanted. The majority of the board wanted to turn into a military contractor, patent as much as possible, hold on to the patents until they were valuable, and then sell them. It’s a very reasonable business dispute. So I left, and the company continued on trying to do the other plan.