Junior Faculty Awardee Profile: Brian Smith, PhD

So tell me a little bit about your background and your path to Columbia.
I was born in Long Island, but I grew up in New Mexico, in a fairly remote part of the state. I was also a first generation college student. A lot of the kids in my neighborhood had never seen traffic lights, and they'd never seen airplanes flying, they'd never seen escalators. I always was a city person and was always excited about the prospect of making an impact on people and designing things for them. I had that bug really early on. 

How did that manifest in a remote part of New Mexico?

I used to play video games a lot. It was a means of escape or a means of getting to do things that I wouldn't normally get to do in the desert. I was really fascinated by this idea that software could create worlds. Most people see software as tools—things that they use to get tasks done. But in reality, software informs our world, the ways in which we perceive and communicate with each other, how we learn about news and other events. Software is more than just a tool; it really does shape our lived experience. 

That's really powerful. How did you go from imagining worlds to thinking of yourself as a person who could create them—what was your scholastic journey? 

Growing up, I never saw myself as a computer scientist or professor, for that matter. I was really more interested in being a creator and creating new types of fun or fulfilling experiences for people. I had no experience actually learning game design until my master's degree—I did all of my degrees here at Columbia, so I've been here for a while. 

Going into college, I wasn't even set on being a computer science major. I thought I was going to do chemical engineering. I loved how prescriptive and surprising chemistry is. As much as you learn the ground rules of how chemicals combine, there are still so many surprises that come at you.

I only decided that I wanted to major in computer science after my first year, after having tried out a lot of different things. I combed through the course bulletin and wrote a check mark next to any class in the engineering school that sounded interesting. Computer science had the most check marks next to it, so I figured I'd probably enjoy my college years more if I majored in it.

I bet many people don't have a sense of what goes into building games. Can speak a little bit about that process and what parts of it appeal to you?

Game designers need to train themselves to be selfless and to think very passionately about the experience that the player is going to have. Often, novice game designers might think that they want to create some kind of experience in which a player feels lost and then becomes scared and then achieves something. And it's very crafted. But players won't actually feel these things unless they get to make meaningful choices and feel the reactions of those choices. And so a mantra that I've come to learn is, “Get the player on stage and get yourself as a designer off of it.”

It's all about making the players feel good about themselves. And if the goal is for a player to feel a sense of discovery or accomplishment, then you need to let them discover your world and find its secrets. Maybe you give them very subliminal hints. As a game designer, you need to take a step back and let the player have all of the fun. 

What games are the quintessential example of this user experience? 

I think most games that have really large player bases are doing some things right. So Tetris, Simcity, Minecraft—games like that. The Sims...even Bejeweled to an extent. Bejeweled creates a very specific type of experience for players: helping people pass the time on subway commutes.

Can you speak about your PhD work and what your research is about?

I worked with Professor Shree Nayar in the Columbia Vision Laboratory (CAVE) during my very last semester of undergrad here at Columbia. That extended to a summer project and then a master's and a full PhD.  I was excited about working with him, and I was excited about computer vision, an area I didn’t know much about. I figured that if you had a camera that could somehow observe what a person was doing, that could lead to interesting projects.

Computer vision is about teaching a computer how to see: you feed a computer an image as input, and the computer says what's in that image. What type of scene is this office? How many people are in the image? Who are they specifically? What kinds of objects are there? All of that falls under computer vision. The ability to see and perceive the visual world is a bedrock of artificial intelligence and intelligent systems. 

Does this also have an adaptive/assistive technology application?

Yes. Accessibility is a huge and really promising use case. Imagine, as an example, a person who's blind or visually impaired clipping on a small camera to get guidance and know who's in front of them or when it's time to cross the street. 

Over time, I gravitated more and more towards HCI, which stands for human-computer interaction, another area of computer science. My first project in this area was to be able to sense attention from images. So if devices had cameras embedded in them, how could they know when a person is actually looking at it and using it? This has a lot of interesting applications. HCI research really appealed to me and I moved in that direction. 

I ended my PhD fully in the space of accessibility research, making it possible for people to play video games using sound instead of sight. That required an understanding of what's important to playing video games and what visual cues we must communicate to the player in some other way. I'm using sound.

Today I direct Columbia's Computer Enabled Abilities Lab (CEAL). Our goal is to make it so that computers can help people better experience the world, both figuratively and literally. Figuratively in the sense of creating user interfaces that are more adaptive and intelligent and personalized for people so that people don't have to learn new systems every time they download a new app. Maybe the app learns from the person and adapts itself to them and their abilities. And literally in terms of accessibility and giving people new abilities. Lately, I've been working in the area of blind navigation, helping people who are blind or visually impaired navigate the world using sound and perhaps haptics like vibrations. 

The grand vision is to make it possible for a blind person to go to a public space such as a grocery store or a metro station and navigate around that space the same way that a sighted person would, and to do so in a self-directed way — by simply looking around rather than use a turn-by-turn navigation system. We are prototyping different types of audio cues that users would hear using wireless earbuds, and we’re iterating on them in a video game prototype. The exciting thing is that we can deploy our system in the real world once we’ve polished it, but we have the side effect of making video games and virtual worlds blind-accessible for the first time along the way. 

Are you navigating any challenging ethical issues? 

Several. One is regarding the finished product. People don’t like this idea of other people wearing cameras. That's a big ethics problem. It's important to design any camera-based systems in a way that respects others' privacy and that communicates itself so that people around the user can instantly see and understand what this camera is for,  what its capabilities are, and when it might be actively capturing video. Those factors are very important. 

Another ethical issue is about how to involve visually impaired users in the design without overburdening them. This community gets asked to do user studies all the time, and most of the time they don’t get to directly benefit from the study results. There’s an ethical dilemma there, and I want to do this right.

Yet another ethical issue is how to design the system in a way that doesn’t make visually impaired users “stick out.” Maybe make it something that sighted people will want to use as well—just ask anyone who's visited Penn Station for the first time. That’s why we’re designing audio systems that we build to work with regular wireless earbuds.

What is your project for the Provost’s Award?

I am working on an audio navigation system. Ideally it will work with just a standard pair of headphones or earplugs, that makes it possible for people to navigate using sound in a self-directed way. And so what I'm envisioning is a very discreet user interface that allows people to scrub their finger on a smartwatch touch screen to survey what's around them—twelve o'clock, one o'clock, 2 o'clock— and form a spatial model of the environment. 

Can you speak about your teaching? 

Last semester, I taught user interface design, which shows students how to engineer usefulness and usability in interactive systems. The biggest takeaway is that great design does not come from divine inspiration. There is actually an engineering process that you can go through to guarantee that you reach a good design — one that is actually useful to its users and is usable to those users. In the final project, groups work on some pretty ambitious concepts, Web apps and websites, to solve problems that people who are different from themselves have. 

Mentoring students has also been wonderful. It's been so nice to be grounded in that initial sense of discovery and fascination that students see as they begin their research. And it's been fun to brainstorm with students on potential new ideas right after just having completed a PhD myself. Getting so many different and complementary views on research very early on is really awesome. 

What is the most challenging part of your work?

Forming meaningful relationships with users who are blind and visually impaired. I want to create a culture around what's called contextual design, really learning from this community and learning what their needs are and matching what we do to what they need. That's tough, and it is taking a lot of legwork for me to make that happen since I don’t have any connections with the community. People often ask me if I have anyone in my family who's blind or visually impaired — if that inspired me — and the answer is no. I was inspired by the problem and how interesting and impactful the solution can be. But finding people who are blind or visually impaired to engage with the design process is difficult. I've gone downtown countless times to different organizations serving that community, just to introduce myself and try to make that connection. I'm also looking to host at least one undergraduate student in the lab who is blind or visually impaired so we can learn from each other. 

What do you do in your free time? 

I try to keep current with video games. I like to travel; I often go out to Long Island on the weekends. I've become somewhat of a foodie, so I enjoy checking out new restaurants in the city. I go to the ballet and the opera relatively often as well.

Do you have any book recommendations for us?

Yes, I would say two recommendations. One, regarding game design, is The Art of Game Design by Jesse Schell, who's at Carnegie Mellon. It’s my favorite book on game design and covers almost everything you need to know about how to design for the player. And the second would be Understanding Comics,  by Scott McCloud, which is probably the single greatest example of an art form decomposed that's ever been published.

To learn more about Brian Smith's work, please visit his website, or you may email him directly.