Interview with Tetsuya Mizuguchi on Synesthesia, Tetris Effect, Rez, Lumines and more
Tetsuya Mizuguchi is a game developer, University professor, and a man fascinated with the potential of digital entertainment. I managed to get the chance to sit down with the man himself so I could ask about his incredible career.
The interview took place at Reboot Develop Blue 2019 in Dubrovnik, Croatia, where game developers from all over the globe gathered to share their different approaches to creating video games. Mizuguchi-san’s talk on synesthesia and his history in the game industry was fascinating and I looked forward to hearing more of his thoughts.
Although a translator was present to clear up any confusion, Mizuguchi-san spoke English incredibly well and spoke to me easily just as he had performed his talk in English. It became obvious during our conversation that Mizuguchi wishes to push video games beyond just simple entertainment, and wants to achieve brand new emotional responses in people, with Tetris Effect just being the latest, but certainly not the last, step in that goal.
With Tetris Effect, it really felt like a revitalization of the franchise, thanks to the excellent music and sound effects. An older game had become relevant again. Did you expect this to happen?
We hoped it would. But you never know. So we were scared. But, you know, we never thought about that. Just the game production. You know, just talking about how we can make it better and better and better. And we created a ‘wow’ experience with the music, even though it’s Tetris. We wanted to create a new experience, even it’s Tetris. If the people played Tetris Effect and just thought ‘oh, this is Tetris with good music,’ that would’ve been a failure. So yeah, that was a big bit of pressure. But we have some confidence, we have no reasons to be, but we knew we could make something new. And we kept trying and trying. So was really good to get a really good, positive reaction from the people who played Tetris Effect. Yeah, it’s amazing.
Obviously, the musical component is there because you wanted to have the effect synesthesia. How much of the music was put there because of your own taste in music?
I think this is a mix… Basically, we determine the music from the pieces of the sounds, you know, each piece, how can we make it blend well with the gameplay? And so we build the music first, from just simple inspirations. But when making the game, we cut the sound and then sync them to each action. And so this is an interactive process. And we ask the team what they feel when playing. Everyone will be talking about it, the sound designers, artists, and the programmers. Everybody will be talking about it. You know, what did you feel? And do we feel this is fun? We find what is no, and what works. We want to feel some ‘wow,’ a small wow. I love to feel that. And so we need to change and we need to tune aspects all the time. So throughout development, we are doing this. So there’s some music originally different from the game experience. And there’s some music, including the human voices. And it’s not just music from a popular artist. So, the song Connected, we made that. But mostly we wanted to create music just from the sound effects of the game itself. The human lyrics and the voices, and the songs are there to give you some emotional effects as a result.
When you first started to experiment with synesthesia in video games, what were your first methods of how or how to replicate that effect? In your talk, you showed off a vibrating chair and a synesthesia haptic suit, were vibrations one of the first things you tried when trying to replicate synesthesia?
I focus on three elements, vision, audio, and haptic. No smell and no taste.
I want to know what Tetris tastes like.
[Mizuguchi laughs] We had an experiment with Sony, at SXSW, last month, in Austin, Texas. Sony is producing insane devices, and I almost think you could use smells like this. You can use an electric device to give off smells so you can feel, you know, no stress, higher concentration. We mixed that technology and the Tetris Effect with PSVR. So we tested it. So when you play Tetris Effect and a flower blooms and you can smell it at the same time. This adds something. So that’s not the main plan. And haptics, that’s interesting to me, the recent technology, the haptic technology, you can generate the texture of a haptic picture. You can feel detail in music, the bass drum, hi-hat, or human voices, you can feel. But 20 years ago, technology was simple, just static vibrations. This is a kind of experiment, we put 26 vibrators on the body, it was a kind of smooth texture. You feel almost as if you were in the game, and the haptic feelings, like feeling the powerful speakers in a cinema, boom, boom. It’s kind of a new experience that nobody can describe. When you can feel that detail, you know, everybody is like; ‘wow!’ Yeah. Just wow. But we are standing in just the gate to the next era of experiences. Synesthesia experiences leave an impression, you smell the sound and they mix in your mind, so you feel something, or you’ll watch the sunset, and some sound effects will help you to memorize it more, and that kind of impression. So, the game is an experience and a new era and a new area of synesthesia. Yeah, we are having a revolution. So I think VR, AR, XR has two directions it can go in the future. It’s either too real or too abstract. So that I think abstract is much more artistic and is new territory. A new canvas.
When did you first realize the potential of virtual reality for synesthesia? Was it as soon as you saw modern VR headsets?
Immediately, yeah. Immediately, I decided, oh, this is the time. Yeah, no doubt.
Yesterday, you spoke a bit about making people cry when they played some of your experiences, like in Tetris Effect, Lumines? What was it like when you first elicited that sort of emotional response from someone?
That was a big, big achievement. That was one of the goals of the creation. I started my career with arcade games. And so this goal is continuing from that time. How can I move the people by having them play arcade games in a game center? A noisy game center? And just on these three minute, five minute, 10-minute experience? That was impossible. But we could make a separation between the physical environment of the entertainment. Mostly. But nobody cried. But the console games, you can play the game in your house, and you have plenty of time. So it’s getting possible, but the resolution was so low and 2D graphics. But some people cried, I think. So, you know, I started teaching at the University from 1997, 20 years ago. All the time I asked my students, how many people cried at the gameplay? And 20 years ago, nobody raised their hand. ‘What? What are you talking about?’ But now, most of the people raise their hand. It’s very interesting. So I think the graphics can make adventure RPGs, like Final Fantasy, any type of game, very emotional. It’s becoming real. So the game is like a movie. But you know, Tetris… Yeah, I wanted to prove that even if it’s Tetris, it can still be that emotional experience. And how? It was a big achievement for us. Yeah, definitely the music and the narrative. Yeah.
I almost cried playing Tetris Effect, but it was because on was on the last level in Journey mode, and failed on 89 lines out of 90.
[Mizuguchi laughs] That’s a different taste.
It was bitter. My girlfriend was laughing at me. [Laughs] So, when did Tetris become the next game you would attempt to use to create synesthesia?
When Sony announced the PSP. I had a very simple idea to make a Rez and Tetris mixture, kind of experience. But at the time the Tetris company had a contract with EA so I couldn’t make that. And then we made Lumines. That was a really good thing for us. Years later, the chairman of the Tetris Company, Henk Rogers, a good friend of mine, he also played Rez and Lumines, and he had the vision to create beautiful Tetris with music. And so when I visited his house in the United States I began. We had an overnight discussion with fire and watching the stars, and he mentioned the zone experience of Tetris. You know, you play Tetris and it gets faster and, you enter a flow state. And that was very interesting. Tetris has a very unique form and if we could amplify that kind of experience, and if that experience was creating music it would feel much more evocative. But how? And we started to think about that, and we started to draw images. And that means, you know, we started the pre-production, me and our art director. And we drew many, many pieces of art, and imagined the gameplay, and then we make videos. And then finally, yeah, I went to Henk again. Okay, let’s make this. Please license it to us. So we made the game.
And I’m glad you did. We have one question left, so here’s my most important one. What’s your favorite Tetris block?
[Mizuguchi focuses for a moment] The T block. It’s a simple shape. But neutral and symmetric.
A great choice. Thank you so much for your time.