A Chat With the Creators of Age of Ascent, an Ultra-MMO Powered by the Cloud
Let’s face it, indie gaming has been tremendously important for the industry in the past few years and that’s coming from someone who doesn’t even play that many indie games.
Why? Well, indie developers are the ones taking risks and braving uncharted territories, while AAA studios are often shackled by the logic of investors who want solid revenue on a yearly basis. Indie developers, on the other hand, need to innovate in order to be noticed by the public and thus survive in the large sea of the gaming industry.
UK studio Illyriad Games are literally aiming for the stars. Their latest project, Age of Ascent, is an Ultra-MMO where dozens of thousands of players can play in the same battle (they’ve tested up to 50K concurrent players) and across various platforms such as Steam, Xbox One, Windows Store, iOS/Android and more.
We’ve interviewed CEO James Niesewand to learn more about it. Brace yourself, for it is a long but very interesting read…
How did you come up with the idea for Age of Ascent? Was that something you had been considering for some time?
We started thinking about our second MMO title Age of Ascent in mid-2012, and began working on prototypes in 2013.
It’s always best to make a game on a subject that you’re passionate about, and the Illyriad team’s shortlist of all-time favourite games includes Elite, Freelancer, Wing Commander and Eve Online. I’m also an amateur astronomer, and my office is just up the road from Cape Canaveral, so I get to watch most of the rocket launches from my rooftop. On a practical level, this also means we have a lot of real-world space expertise nearby that we can tap into when we have questions.
So it was pretty inevitable that we’d make Age of Ascent (AoA) – which can be essentially defined as a sandbox-universe, space-based, twitch-combat Ultra-MMO running natively in 3D in the browser.
The idea behind Age of Ascent’s “Ultra” scale was driven by a combination of necessity and hubris!
To explain, we’re a tiny independent game studio (7 of us in total) and the space game genre is having quite a renaissance at the moment, so we had to come up with a plan to clearly differentiate our game from all of the other games out there in a way that went beyond pure game design.
One thing that’s always struck us is that if you look at the technical progression of games over the last 20 years they’ve become ever more beautiful graphically, and they’ve had ever more realistic physics engines… but the something that hasn’t really changed in this time is the number of concurrent players that a game can support in a single shared battle experience.
Of course, there are outliers like the frankly awesome Eve Online that can squeeze 4k players into the same battle, but they only get to these numbers by slowing the framerate to 10% and they’re not actually a twitch, direct-piloting game, which is where the challenge of scale really occurs. Fundamentally, we’re still experiencing twitch combat with a limit of dozens/scores (and very rarely, hundreds) of players concurrently in the same battle.
When we began asking our industry contacts why no one had really pushed the concurrency limits for twitch combat, we were told that it was impossible to do because neither the networking layer nor the physics engine could cope with that concurrent data and compute load. However, if you tell someone like our CTO, Ben Adams, that something is impossible then he essentially sees it as a problem that no one’s yet solved 🙂
It sometimes surprises people to hear this, but not a single member of our team has had any experience working on video games prior to coming to work for us. I actually think that’s a huge benefit, as it means that no one has any pre-conceived ideas about how things should be done, or what’s possible. And so we began work on devising an architecture that could support the end-to-end challenge of a twitch-combat full-3D battle with tens of thousands of simultaneous participants, borrowing ideas and concepts from commercial structures such as Wall Street algorithmic trading systems: systems that live or die by their performance.
Overall, it’s a tough problem to solve because the game will only run as fast as the slowest component. We have to get the networking layer running at the same speed as the physics and interest management MMO engines, data handling and transfer – everything – otherwise the system doesn’t work and the end-user’s framerate will stutter. The volume of data we’re talking at scale is staggeringly large. When we tested to 50k concurrent players in the same battle, we were handling 267 million event network application messages per second. That’s 23 trillion a day across the cluster(s), and that’s pretty extreme by anyone’s standards.
How exactly are you leveraging the power of Microsoft Azure’s cloud, and why did you choose this particular service over others?
We knew that the only possible solution to the sheer network and compute problem thrown up by AoA’s scale could only be handled in the cloud, as any attempt to provide your own inhouse solution would be both prohibitively expensive and utter madness.
The load has to be spread out. And not only that, it has to spread out and to fold and unfold space based on unanticipated demand. Say, for example, 2 large player clans suddenly decide to go to war on a whim (over asteroid mining rights or, more likely, some smacktalk in chat!) and they suddenly meet up in an area of space to duke it out. As the number of players engaged in combat in that small chunk of the universe skyrockets from 50 players to 500 players to 5000 players to 50k players… we have to be able to cope with the load seamlessly.
And that means bringing in additional servers rapidly on demand, quickly bringing the new nodes up to speed with what’s happening in realtime and then sharing the fluctuating load amongst them – without skipping a beat. After the spacedust has settled on the debris field we then have to de-provision those additional servers so that we’re not (as a company) paying for server resource that we no longer need.
We chose Microsoft Azure over other cloud solutions for 3 main reasons:
Firstly, we’re extremely familiar with Microsoft technologies; we know what they’re awesome at and we know where the rough edges are – so it’s a natural fit that way.
Secondly, Microsoft themselves have been extremely supportive, providing assistance and advice throughout. Uniquely amongst all the people we talked to about what we were trying to do with AoA their first reaction wasn’t a wave of laughter 🙂 but instead the offer of an Architectural Design Session to validate and critique our plans. That relationship has continued and deepened throughout the development process and has given us access to build AoA against, and to influence the direction of, essential core Azure technologies such as Service Fabric and Nanoserver. The BizSpark programme, where MS give free Azure time and software licences to startups, has also been exceptionally useful.
Thirdly, Microsoft’s decision to open-source .NET and ASP.NET a couple of years back made it a no-brainer. We’re extremely focused on open-source and open-web and we participate in W3C activities, contribute conformance tests to the Khronos group, and work exclusively at the front-end in HTML5 technologies – which is why we insist that Age of Ascent needs to not only run natively without plugins or installs in the web browser, but that it should also run without requiring expensive hardware. AoA runs happily at 30fps on a $200 chromebook, for example.
When MS open-sourced we could stop working on our own server solutions and start working, in conjunction with Microsoft and the rest of the open-source community, on solutions that would benefit everyone.
For example, the new MS libuv-based webserver called Kestrel that’s currently in development for ASP.NET 5 Core is open source. Ben, our CTO, is the #1 contributor to this project, including Microsoft developers, and has helped make it more than 10 times faster than the previous version, so that it’s now pushing way over 1.25m requests per second. We didn’t do this solely out of the goodness of our hearts, but because our game actually needs our Azure webservers to be handle millions of http requests per second.
So it’s true, then, that the Azure cloud can actually provide additional computing power like Microsoft advertised for its Xbox One console? Most people dismissed that as a joke.
Yes, it is absolutely true.
Games like Titanfall on the Xbox used Azure to provide locality, dedicated servers and an external AI system, and similarly Crackdown 3 uses the compute power of Azure to make entire cities destructible. However, both games had their gameplay focus on small-team multiplayer rather than massive/ultra concurrency amongst the players themselves – which is a very different set of challenges to overcome.
The gameplay model for Age of Ascent has a high-concurrency multiplayer focus from the ground up, where we treat the game client essentially as an input/output and rendering device; and use Azure to spread out the staggeringly-large (and rapidly fluctuating) bandwidth and compute demands that we have to cope with.
But yes, the essential purpose of the cloud is to offload compute and I/O, and to scale them rapidly, as required. One of the main challenges is getting the results of that cloud-distributed compute back to the clients that need to know, whilst keeping track of the bigger picture across all of the cloud nodes and clusters that need to know more.
Do you think that a game structured like Age of Ascent in the cloud, perhaps scaling back the sheer size a little, could work with high fidelity graphics akin to a triple A game?
That’s a really interesting question, and I think the answer is ‘maybe’ 🙂 To explain:
The graphics for AoA are fairly low fidelity by design; but we don’t believe that affects the gameplay or the fun. There are two reasons we’ve deliberately designed the game this way.
Firstly, as I’ve already mentioned, we’re firm believers in ‘any device, any time, anywhere’ and the open-web. So, choosing the web browser as the client is a natural fit. That means that not only do we have to cope with the different capabilities of the different browser vendors, but we also don’t really have a minimum equipment requirement. Whilst I think you’d have problems playing AoA on a low-end cellphone, it does run just fine on high-end ones as well as $200 Chromebooks etc. These devices certainly don’t have powerful GPUs, and some of them don’t have GPUs at all! So we’ve kept our client engine and graphics rendering as lightweight as possible.
Secondly, there’s a very difficult issue to address with the sheer volume of rendering when you’re in the midst of a multi-thousand player realtime twitch dogfight. Yes, GPUs and CPUs do get faster every year – but, for example (and *many* apologies for the following horrible over-simplification…) let’s say your game’s minimum spec client can render 30M polygons with textures, lighting etc at 30fps – but begins to struggle at higher numbers.
If there’s a choice between fluidly rendering one hundred spaceships of 300k polys each vs ten thousand spaceships of 3k polys each… Assuming all the other scale difficulties such as networking, physics engines, IM algorithms etc have been cracked, I know which option I’d choose!
Basically, unless you have the highest spec gaming rig on the planet, we all already know that there are limits on the graphic fidelity that’s able to keep pace with the framerate of any game under load, whether that’s a single-player or an MMO title. Most people who play triple-A games that allow high levels of onscreen concurrency will – at the first whiff of a huge battle – dial down every graphics setting they can to a minimum just to keep their framerate up and enjoy the epic gameplay, instead of fighting with their graphics card.
Notwithstanding, Age of Ascent is not intended to be a high fidelity graphical experience, because at our scale it’ll bring nothing but grief to everyone involved. I’d urge your readers to take a look at some of the gameplay videos from our playtests, and decide for themselves whether they’d rather have a perfectly rendered high-fidelity model (but only, say a few dozens or so of them) or the mayhem of a thousand-player twitch combat.
Of course, there may be a half way point between the two extremes and – especially if GPU-equipped cloud machines become more generally affordable and available to developers – there may also be ways of further off-loading some of the render compute load into the cloud itself so the rendering threads can keep pace with the physics threads (and I think that’s where Crackdown 3 was going).
So, to answer your question directly – I’m sure it’d be possible to have a high-fidelity game with, say 256/512 concurrent players in the near future; and that number will surely only rise over time; given that the developers have cracked the other problems that hit you at this scale. Cloud gaming is a developing field and I think we’re all just beginning to scratch the surface of what it’s capable of. At the end of the day the critical question is whether the game is any good or not; players don’t really care about the specific technologies that deliver their game experience to them – it’s all about immersion & entertainment.
Why do you think AAA games haven’t really used the cloud until Crackdown 3? Do you reckon there will be an increase in the usage of this technology even in big projects?
Fundamentally, I think it’s because the technology is both new and “difficult”. I don’t mean, necessarily, that it’s hard to use. I do, however, mean that it requires a very different mindset to your typical programming model. You have to abandon everything you think you know about typical client-server architectures, and think very ‘differently’ about what you’re trying to accomplish with a cloud architecture. Writing for the cloud is entirely different, and requires different modes of thought.
For example, when I was speaking at //build a couple of months ago, I was asked what our “server tick” was. (For those who don’t know, you can consider the server ‘tick’ as the fundamental “framerate” of the game’s MMO engine).
And the question flummoxed me: because we don’t have a server “tick”. We sorta have an average physics/IM framerate across the cluster(s); but the nanosecond that a server is getting too hot, you pull another server in and ‘share’ the heat. In the cloud, you want your servers to work as hard as they can happily work – no more, no less – and you pull in more resource as the demand increases. There’s not really any kind of ‘heartbeat’ to a cloud system and there shouldn’t be – by design!
I also think that a lot of the game engine tools that many game developers use are horribly ill-equipped for cloud infrastructures, and that’s certainly a limiting factor as well. Writing a cloud-savvy game does *not* mean running the same server hardware in the cloud, rather than in your own datacenter 🙂
Have you ever considered whether to release Age of Ascent on consoles, like PlayStation 4 or Xbox One?
I’ll probably get spanked by the Microsoft PR department for mentioning this – as I don’t think this is public knowledge, yet…. 😀
But if you freeze-frame at 1m33s on this video (a video produced about a month ago by Microsoft themselves, I hasten to add)… I think it’s pretty clear, and you can reach your own conclusion 🙂
Having worked with Microsoft closely, did you have the chance to evaluate the Universal Windows Platform?What do you think of it in comparison to the Win32 format? Epic’s Tim Sweeney has not been kind towards UWP.
We’re heavily involved in getting AoA onto UWP. In fact, UWP actually runs slightly better for AoA than pure native web, because we can actually run multiple concurrent rendering threads in-app (eg separating the UI and the inspace combat rendering layers from each other). Technically, UWP is a really good platform for us – as we’re all about universality and writing code only once; and then deploying it to every platform we can get our hands on, via UWP, Xamarin, or even Atom/Electron shells for (eg) Steam executables.
I read Tim Sweeney’s article in The Guardian very closely – and I am sympathetic to his point of view. I do get what he’s saying, even if that means I’m a “boiled frog” 🙂
However, speaking as a tiny indie studio, we’ve got much bigger fish to fry than worrying about the possible future monopoly of UWP. For example, for Age of Ascent it’s *much* harder to convince game journalists that a native browser-based game can actually be Triple-A quality. That’s an infinitely harder problem for us to overcome than any concerns we might have about the possible monopoly of UWP. Whilst my eyes water at the idea of paying 30% of our revenue to a third party in return for access to their platform, it’s the same on iOS and Google Play as it is with Steam and Microsoft. Fundamentally, I think the view from the top is probably pretty different to the perspective we have 🙂
Generally, it is believed that such huge player numbers may diminish the importance of individual players too much. How are you countering this issue in Age of Ascent?
I don’t think that’s true at all. History is full of epic examples of single individuals turning the tide of a battle through heroic efforts.
Sure, not everyone can be Commander-in-Chief… but that’s not to say that everyone can’t have a role to play. This is the stuff that myths and legends are made of. Legendary storytelling and heroic acts – from Pheidippides at Marathon to Beowulf’s slaying of Grendel – isn’t going away anytime soon!
Do you plan to have a full launch of the game within 2016? Also, are you going to be on Steam?
We’ll be launching AoA this year, yes. The exact shape and size of the launch is yet to be determined (a few things are out of our control – for example, the xbox release schedule); but it will definitely be available 24/7 in 2016.
And yes, we’ll definitely also be on Steam. For us it’s all about having a game that’s accessible on as many platforms as possible, and Steam is certainly an important platform. The more availability, the merrier!
Will there be cross-play with PC and if so, will it be with both the Windows Store and Steam versions?
Yes, the universe will be concurrent, shared and ubiquitous. The xbox players will play with the Windows Store players, the Steam players, the iOS players, the Android players, as well as the native web Chrome/Firefox/Edge etc players. One universe, shared by all.
What was your experience developing for the Xbox One? Did you have any troubles using the esRAM module?
It’s been very easy via UWP to port to the Xbox One. We don’t have access to esRAM via UWP, so that’s not been a concern or a help/hindrance at all. The simple fact is that if the game can be played on a $200 Chromebook, it can be played on an Xbox one without access to the specific hardware abilities of the device. Universality is the name of the game for us.
Did you have to tone down the graphics or scale of the game on Xbox One at all? Can you tell us the target resolution and frame rate?
No, we haven’t had to change the graphics or scale in any way. Most of the hard-stuff-concurrency-scale of AoA is handed off to the cloud anyway; so the actual device that renders the game world matters fairly little. As mentioned earlier, a phone without a dedicated GPU might struggle a bit – but we’ve tested AoA on a 950XL cellphone, casting via Continuum to a 1080p TV at 60fps… and that worked fine!
We target 60fps at 1080p, with a minimum playable baseline at 30fps (any resolution). We’ve had players playing on 4k+ super-widescreen custom rigs quite happily sending us videos of their PvP combats at 60fps, whilst playing the game in Chrome (!) – which is great to see. Depending on hardware etc, ymmv.
We’ve always felt that players should be able to control their rendering environment directly. The software shouldn’t fight the hardware unnecessarily. The most important thing to a player (in the fast-paced depths of an epic PvP combat) is that the framerate stays up there and that their control inputs and rendering outputs match the simulated shared world physics engine at par value with everyone else in the same gameworld. And if a player is on a specific device that can’t do X or Y thing as well (be that lighting fx, shadows etc) then they should be able to dial that down to a setting that satisfies their hardware – if they find it’s getting in the way of playability. We will, of course, provide recommended settings for each devices’ capabilities as far as we can ascertain them – but players should have the control to override these up and down, as they wish.
What do you think of the rumored enhanced Xbox One and PlayStation 4 console as a developer? Would they be useful or more of a hindrance, with more platforms to target?
It’s wonderful for us. As the Age of Ascent code broadly exists in a “write-once” environment, we really don’t have any particular platform preference. So the more platforms we can get our hands on, the better it is for our shared-universe players – and the better for our distribution!
Thank you for your time.