Autopsy of an Indie MMORPG - Part 2

by Matt, posted on Wednesday 8 July 2015.

Last time we chatted about networking concepts, the soul­ destroying grind of MMORPGs, and the start of a university project experimenting with web technologies.

Up to this point in development, it’s been the server-side of things that has gotten the most thought and the most planning, for understandable reasons. The client can be thought of as a somewhat dumb terminal, in the sense that it merely displays to the user what the server tells it, and accepts input from the player. The client’s design of the client is largely dependant on how the server is put together. One example of this is the way the world map is split up into ‘screens’.

In the classic 2D Zelda games, when the player moves leaves the screen bounds, the camera scrolls to the next screen. Incorporating this into the game design simplified the design of the server, suddenly we didn’t need to be simulating, or even iterating over, screens that players were not in. (While that’s essentially true, it’s a bit simplified. There were eventually a number of scenarios where simulating a screen with no players was necessary, such as when a player shot an arrow, and immediately left the screen. If that screen stops being simulated, a player may enter five minutes later, and suddenly there’s a previously frozen arrow shooting at them from nowhere.)

But marking screens as hot/cold is an optimisation that applies to the client as well, as when players are too far away (again, in terms of screens) we can ‘deactivate’ and essentially forget about them.

As the server was heading towards maturity, I took some time optimising the client, adding support for player animations, and rendering a world of tiles efficiently:

In my mind, those trees are the biggest nod to Zelda, specifically A Link to the Past, in the whole game. Though I suppose this is a fairly major one too:

That rat enemy was an enemy that didn’t make it into the final version. Every MMORPG has to, by law, include a rat enemy, though this version’s animation wasn’t smooth enough, and eventually I binned it as it increasingly didn’t fit in with newer graphics. Here’s another early shot, where I was working on simulating a large number of moving entities in order to get the client rendering at the targeted 50fps.

It’s also fun working that out in terms of network packets, how much data can you stuff over a line? Remember, only a couple of years ago, things like web sockets were in their early days, and so a Flash client was stuck using TCP, rather than UDP, so working out the effects of bufferbloat, and turning off Nagel’s Algorithm to fine-tune things manually was all part of the fun.

Bugs were great fun during this time, and by ‘great fun’ I mean annoying and occasionally amusing. One bug involved when and exactly how a remote player was ‘deactivated’, when they moved beyond the confines of the locally active screen. The player body would be correctly removed, but the clothing would just… hang there in space. And then when the player came back into view, the body wouldn’t be reattached, and so they would run around in a nearly-naked state.

Another far more tricky issue, was a bug in the networking library I was using, which was essentially a black-box, DLL-style. Messages above a certain 2^n number of bytes would cause the whole Flash applet to crash, and in particularly bad moments, crash the whole browser. But the way in which it crashed was like a slow-motion train-wreck where parts of the client simulation stopped working, while others attempted to keep on trucking. This causes some horrible pasting bug as the renderer freaked out, drawing the player’s names everywhere, while being particularly unresponsive. In order to fix that one, I had to reverse engineer the library and fix the obfuscated ActionScript. Not a very pleasant experience. (Use open source libraries wherever possible!)

The most important, on-going challenge that comes with developing a networked game is having a collection of people willing to test your game, including people in a variety of locations around the world so you can experience what exactly happens when the game is running over real, rather than idealised, networks. Game development friends provided the everyday kind of tests, but every now and then I’d post the game for larger scale play sessions. /r/gamedev on Reddit is a lot stricter on this sort of thing nowadays, there’s a lot more rules about when you link to your own projects, but back in the day (2012), before Feedback Friday was a thing, you could post more freely.

These feedback threads were invaluable. It’s wonderful that a student in south England can find out how their game plays on both coasts of the US, how it plays in the middle-east, how it plays in New Zealand, all while connected to a small virtual server in Germany? It was a wonderfully helpful way to get 10-20 real connections playing the thing for twenty minutes.

Now, it’s helpful to have real connections, but it’s very difficult to get lots of them. For this particular task I wrote a tool that would generate bot clients. Essentially, these bot clients are similar to bots that most online games try to ban, except not quite as complex. These bots had real-connections, but were headless, with no GUI, and moved randomly around the map, walking into things, attacking randomly, generally stressing both the network portion of the server engine, and the simulation side of things, forcing the server to work hard simulating lots of collisions over a wide area of the world map.

Here’s an early test spawning one hundred bots in one small space:

As you can see, a frame rate of two meant there was some optimising to do. Not that I expected that many players to be packed into one screen, but it was worth improving some of that performance. Later tests would spawn these bot players over every spawn point in the game, to give a more natural distribution of players.

On the server-side, I plugged various key performance metrics about how the server was performing into a graphing service named Stathat, which receives stats over http and aggregates them. At the time it was free, but now I see it’s over $100 a month. Gosh. Anyway, a metric that became quite important as I was pursuing an MMORPG number of players was the ratio between the total concurrent player counts, and the ‘server tick load’. The game client was intending to run at 50fps, which is interesting for a Flash game given recent internet scuffles over big, new PC games being locked to 30fps, though I suppose they can’t really be compared. The server was running at 25 ‘ticks’ (essentially frames) per second, the ratio being that it was very easy to double all numbers to easily display what was happening server-side. 25 ticks per second might sound quite slow, but Quake 3 was originally 20 ticks per second. The client can, of course, run at any FPS it likes, and scale accordingly. Here’s a Stathat graph from later on in Scarlet Sword’s development, April 2013, showing Active Entities and the Tick Load Percent, thanks to that bot tool:

A quick explanation of what I mean by Active Entities, when it comes to players and server controlled enemies there’s very little little difference on the server when it comes to which places more load. Sure, the players maintain a network connection, and yes, enemies run some AI routines, but the most expensive part of the simulation computationally is just simple collision checking. Collision checking against static objects, as well as players. The physics engine had evolved by this point to include forces, allowing splash damage that sent enemies flying, pushing enemies out of the way, or being pushed by enemies… it all felt quite physical, working against the usual ‘floaty’ feel of MMORPGs. So grouping both players and enemies under Active Entities was a good thing to track. That image of the two graphs shows four hundred active entities, running at about 80-90% capacity. If I remember right, that was three hundred bot clients, and one hundred enemies, which I remember thinking wasn’t bad for a Python server. Pypy was running amazingly, it’s no exaggeration that the JIT compiler gave the game server twice the performance it would otherwise have had running in Pure Python, for free.

As the game neared releasable state by the end of my last university year, and I was typing up my dissertation, I was still working on content and weapons and areas. I could have wound down the project, but being in the zone, I kept going. Here’s an early shot of the character creation screen from the prototype, followed by a new version:

The inventory also got some attention:

Since I had completely ruled out a subscription fee, and was planning on focusing on aesthetic micro-transactions, I added support for costumes that didn’t affect any of the game’s stats. They would display instead of the actual equipment the player character was wearing.

To be able to produce all the many types of content, I needed some tooling.

Now, I hate writing tools for game development. Some people seem to love it, and that's fine, but writing a level editor, or an asset packer, or whatever the tool is, is sufficiently divorced from the 'normal' business of writing game code that I find it a bit frustrating. I expect this is almost entirely physiological, but writing tools does not, to me, feel like the development is progressing. The truth is of course that when approached correctly, the tools part of development dramatically speeds up the job in the long run.

Having written a number of games now that have required some significant tooling the approach I'm most comfortable with is "do the bare minimum." The tool is not the game, the tool facilitates the production process. If it works as a command-line tool just fine, it doesn't need a window. If you can get away with printing a bunch of keyboard shortcuts to the terminal on start-up instead of coding a menu-bar, then do that. Occasionally a fellow independent developer will show me a beautiful tool, it looks great and works smoothly across multiple platforms. They even took the trouble to craft an icon for the thing, at all the various 2^n sizes that icons require. This isn't really my style, my tools are rough, messy, plain ugly… but they do what they're supposed to do, again, because I really just want to get on with the development proper. Who cares if the level editor no one will see looks nice? It's the levels the editor creates that really matter.

Scarlet Sword required one tool in particular, to be more fully featured than they usually tend to be for me, and that was the world editor. Scroll to the top of this post for a big ol’ screenshot of that tool. Ability to place tiles, edit the collision map, place bushes, enemies, bosses, NPC’s, items, packing the whole file into a compressed format for streaming to Flash clients… all of this was very irritating to have to code for someone who doesn’t like tool development! But when you look at all the existing, off the shelf map editors, and you see there’s nothing that does exactly what you’re after, what choice do you have?

Creating the content for the project was some of the most fun I’ve ever had writing games. Writing the story, the NPC’s and all the dialogue, whilst a small part of the over all project, is what sticks out in my mind as the most satisfying part. Here’s a cross-section of some of the NPC’s that were in the game:

Despite the great enjoyment of the process of writing grumpy rabbits, detectives, 1950’s gangster weasels, sarcastic ducks, the direction I went with the content was, in retrospect, the wrong one.

Here’s what Scarlet Sword ought to have been. It should have been 1 - a sandbox and/or 2 - all of randomly generated content. One developer cannot create enough content for a traditional MMORPG type game in a quick enough fashion. If I could go back in time (and I can) and give me from 2012 two bits of advice, I'd say that. Because by the time it was done and the first release was online, and players were signing up and playing it, I was exhausted. I had created more than 200 screens of content, dozens of NPC’s, dozens of quests and achievements, etcetera, etcetera… and the whole game had taken about a year and a half to get to this stage, and I was tired. And when it went live, people came and played it! And they came, and played it, and they finished all the content in about three hours of play-time, usually in two or three sessions, and then they left.

For some games, three hours of story isn’t bad. There are games on Steam that I’ve paid for and played and loved in which you can see and do everything in less time than that. But for a free-to-play MMORPG, that’s not how that market works. Too late I realised, the grind that these MMORPGs have, it’s not mindlessly added. It’s not just some managing executives saying “let’s copy World of Warcraft, they’re doing really well after all these years!” there’s a conscious decision to spread the content over as much game time as possible. To a certain degree, I understood at the beginning that this was the case, and I reacted against that. “Let’s go back to the roots! Let’s go back to gameplay!” was my mantra, and the reason I drew so heavily from the old-school Zelda games.

MMORPGs (at least, the traditional ones) have a time sink element that is completely alien to most other genres, whether it’s to keep players invested for months, as in the old subscription model, or to keep players playing to get them more invested in their progress, performance and appearance, as in the newer free-to-play model, it’s all for the companies benefit.

Procedurally generating the game’s content would have allowed me to keep the same focus on the mechanics of the gameplay, and keep them devoid of grinding, while massively reducing the amount of time to create unique content and increasing how far each asset went.

Adding sandbox elements, even simple ones like capturing territory, building simple structures, would have extended player’s interest and input in the game beyond the set, designed game world.

While I'm talking about the bigger design issues, let me just pick out a smaller annoyance. When I originally named the game 'Scarlet Sword', that was a fairly original sounding name, as so many MMO's seem to be called "Age of Conflict" or "Rise of the Eternals". I've picked those names out of thin air, but I assume if you Google them, that will actually cause MMO's with those names to exist, like some kind of quantum state behaviour. Half-way through development, however, another company released a game with a very similar name. To this day, Googling 'Scarlet Sword' will show this particular game long before anything related to my project appears. I'm not going to link to it here, because it's particularly unpleasant and slimy, and I hate that Scarlet Sword ever got associated with it, even if only by misunderstanding. That's a smaller issue, but it did cause some confusion.

I mentioned in the previous post that there were ‘financial’ reasons for eventually shutting the game down. Lots of people have assumed that that was server costs, but it wasn’t. Running the game was fairly cheap, in terms of money, because I could run a game server on a Digital Ocean $10 instance very comfortably, so I could run a few in multiple places around the world, and the main website and master server sat on a slightly larger virtual server in Germany. Even just five, six years ago, the costs would be so much higher, but they’re pretty tolerable today.

The biggest ‘financial’ cost is time. I estimated that going back and changing the game’s design, or just keeping on trucking with the current plan and adding more content to make the game closer to ten hours in length would take at the very least another year’s work. That’s fine if you’re at University, you’ve got tonnes of time, but that wasn’t the case by the time the game was completed and had been released for a little bit, I was three months out of University. So rather than the servers, it was the producing of content that was the, very real, financial cost.

But even though I’m quick to point at the various flaws in the game design, the game didn’t do terribly. Scarlet Sword actually picked up a small player-base in some Eastern European countries, Poland, Romania, and The Ukraine, a number of them actually paying money for premium accounts, panda hats, coloured hair. Selling micro-transactions straight to players was a great experience.

The game won some award at the end of University, and I got some freelance work from people familiar with the project, and at least one job offer as a direct result.

However, a handful of thousand players wasn’t enough to convince me to keep on going, or to revisit the game’s design and start fresh in new procedural or sandbox directions. By this point in the project there was just enough technical debt, just enough general tiredness, that I needed to take a break. Apart from bug fixes, small updates, I didn’t touch the game much for some months. After six months, as the servers ran dry, and traffic dwindled, I took a backup of the database and pulled the plug. It was a sad moment.

And yet, I learnt so much. I don’t regret the project for a moment, there’s always value in finishing a thing, and in this case finishing a thing-that’s-a-big-online-game has really changed the way I make games. Of course, every game does that, but Scarlet Sword, out of the twelve games I’ve finished so far, is only one of two that I’m really proud of, for all its flaws.

I think when I started writing this retrospective, I thought I’d finish with saying “and so don’t make an MMO, even kind-of-accidentally.” And you know what? I don’t think I will say that. Go right ahead, if you’re interested. You’ll learn how to write fast servers, design database schemas, charge customers, write NPC dialogue, interact with a player base from around the world, hide secrets behind bushes, update a game in production via continuous integration, and a hundred other technical and design based challenges. It's definitely not the sort of game for somebody who doesn't finish games, but then, most game genres aren't for that person.

So, if you're thinking of writing one of these sorts of games, maybe you've got a handful of completed projects under your belt already, I'd say go for it. Doubly so, if you're a student as I was, with the time to devote to something like this. Make it sandbox, and procedurally generate content unless... you don't really want to. Do whatever you like, there's so much still not done yet within the genre. However it goes, you'll end up a far better developer than you were when you hadn't written an MMORPG.