- MEDIA ˇ
- GAME ˇ
- DEVELOPMENT ˇ
So, it’s another happy day in the land of the Backend Programming Tyrannosaurus. I’ve made peace with the sheep; We were just too different, I suppose, but I’ll miss the wool.
Anyhow, let’s talk about my other project, Y’know that game thing? We have servers! In fact we have lots of servers, that do different stuff, and I thought you might like to know what the hell that stuff is. Let’s have a look at a pretty picture, shall we?
What we’ve got here is…
failure to communicate a server architecture. It’s designed to keep you, the players, happy. And when are you happy?
Today I will tell you a bit about the system we use to manage the server setup for Heroes & Generals.
A sample graph of the CPU load of one of the servers. You can see that the server load is higher from noon till midnight, but there is still load other hours as well as players from all over the world are playing H&G 24/7.
When seeing the game from the outside it’s really hard to get an idea of just how many machines and services that is actually involved to run our game Heroes & Generals. So, I just wanted to give you a brief overview of the setup. While Reto.Hansg has already blogged about the actual server-hardware and physical locations, I’ll describe the various components.
Below is a diagram describing the setup and the communication channels involved.
My name is Hans-Henrik and my H&G name is Reto.hansg. It’s my responsibility to setup, maintenance and monitor the gameservers, and if they crash, I also get them on their feet again. I have been working as IT-dude in about 20 years now, as system administrator on windows and unix (mostly linux systems).
It is critical to the game that the network is running fast and that we don’t have network delay on the routers in the network to you. So when we order new servers we need to test the stability and the hosting providers network speed to check that it will deliver as planned. I really love this task as I get to talk with some really nice hosting providers all over the world. This gives me the chance to do all kinds of interesting network stuff and I get to see what internet quality we can get in different countries.
The network quality is not great with all hosting providers. We had some issues a while ago with one provider, where we experienced many network errors to the US internet, but the hosting provider managed to fix the problem for us. This sometimes happens, so some of the issues you might experience with bad connection or lag might sometimes be a bit out of our hands, as we currently don’t have our own hosting center. But we do everything we can to make sure that the hosting providers are delivering a high quality service. For example, we use a tool called Nagios to monitor the servers, and to check that the gameservers are running as fast and stable as possible.
One of the areas we focuse on now is to increase the amount of users our servers can handle at once. This goes for both the action servers (currently limited to 24 people per battle) and the strategy game servers. This post is about the strategy servers.
Of course, we design, build and program the servers, to handle high numbers of players at once. But there will be bugs and unexpected bottlenecks, so we run some internal simulations of lots of people, to test if everything is ok. But no simulation is as good as the real thing, which is why we also run load tests with the players from time to time.
The idea behind the load tests is to get enough people on to get at least one of the servers into trouble. The tests tell us 2 important things:
1) How many people we can handle currently
2) What we need to fix in order to increase that number
In order to maximize the amount of people coming in, we close the servers for about 24 hours leading up to the test, and send out a bunch of new beta keys.
The tests involves a lot more people than are usually on, so they are usually quite chaotic and the servers are less than perfectly stable. The chat channels are burning with questions from new players (and old), while we on the dev team are focused on the performance data coming from the servers. Stressful as it is, i still think it is fun to be part of.
Last time we had a load test was about two months ago. When we reached 300 users, one of the servers started showing problems and at 600 it had all but given up, and we stopped the test (it was the server that handles statistics about everybody’s kills, deaths, captures, favorite weapons etc.). The good news is that all the other servers seemed to be coping just fine. Since the last test, we have been working on fixing that issue, and we are almost ready for another test. Thanks to everyone who joined!
See you soon for another test.
While we are developing our game, the world is changing. Change is coming to the Flash Player Plug-in as well and sometimes this change is something we’ve been waiting for!
So how will this affect Heroes & Generals? Read more to find out!
Knowledge is power. Information is power. The secreting or hoarding of knowledge or information may be an act of tyranny camouflaged as humility.
– Robin Morgan, American Poet and political theorist
Woah woah woah…. Hold your horses Ms. Morgan. And stop calling us at Reto-Moto “tyrants”! Alright?! Its not like we don’t want to give information to our players about the war and their assault teams, we just haven’t got around to making the feature yet…. Ok, ok – stop grumbling. I’m on it.
Sorry folks, you had to hear our little tussle with the political activist where she was complaining about the game not keeping players updated about important events in the war. But she just doesn’t get it! The game’s still in “Closed Beta”. Anyway, that’s not a good enough excuse. And we know it.
So, here goes a few words about us preparing a framework which stores all the important events in a war, which would pave the way for a “War Log” feature to be presented to the players in a future build. We’re starting out with a few select important events that are being tracked, such as:
- Player Login/Logout
- Players Joining an Action game
- Assault Team movement
- Assault Teams losing resources
- Assault Team’s supplies being created and when they reach the AT
- Assault Team initiating battle
- Assault Team retreating from battle
- Assault Team movement blocked
- Faction winning and losing control of Battlefields
That’s not to say it’s the end of it – more info will be added in the log as we go along. The framework is actually already in place and its recording some of the above mentioned info on the Live server – right this very moment! Here’s a sample of the raw data dump taken from the server (some pieces of info are removed, in order to adhere to mine and Reto.ogssan’s privacy):
So here I am, the new guy, and someone says ’can you write a blog entry?’.
Well sure. Sure I can.
However, it’s going to be real hard to come up with something that you guys would want to read. I could just flip completely and not write about the game at all, instead documenting my adventures in Copenhagen’s seedy nightlife (I am, after all, a goddamn sexual tyrannosaurus). But I somehow doubt that would make it past the editor.
See, what I do on H&G is mess about with the backend, stuff we dearly hope you never even notice. Whenever I’m working on something that you did notice, it probably pissed you off. Think of me as those guys at Blizzard that work on the Diablo3 backend. Those guys sure got noticed around the launch of D3, and hoo-boy don’t they wish they didn’t ;). However, I really shouldn’t be pointing the finger too much at Blizzard, since we’ve had our own little spot of bother with H&G’s login a while ago, of which I shall now sing :