- MEDIA ˇ
- GAME ˇ
- DEVELOPMENT ˇ
Today I will tell you a bit about the system we use to manage the server setup for Heroes & Generals.
A sample graph of the CPU load of one of the servers. You can see that the server load is higher from noon till midnight, but there is still load other hours as well as players from all over the world are playing H&G 24/7.
When seeing the game from the outside it’s really hard to get an idea of just how many machines and services that is actually involved to run our game Heroes & Generals. So, I just wanted to give you a brief overview of the setup. While Reto.Hansg has already blogged about the actual server-hardware and physical locations, I’ll describe the various components.
Below is a diagram describing the setup and the communication channels involved.
My name is Hans-Henrik and my H&G name is Reto.hansg. It’s my responsibility to setup, maintenance and monitor the gameservers, and if they crash, I also get them on their feet again. I have been working as IT-dude in about 20 years now, as system administrator on windows and unix (mostly linux systems).
It is critical to the game that the network is running fast and that we don’t have network delay on the routers in the network to you. So when we order new servers we need to test the stability and the hosting providers network speed to check that it will deliver as planned. I really love this task as I get to talk with some really nice hosting providers all over the world. This gives me the chance to do all kinds of interesting network stuff and I get to see what internet quality we can get in different countries.
The network quality is not great with all hosting providers. We had some issues a while ago with one provider, where we experienced many network errors to the US internet, but the hosting provider managed to fix the problem for us. This sometimes happens, so some of the issues you might experience with bad connection or lag might sometimes be a bit out of our hands, as we currently don’t have our own hosting center. But we do everything we can to make sure that the hosting providers are delivering a high quality service. For example, we use a tool called Nagios to monitor the servers, and to check that the gameservers are running as fast and stable as possible.
One of the areas we focuse on now is to increase the amount of users our servers can handle at once. This goes for both the action servers (currently limited to 24 people per battle) and the strategy game servers. This post is about the strategy servers.
Of course, we design, build and program the servers, to handle high numbers of players at once. But there will be bugs and unexpected bottlenecks, so we run some internal simulations of lots of people, to test if everything is ok. But no simulation is as good as the real thing, which is why we also run load tests with the players from time to time.
The idea behind the load tests is to get enough people on to get at least one of the servers into trouble. The tests tell us 2 important things:
1) How many people we can handle currently
2) What we need to fix in order to increase that number
In order to maximize the amount of people coming in, we close the servers for about 24 hours leading up to the test, and send out a bunch of new beta keys.
The tests involves a lot more people than are usually on, so they are usually quite chaotic and the servers are less than perfectly stable. The chat channels are burning with questions from new players (and old), while we on the dev team are focused on the performance data coming from the servers. Stressful as it is, i still think it is fun to be part of.
Last time we had a load test was about two months ago. When we reached 300 users, one of the servers started showing problems and at 600 it had all but given up, and we stopped the test (it was the server that handles statistics about everybody’s kills, deaths, captures, favorite weapons etc.). The good news is that all the other servers seemed to be coping just fine. Since the last test, we have been working on fixing that issue, and we are almost ready for another test. Thanks to everyone who joined!
See you soon for another test.
While we are developing our game, the world is changing. Change is coming to the Flash Player Plug-in as well and sometimes this change is something we’ve been waiting for!
So how will this affect Heroes & Generals? Read more to find out!
Knowledge is power. Information is power. The secreting or hoarding of knowledge or information may be an act of tyranny camouflaged as humility.
– Robin Morgan, American Poet and political theorist
Woah woah woah…. Hold your horses Ms. Morgan. And stop calling us at Reto-Moto “tyrants”! Alright?! Its not like we don’t want to give information to our players about the war and their assault teams, we just haven’t got around to making the feature yet…. Ok, ok – stop grumbling. I’m on it.
Sorry folks, you had to hear our little tussle with the political activist where she was complaining about the game not keeping players updated about important events in the war. But she just doesn’t get it! The game’s still in “Closed Beta”. Anyway, that’s not a good enough excuse. And we know it.
So, here goes a few words about us preparing a framework which stores all the important events in a war, which would pave the way for a “War Log” feature to be presented to the players in a future build. We’re starting out with a few select important events that are being tracked, such as:
- Player Login/Logout
- Players Joining an Action game
- Assault Team movement
- Assault Teams losing resources
- Assault Team’s supplies being created and when they reach the AT
- Assault Team initiating battle
- Assault Team retreating from battle
- Assault Team movement blocked
- Faction winning and losing control of Battlefields
That’s not to say it’s the end of it – more info will be added in the log as we go along. The framework is actually already in place and its recording some of the above mentioned info on the Live server – right this very moment! Here’s a sample of the raw data dump taken from the server (some pieces of info are removed, in order to adhere to mine and Reto.ogssan’s privacy):
So here I am, the new guy, and someone says ’can you write a blog entry?’.
Well sure. Sure I can.
However, it’s going to be real hard to come up with something that you guys would want to read. I could just flip completely and not write about the game at all, instead documenting my adventures in Copenhagen’s seedy nightlife (I am, after all, a goddamn sexual tyrannosaurus). But I somehow doubt that would make it past the editor.
See, what I do on H&G is mess about with the backend, stuff we dearly hope you never even notice. Whenever I’m working on something that you did notice, it probably pissed you off. Think of me as those guys at Blizzard that work on the Diablo3 backend. Those guys sure got noticed around the launch of D3, and hoo-boy don’t they wish they didn’t ;). However, I really shouldn’t be pointing the finger too much at Blizzard, since we’ve had our own little spot of bother with H&G’s login a while ago, of which I shall now sing :
We’ve been running with Cascaded Shadow Maps (CSM) for quite some time now, with a user adjustable cascade count ranging from 1 to 3 cascades. The last cascade (number 4) has been a one time calculated shadow map for the entire scene, to avoid shadows missing in the distance. The last has been an important addition to standard CSM as we have airplanes in the game, and the landscape just looks flat if we omit it. Initially we talked about precalculated shadows/lighting, but we wanted a more dynamic solution to allow weather changes, and didn’t want to increase the download size of the game.
Even though CSM solves a lot of issues with the quality when covering a large area, they have some really heavy performance requirements. Especially when running with 3 cascades, the number of objects visited and revisited during drawing can become very high. In certain scenarios the CPU time for setting up the objects to be rendered would climb up to 16-20 ms on high-end machines, effectively dropping the framerate from 60 fps to 30 fps.
So I started looking around for alternatives to the CSM and fell across an article written in GPU Pro 2 by Pavlo Turchyn about the game Age of Conan. They were fighting the same issues as I was, but had solved the problem by switching to Adaptive Shadow Maps instead.A debug view of the Adaptive Shadow Map Tiles. The red lines shows the active tiles, and the white lines shows the current camera frustum as seen from the light.
In short, ASM works by dividing the world seen from the light source into tiles, as compared to CSM where it’s the camera frustum that is divided. Each tile is then given an index into a tile map and a shadow map for that tile is calculated and stored in this tile map. For those familar with virtual texturing/mega textures etc.. the algorithm is very much the same. To avoid rendering many small tiles in the distance, the tiles are combined into progressively larger tiles in a quad tree hierachy using the distance from the camera as a guide to the depth of the quad tree. This quad tree is evaluated each frame, and if a new tile has entered the visible range or if the quad tree detail level has changed somewhere, the new tiles are calculated. Typically a single tile every several frames has to be recalculated, giving a huge performance burst compared to CSM.
ASM has one short coming. It doesn’t do dynamic shadows from vehicles or characters. To overcome this problem I maintain a separate shadow map for dynamic objects and combine the ASM with the dynamic during drawing.
The next major release of the game (internally called Avery) will contain the new shadow map system.