Psychochild's Blog

A developer's musings on game development and writing.

19 July, 2006

System design for online games
Filed under: — Psychochild @ 6:55 PM

Online games last a long time. Meridian 59 was launched commercially in 1996, and it’s still kicking. This is quite different when compared to traditional games, where a game has a shelf-life measured in single-digit months in most cases. Because of this, most game systems need to be designed to be extensible and easily maintainable.

What really goes into designing and implementing major systems in an online game? I figured I would talk a bit about this so that people understand why games are designed and implemented the way they are. All decisions are a trade off, and I’ll describe some of the more common ones. I’ll also show some bits of Meridian 59′s code as well. :)

What are the goals of gameplay systems in online games? As I mentioned above, extensibility and maintainability are key; unfortunately, people that come from more traditional game development sometimes undervalue them. These features usually only add unnecessary time to development in traditional games.

So, what do I mean by extensibility? This is the ability to add new content to the game easily. As many of you know, having enough content is hard. Therefore, you want to be able to add content quickly and easily. If it is difficult to add content, it will take more time and you will not be able to add as much content. In addition, an easy-to-use system requires less training so you can bring on less experienced developers to help produce content.

What does maintainability mean? This primarily deals with bugs: Can you add new content and features without inadvertently adding new bugs? Can you fix new bugs easily? Unfortunately, as code gets older it tends to get less maintainable: you use older APIs which just don’t have the robustness of newer code in most cases. Also, as multiple generations of developers touch the code, you get pieces and patches onto the system, resulting in a very fragile system. So, it is important to start with maintainability in mind when developing online game systems.

So, how do developers usually address these issues? There are a few common ways many teams address these issues.

I think one of the better ways to do this is to embed a scripting language into the system. This helps extensibility because scripting languages are usually developed for people with less programming experience. Requiring a fully trained programmer to assist with all gameplay development tends to be a serious drain on resources; if you can have a technical-minded designer implementing the gameplay systems, you can assign that programmer to one of the other vital technical areas that always seem to exist. This type of architecture can also separate out the ugly code (such as network communications, sockets, etc) from gameplay code. You can also develop the scripting language to be much more forgiving of errors than typical C/C++ code would be.

The disadvantage to this is finding a good scripting language. Creating a solid language is not something a casual programmer can do over a weekend: it takes an experienced developer months of work to design and implement the scripting system. In addition, you must invest more time into training new developers to use the scripting language since there are not likely to be a lot of books they can learn from. Alternatively, using an existing scripting language can save time but sometimes the scripting language isn’t exactly what you want. The language may not have as much focus on speed as your project requires, or it may not be easily distributed over multiple machines in a cluster, etc. In addition, you may find yourself supporting a language if the developers decide to stop work on it. You must also be careful not to pick a language which is too technical.

It’s good to look at existing scripting languages to see what the alternatives are. Lua is a popular choice for game development due to it’s small technical footprint, but it’s not necessarily a language that someone without a CS degree will be able to pick up very easily. Python tends to be easier to learn, but the language developers have not focused on speed as many game developers need. The Stackless variant of Python fixes some of these issues, but there is precious little documentation and the primary developers have many other projects that demand their time. However, EVE Online has used Stackless with much success.

Meridian 59 uses a proprietary scripting language called Blakod. It has syntax like C, but with LISP-like lists. A typical bit of code looks like:

   FindHolding(class = &Object, sequence = 1)
   "Finds the 'sequence'th (default first) entry of type 'class' in the passive list."
   {
      local i, each_obj, iFound;

      iFound = sequence;

      for i in plPassive
      {
         each_obj = Send(self,@HolderExtractObject,#data=i);
         if IsClass(each_obj,class)
         {
            if iFound < = 1
            {
               return each_obj;
            }
            else
            {
               iFound = iFound - 1;
            }
         }
      }

      return $;
   }

It's a fairly well-designed language, created primarily by Christopher Kirmse who now manages the XFire team. This language was developed in the late 90's, and includes neat things like document strings which weren't widespread until the development of Python. For the curious: % signifies a comment, and $ is means "nil", or is similar to the Python value "None", and signifies no value. The Send(...) syntax is a function call with parameters. I rather like the language, although I do have a CS degree and am a bit of a language geek, so it may not be optimal for use by less technical people. However, the language was highly optimized for speed, and it runs very well on low end machines.

Another philosophy of system design is to "separate data from code". For example, a programmer or scripter might code weapons into the game. However, actually defining each weapon might be done in a separate file (often a spreadsheet), which each variable assigned by a designer filling in the values for each weapon to appear in the game. So, one weapon might be called "Short Sword", have a speed value of "0.9" and have a minimum damage value of "3" and a maximum damage value of "6". The "Steel Long Sword" might have speed value of "1.2", minimum damage value of "5" and maximum damage value of "12". By entering values into a file, the designer can focus on creating and balancing content instead of worrying about coding syntax and introducing bugs.

Unfortunately, this isn't a perfect solution. If you want any special features, such as a sword with a "proc" that does a spell effect, then you need a way to represent that in the data. Usually this is done with flags, so if you want a sword that casts a Hold spell when it strikes, you might include a flag like "PROC_HOLD" in the data. If you want different weapons to have different chances to apply the special ability, then you might need another value in the data to hold how often the special effect will take hold. You also have the problem that the designers are restricted to the flags they are provided. If you decide you want to have a vampiric weapon that drains life and gives it to the wielder, you might not be able to do that. Perhaps if you had "PROC_DAMAGE" and "PROC_HEAL" you could put them both on the same sword, but the code might not always have both of the special abilities fire at the same time. Adding in a new "PROC_VAMPIRIC" will require some programmer time.

However, when you are limited in what you can do, you start to see patterns in the content due to these restrictions. Perhaps your quests seem a bit stale because they're just variations on a few different types of quests. Well, those different types of quests are probably what are coded into the game: deliver X, kill Y, collect Z. Creating a quest is probably a matter of plugging in specific values into a spreadsheet in order to create the quests. This isn't to say that you can't do a lot with such a system, even make it very appealing such as in the case of WoW, but eventually the repetition can bore some observant people.

In Meridian 59, the data is part of the code. Each weapon is its own class: short sword, long sword, mystic sword, scimitar, etc. The scripting system uses inheritance, so we have a weapon superclass that defines how a weapon works, that it can be wielded, etc. Each individual weapon file includes details about damage type, bonuses and penalties, etc. We did abstract things out a bit in weapons, so that we have different types: you have three types of weapons (swords, blunt weapons, heavy weapons) and three qualities (low, medium, high). Low quality weapons are what newer players will find and use: they do a bit less damage, but have bonuses to hit to compensate for low skill. Swords are easier to hit with, blunt weapons don't penalize spell very harshly when wielded, and heavy weapons deal the best damage. Meridian 59 has few weapon types, so there would not be a big benefit to extracting the data into a separate file.

"Magical" weapons in Meridian 59 use a separate code system called "weapon attributes". Each instance of a weapon keeps a list of attributes it has and specific data related to that attribute. So, for example, a weapon that has the "glow" attribute would point to the bit of code for that. When a character wields the sword, they get a bit of light to see better. If they unwield the sword, they lose that light: this is very similar to the flags system mentioned above. This system works very nicely as the two systems are not intertwined and changes to one do not necessarily affect the other. The attribute system was designed and implemented by Damion Schubert.

One area where the previous developers did try to separate out data is Meridian 59's quest engine. The data is still stored in a script file, but it is separated out into list data structures instead of embedded into code. Unfortunately, this system didn't work out as well as they might have hoped. The system should theoretically allow for easy addition of new quests; in practice, it's a pain in the ass. You have to include a bunch of data with very little reference to what the values mean. A typical quest entry looks like this:

   % QST_ID_PARTY
   plQuestTemplates = cons([  piDefaultNumPlayers,\
                  piDefaultQuestType,\
                  Q_PLAYER_NOTTRIED_RECENTLY | Q_PLAYER_LAWFUL | Q_PLAYER_NOTNEWBIE,\
                  [ 152, 153, 154 ],\
                  20,\
                  [],\
                  100,
                  [  ]
                                                ], plQuestTemplates );

Each quest is made up of different quest notes, which are built and stored in a separate list. That means if you want to add a completely new quest, you have to build each quest node in the same file (which is one step of the quest, such as approaching the first NPC, fetching an item, killing X monsters, etc.), then put them together in this format. Not pretty.

As you can see, not an easy thing to decipher except for the constants in all capitals. Now, there is a bit of comment at the top of the area that defines all the quests, but this is a the top of a 7450 line file that is about 355 KB in size! keep in mind that M59 does not have a whole lot of quests, since that really isn’t the focus of the game. So, this method can get very unwieldy.

In addition, adding new quests that don’t follow a typical pattern can be very difficult. At one point in M59′s development, we thought about adding in multiplayer quests: quests that players could cooperate or compete against each other while doing the quest. Unfortunately, the existing setup did not allow us to add in these options very easily, as it would have required re-writing major sections of the quest engine. As mentioned above, it can be difficult to deviate from what has been established.

Hopefully this gives a tiny bit of insight into some of the decisions that go into system design. You have to have a good, solid base to work from, and you have to keep extensibility and maintainability in mind when you develop the game. However, there are trade offs for every decision: making a generic system to allow easy addition of new content could end up restricting the type of added content. Adding new content not anticipated at the time of the system development can also make it hard to add new types of content to the game. As usual, there is no silver bullet.

Any other thoughts on this topic?







13 Comments »

  1. Awesome, awesome post! Thanks for sharing that — it was quite an eye-opener for me!

    I do a fair amount of database work in my day-to-day job and often I’ve thought about how databases could be used to manage the various “specs” of different objects in games. Since I’m not a game designer, this seemed like sheer brilliance to me on my part (I’m often given to such flights of fancy). Some time ago the bubble burst when I learned that — duh — of course that’s how they do it! But this post was the first really clear illustration for me.

    Comment by chabuhi — 19 July, 2006 @ 8:51 PM

  2. Yeah, databases are very commonly used in modern online games. Some use them to store data as describe above, but others use them to store the state of the world. Since you work with databases, chabuhi, I’m sure you know how notoriously cranky they can be. :)

    However, Meridian 59 actually does not use a database. All data is in script, and all world state is in RAM. World state is written out to flat files on a regular basis: one of ugliest bits of M59′s tech is the fact that the save cycle stops the whole game and writes everything out to disk. With faster computers it’s not so bad, but it can still take 30 to 60 seconds if you have a reasonably well-populated world. Almost everything else on the server side, however, is very nice to work with.

    Have fun,

    Comment by Psychochild — 19 July, 2006 @ 9:21 PM

  3. Socket code is ugly?!?

    I’m very hurt, Brian ;)

    Comment by Cael — 20 July, 2006 @ 2:34 AM

  4. Using Stackless Python is not exactly rocket science :)

    All you need to understand is that you can launch functions to execute concurrently in a cooperative manner, where they cooperate by yielding or sleeping for a specific amount of time, and that you need to implicitly schedule them. EVE doesn’t do much more than this, and what it does do over and above this and why it does this, have been mentioned in the presentation made at PyCon 2006, and on the mailing list.

    I would like to note that there is only one primary developer, Christian Tismer, but he does have other projects to devote his time to. But the mailing list has people who are willing to try and help. There is even a tutorial someone has written recently. But still, some documentation would not go amiss, you are right there :)

    Comment by Richard Tew — 20 July, 2006 @ 2:41 AM

  5. Cael wrote:
    Socket code is ugly?!?

    Yes. That’s why you get paid the big bucks, right? :)

    Richard Tew wrote:
    Using Stackless Python is not exactly rocket science :)

    I actually really like Stackless Python. But one of the advantages of Python, in my opinion, was that it can be learned fairly easily by someone without a CS degree. Stackless’ lack of documentation works against that advantage. Having people willing to help, like Richard helped me when I was learning Stackless, does help manage the learning curve. But, it’s not something a casual scripter can get into over a weekend. That tutorial does look like a step in the right direction, though.

    Comment by Psychochild — 20 July, 2006 @ 5:20 AM

  6. Really interesting post :). These are the kind of things non-game developers (like me) wonder about sometimes (a lot).

    Looking above to chabuhi’s post – what was the design philosophy behind storing world state in RAM? It seems like one of those ideas that make some people go ‘uh-oh.’

    Comment by Jpoku — 20 July, 2006 @ 12:36 PM

  7. P.S. Your category links seem to be busted. :S

    Comment by Jpoku — 20 July, 2006 @ 12:38 PM

  8. Excellent post, Brian. A great piece of insight for those of us on the outside.

    To the database developers out there: imagine writing an MMO that made heavy use of databases – unparalleled, even. Imagine, the whole time its under development that you get assurances from the software vendor that it’ll withstand the massive number of transactions per second that you’re relying on it for. Then imagine, in the first stress tests, you found out the vendors were… a little off in their estimate. OK, A LOT off… and you have to spend the next crunch session making as many dynamic elements into static ones, so they won’t need as many database calls. When the original team of SWG talks about the “technological barriers” they ran into that hindered their original vision… that’s the memory they’re evoking.

    I’m not an MMO developer, but ever since I heard THAT story, every idea I come up with is tempered by that vision… and the question… would it scale? it’s interesting to note that even the selection of scripting language is impaced by the performance. A single player, turn-based game like CivIV can get away with using Python & XML… MMO’s still need to trim the fat wherever they can.

    Comment by Chas — 20 July, 2006 @ 2:55 PM

  9. Chas,

    You are exactly right with the SWG issue. It really put a crimp in their work, from what I heard. :(

    The trick to a good scripting language is finding something that can be extended in C. Python is good for this, because if you’re doing anything that is highly time sensitive or where you spend a lot of processing time, you can re-write that section in C. It’s a bit less flexible than having everything written purely in a scripting language, but it can speed things up. Also keep in mind that you can throw some “big iron” at the issue on the server side. While it’s an admirable goal to try to use off-the-shelf components and machines in a cluster for each shard, a well-funded game could get some rather expensive machines on the back end to handle more intense processing.

    Some more thoughts,

    Comment by Psychochild — 20 July, 2006 @ 3:38 PM

  10. What we find is useful is having two scripting languages. We have one fairly robust scripting language that the game logic is written in (everything from the political stuff to combat to ships, etc) and another, much more limited and specialized scripting language for the world builders to use. It’s powerful enough to let them do up to and including some simple-medium game logic (someone made an entire theatre system with it once that allowed players to stage productions, sell tickets, use costumes, etc), but it’s simple enough for complete non-coders to use.

    We’d have a real problem with world-building if we required people with coding aptitude (even if not expertise) to world build, so the second language solves that problem nicely.

    –matt

    Comment by Matt Mihaly — 20 July, 2006 @ 5:06 PM

  11. I like to use C# as a scripting language as well as the programming base. It’s entirely possible with a decent OO structure to include the base operations blocks and derived classes within a limited functionality environment.

    And that’s the last time i show you my beautifully documented and commented UDP stack, dammit ;)

    Comment by Cael — 23 July, 2006 @ 6:27 PM

  12. Thanks for the reply :) It was quite insightful.

    On the whole scripting side of it I think it’s important to consider the provision in a system for end user generated content. Looking at the LUA script stuff for WoW as an example. People have done amazing things with it, but it’s certainly not for the non-technical orientated. Wrappers and things like drag and drop editors, or a custom scripting method (thinking again of Blizzard and their warcraft ‘triggers’ map editors.) are interesting optins.

    I’d be really interested to see how Ryzom ring works out for people from that perspective. Seeing and doing being two different things..

    Comment by Jpoku — 24 July, 2006 @ 8:28 AM

  13. No problem, Jpoku.

    I’ve always been wary of user created content. In some cases it works; Matt Mihaly’s games have been able to take advantage of user-created content very easily; however, I think this is largely because of the niche market and because of a text interface instead of a graphical one. Nothing against Matt’s games, they’ve been very successful, but I’m sure he’d be the first to agree that the system isn’t necessarily one that would work in many other situations.

    It’s also interesting that you mention WoW’s scripting system on the client side. One thing that I found surprising is that there are a number of “gentleman’s agreements” about what the scripters will and will not do. There are some potentially serious issues that could arise, and I’m sure that the gold farmers have probably written custom UI elements to make gold farming easier. It can be tough to enforce this type of thing.

    However, I think that offering some level of scripting is interesting. You would definitely have to work hard to make sure the system isn’t exploitable. It’s worked in some situations, so it’s something worth looking into to see if it would benefit a particular game.

    My thoughts,

    Comment by Psychochild — 24 July, 2006 @ 5:03 PM

Leave a comment

I value your comment and think the discussions are the best part of this blog. However, there's this scourge called comment spam, so I choose to moderate comments rather than giving filthy spammers any advantage.

If this is your first comment, it will be held for moderation and therefore will not show up immediately. I will approve your comment when I can, usually within a day. Comments should eventually be approved if not spam. If your comment doesn't show up and it wasn't spam, send me an email as the spam catchers might have caught it by accident.

Line and paragraph breaks automatic, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Email Subscription

Get posts by email:


Recent Comments

Categories

Search the Blog

Calendar

November 2019
S M T W T F S
« Aug    
 12
3456789
10111213141516
17181920212223
24252627282930

Meta

Archives

Standard Disclaimer

I speak only for myself, not for any company.

My Book





Information

Around the Internet

Game and Online Developers

Game News Sites

Game Ranters and Discussion

Help for Businesses

Other Fun Stuff

Quiet (aka Dead) Sites

Posts Copyright Brian Green, aka Psychochild. Comments belong to their authors.

Support me and my work on