Data design for Civilisation games

I seem to have Civilisation on the brain at the moment; and with the recent 0.6.4 release of Unangband out the door have been spending some time looking at the Freeciv project. If you're looking for roguelike game design, you might want to stick around though, because I'm going to be discussing data file design in some detail.

Freeciv is a Civilisation clone, written originally as an exercise in client server programming (There's a nice history summary at the bottom of this AI discussion). Its implementation of the various Civilisation games falls somewhere between Civilisation 1 & 2 as Alex Mayfield points out the last time I mentioned my Civilisation Revolution house rules. Luckily for me, much of the unit design is very similar to Civilisation Revolution so it seems quite possible that I may be able to get away with limited amounts of source code changes.

The mod system that Freeciv uses is quite similar to the Angband edit file system - with the exception of the format which is seems influenced by the Windows .ini file format. A mod in Freeciv consists of a number of rulesets which are used to initialize data structures with values and flags which impact the game play for that unit. A typical Freeciv unit may have the following format in the units.ruleset file:

[unit_warriors]
name = _("Warriors")
class = "Land"
tech_req = "None"
obsolete_by = "Pikemen"
graphic = "u.warriors"
graphic_alt = "-"
sound_move = "m_warriors"
sound_move_alt = "m_generic"
sound_fight = "f_warriors"
sound_fight_alt = "f_generic"
build_cost = 10
pop_cost = 0
attack = 1
defense = 1
hitpoints = 10
firepower = 1
move_rate = 1
vision_radius_sq = 2
transport_cap = 0
fuel = 0
uk_happy = 1
uk_shield = 1
uk_food = 0
uk_gold = 0
flags = ""
roles = "DefendOk", "FirstBuild"
helptext = _("\
This unit may be built from the start of the game. It is the\
weakest unit.\
")

Contrast this with an Unangband monster entry, and you'll see there is not a lot of difference, other than the format which defines how the values are read in:

N:24:Small goblin
G:k:y
I:110:8:20:16:10
W:2:1:0:4
M:0:0:1:0
B:HIT:HURT:1d5
F:DROP_60 |
F:OPEN_DOOR |
F:ORC | EVIL | IM_POIS |
F:HAS_SKULL | HAS_SKELETON | HAS_TEETH | HAS_CORPSE | HAS_HEAD | HAS_HAND |
F:HAS_ARM | HAS_LEG | HAS_BLOOD | DROP_MISSILE | DROP_TOOL | DROP_WEAPON |
F:DROP_CLOTHES | DROP_ARMOR | DROP_FOOD | DROP_JUNK |
F:SCENT | DWARF |
D:It is a squat and ugly humanoid figure.

One initial observation is the Unangband file format encodes a lot more information in much less space. We could reduce the total length of the Freeciv warriors entry to about half the current length by adopting a similar context-dependent format, as opposed to the more context-free format that Freeciv uses. We know every Freeciv unit has to have an attack/defense/move/vision/firepower and upkeep values, so there is no need to have each of these defined on a separate line: and the user interface itself can be used as a guide for how we can lay out the values more compactly in the ruleset file format.

This may not sound especially important, but from the perspective of quickly editting multiple entries - for someone trying to mod the game - screen real estate is a valuable commodity.

The second observation is that the Roles in the warriors unit in Freeciv are not intuitive, in the same way the Unangband flags are. You can guess straight away from looking at the goblin entry the ways that HAS_HAND and DROP_WEAPON might be used in the game. DefendOK may make sense, but FirstBuild? In some ways, this may be an issue with the fact we're dealing with an agglomeration of multiple individuals rather than a single entity, but it is worth bearing in mind that defined roles (and flags) may not 'natural' and may be worth redesigning.

There is a significant difference between the way roguelikes and Civilisation games play that is not reflected correctly in the similarity between the two file formats. Roguelikes rely on your ability to learn and understand special abilities and unique effects and which one to use at any point in time. The fact that small goblins are immune to poison makes a significant difference only if you are facing a small goblin, and have a choice between a poisonous and non-poisonous attack. So a binary flag is an appropriate way to reflect the criticality of this decision.

Civilisation games on the other hand, rely on the aggregation of small differences over a large number of units, and so scalar values are much more important than binary flags. The fact that a warrior can defend a city is much less important than the attack and defense values of that warrior: a difference of only 1 in attack strength is the difference between success and failure of a particular combat, but that combat may be played out multiple times in a single game, which converges towards an average result in a way that a roguelike does not. If you want to think about it another way, most roguelikes allow you to reset to your starting position of full health at any point during the game with a single item - Civilisation games don't allow you to reset to your full complement of warrior untis.

So while the Freeciv file format is undoubtedly a natural evolution from moving hard-coded game data to an external file the same way that Angband does, I believe there is significantly more value to be gained by extending the file format of Civilisation in interesting ways.

If you look at my Civilisation house rules, you'll see a lot of rules of the format 'with technology/civic/building', unit X gets Y, where Y is often a modification to an existing scalar value of some kind. Or more importantly, the set of units of type Z get this change, where Z could be all units, mounted units, pre-gunpowder units and so on. It would be extremely useful and flexible if we could encode this into the Freeciv ruleset without having to have flags refer to hard coded bonuses in the game code.

There's a couple of ways of considering how to do this. We could go with a full blown scripting model, where each unit runs a script to determine which modifiers apply to it's basic values, and the avaible technologies, civics, buildings and so on are exposed to the script system to check. However, I'm uncomfortable with a full scripting system, and I think we can do this in a simpler fashion.

Imagine a unit as a collection of scalar values and properties. Some properties are intrinsic to the unit ('on-foot'), some properties are dependent on the civilisation ('the-wheel-technology', 'democracy-government'), some are global ('nuclear-winter','global-warming') and some are positional ('attacking-city', 'defending-forest') or modal ('fortified'). A unit type then defines a set of rules where the unit having the matching property modifies either a scalar value through setting the value, addition, multiplication or scaled by percentage, or through the addition or removal of a property for that unit. We can use properties to also indicate that the unit is a member of another unit type, which also has a set of rules which further define the unit and so on.

We can then determine the unit scalar values by starting with the base unit type, and matching rules from that unit type and the rules from its properties, applying the effects of each rule exactly once.

For instance, one unit type might be as follows:

military-unit = {
can-attack;
can-defend;
fundamentalism-government:+1 attack;
...
}

We can also evaluate scalar values instead of properties.

military-unit = {
...
experience>=3:veteran;
veteran:+50% attack;
veteran: +50% defence;
...
}

One important extension to this is that the units have to be able to affect other unit types. For instance, aircraft cannot be attacked by ground units.

aircraft = {
*ground-unit:-can-attack;
...
}

And we have to be able to override this:

anti-aircraft =
{
*aircraft:ignore;
...
}

where ignore is a reserved value that prevents the associated property from being processed.

I'm still in the early days of sketching this out, but I hope this will be an improvement on the existing Freeciv rulesets. The implementation of this design will, of course, be another complication.

My question for you is whether I'm reinventing the wheel here? I believe this is a design pattern that multiple games must have confronted before, and there should be something that already does this work for me, short of going to a full scripting system like Lua.