# How to Measure Luck vs Skill in Games?

Anyone who has played a wide variety of games knows that some games are almost purely skill (i.e. Chess, Go), while others are 100 % luck (ie. Candyland, card game of War). However, most games are in between. I would find it helpful as a game buyer if there were an objective measure indicating how much luck is involved in a game, to include as one of several criteria when deciding which kinds of games I'd like to buy and spend time learning. I personally prefer games where luck/probability plays a role, yet deliberate practice increases skill over time (i.e. card game of Bridge). But obviously other people will have different preferences.

I've observed many debates about the amount of luck and skill in certain games and I sometimes use information gleaned form these informal debates to help me decide whether to purchase a game. Very few of these debates cite objective measures to indicate how much luck or skill is involved.

Note that in a BGG luck/skill thread I started, one person did cite an attempt to objectively measure whether skill exists in the game Fluxx.

So what measures exist and how useful are they? Or is there some persuasive mathematics to suggest that useful measures (of how much luck is in a game) are not possible?

• A game of 3 in a row is 100% skill, but how can you determine that. On a 4x4 board, an expert player that goes first should win 100% of the time. Does this mean that the game is 100% skill, or does whatever determines first player introduce luck to the game? My real point was that if you change a 4x4 board to a 5x5 board, a truly random player will get worse, where a basic player won't be affected. If you are using random players as your yardstick, meaningless changes like these will give the appearance of more luck. Jan 7, 2013 at 6:21
• The fact that this question is already turning into an "extended discussion" in the comments sets off warning bells in my head... Jan 7, 2013 at 11:53
• (To clarify, the discussion in comments is off-topic. If someone has a system to share, I'd love to see it.) Jan 7, 2013 at 22:17
• I personally think this is a really good question and have been pondering an answer. Jan 7, 2013 at 22:54
• If I win, it's skill. If you win, it's luck. Jan 8, 2013 at 19:55

Think about the Elo Rating System used to measure the relative skill of Chess players. This is essentially an equation that takes a series of wins and losses and produces a number that can be used to predict the chances of one player winning a game against another player.

One of the inputs to the equation is a "distribution" that describes how much a difference in skill changes your odds of winning. This part is tweakable - they started with a normal distribution, but some groups have switched to a logistic distribution because they've found it to be more accurate.

If we had a fully deterministic game like "Which player weighs more?", the Elo rating would just be your current weight (or some deterministic mapping thereof), and the distribution parameter would be the Heaviside function (0 if less than 0, 1 if greater than 1).

If we modify that game a bit to make a game like "Roll a D6, add that to your weight, and see who has the highest number", the Elo ratings for everyone would still be the same, but the distribution function would have to change so that the behavior from -6 to +6 was a stepwise thing based on the distribution of die rolls.

If we had a zero-skill game like War, everyone would have the same Elo rating (and the distribution function wouldn't matter).

So if we have a two-player game that produces win/loss results, and a set of players, we can force the players to play games against each other and establish Elo ratings for every player, using some distribution function (maybe we start with a normal distribution like chess did). Eventually we'd be able to look at the results of all the games and determine that a logistic distribution (or a log-normal distribution, or some other distribution) did a better job of predicting the results than a normal distribution.

The shape of that distribution - how 'wide' it is, how it tapers near the ends, how steep it is in the center, and so on, is the answer to your question. Of course, this takes a huge amount of effort to measure, but it's theoretically possible to determine that curve.

Addendum: Here are some distribution curves (courtesy of Wolfram Alpha). Apologies that these don't have 100% consistent X/Y axes, but hopefully you get the point.

This is the 'normal' distribution (rather, its CDF). For example, using the yellow line, we'd expect a player with a skill of "+2" to win about 80% of the time.

The logistic distribution is very similar, but has heavier tails (this is really subtle!). The X-axis here is different (why, Alpha?); again using the yellow line we'd expect a player with skill "+50" to win about 90% of the time.

Note that with both the logistic and normal distributions, there is a parameter to tweak how 'wide' the spread is.

Heaviside:

Stepwise (using the "weight + d6" example, this should be transposed up and flatten out at the left/right extremes):

• i am not sure iunderstand how different these curves would be. Coul you add diagrams for a fullly deterministic game (like how old are you, what do you weigh), a competely random game, and maybe one or two games that are some where in between? Jan 8, 2013 at 23:03
• Nice answer! It's good to point out that even in a "pure skill" game like chess, you still need that distribution. I wonder how things would look if you gave each player a personal distribution--more consistent players would have smaller variance etc. Optimistically, there would be a lower bound on the variance, and that would be the luck factor. Jan 9, 2013 at 2:53
• +1 This sounds workable. Did you definitely mean -12 to +12? I was expecting something like -6 to +6. Jan 9, 2013 at 14:18
• Doh, you're right. I was somehow thinking of a D6 that rolled -6...+6. Jan 9, 2013 at 15:27
• I think you're onto something here. Would it be possible to get some images that are examples of sample distributions outlining how different games might appear? I think it would help to visualize things. Jan 9, 2013 at 16:26

First, there is a quite elusive difference between chance and luck, but it is worth noticing before analysing either concept deeper. Any non-deterministic event during the game is necessarily a chance component; but its effect on a given game situation may be exactly neutral (either in practical terms in a unique situation, or in some general, mathematically qualified meaning).

For example, the initial setup in Dominion, where kingdom cards are randomly chosen (and made available to all players), is a major chance event that entirely changes the overall course of game; but the luck effects can be almost as small as the mental differences between the players can ever be. This is because each player gets nearly the same opportunities as everyone else - regardless of what they are.

In games with mixed luck + skill factors, there is a tight interaction between the two. The skill factor includes knowing the statistical distributions within the chance factor. There may exist critical but non-obvious breakpoints between distinct strategies, based on analysis or experience, triggered by observable game events. Conversely, decisions taken by the player can subsequently affect the degree of chance (make the game more random or less random), as well as the expected outcome (bad luck versus good luck). This does not really mean that more skill could translate to more luck, but it should warn against many naive imperfect metrics of luck versus skill based on what looks like magical luck or enormous chance.

In addition to these obvious "combinatorial modelling" and "statistical modelling" aspects of skill, there are also others. Strategic (high level models), psychological (basically a statistical model of the opponent's decision making), memory and so on. Each of these components assigns a difficulty (a learning curve) to every game, and that is a key obstacle to defining skill vs. luck in a player independent way.

To see this, consider Nim (or chess). Either you are bright enough to see the full winning (or non-losing) strategy or you aren't. If you are, the game reduces to a pure game of chance, namely the drawing of who goes first - and maybe some amount of psychology if your partner is still somewhere on the learning curve. If you are not, you are playing a pure game of skill, struggling to be the first player to notice a winning path for yourself.

This reasoning applies to the other components of skill as well. Some board games are actually employing quite obscure skills, including imagination, dexterity, or deception, and the game experience can be drammatically different based on where on the learning curve, or natural disposition with regard to that respective skill the player group happens to be.

So, you cannot really define a rigorous methodology and end up with a single number usable with all sorts of player profiles, and with all skill levels. You need to know your "market" as an assessor, and to know your own player group as a consumer when evaluating a game.

However, knowing the players perfectly, games of skill are easy to detect. You know how. Have a relative newcomer play against a relative expert (within your expected group). What's the percentage of games that the expert will win? This metric however makes slow games appear as comparatively luck based. There is not enough "tries" to even out the factors that do tend to even out, including chance. So you can improve this metric by playing multiple rounds for a defined typical amount of time and looking at the probability that the expert will emerge with the better score.

Unfortunately, there is no universal definition of an expert. It means different skills for different games; and different skill levels for different player groups.

(As a side note, many countries attempt to define games of skill and games of chance for legal purposes, that is, to apply different public regulations to these broad groups of games. In this context, no accurate definition or methodology is ever employed, which is telling. Only a few games most often played in commercial contexts ever really get classified, and that classification is usually based on cultural perceptions and on business models. Example again. If you are really successful in Blackjack due to your memory skill, then the business model (rather than the game structure itself) implies that you are not welcome in commercial establishments where the game is played. Blackjack is culturally expected to be a game of chance. You will experience the opposite expectations in a bridge club or in a public bridge tournament, despite that the same card sets are used in both games, and superficially you control the deals themselves even less in case of bridge.)

• "knowing players perfectly, games of skill are easy to detect . . ." I was thinking about that very same methodology lately and how it doesn't seem to work. First time I played Dominion, I was one of 2 beginners against two somewhat skilled opponents. I won 1 of my first 4 games. I've seen the same thing more dramatically with my 7 year old son who, after his first 7 games, managed to catch up to his father in skill level, at least as measured by wins per game played since those 1st 7 games. Yet I keep being told over and over that Dominion is a game of great skill . . . Jan 8, 2013 at 1:07
• Conversely, many people claim that Stone Age has a great deal of luck. My son has never won a game and I have yet to see someone who has played less than 5 games win against someone who has played more than 30 games. Could all be a statistical fluke of some sort where the noise is drowning the signal. But it does make me wonder about "games of skill are easy to detect" . . . Jan 8, 2013 at 1:10
• "games of skill are easy to detect", right? 1) How many games do you to play to know that the newcomer didn't win by luck? How do you know your newcomer isn't especially adept at game x, and is a quick learner? How do you know the experts expected win percentage (or are you comparing their win percentage vs. 50% chance)? This answer appears to be a lot of talk with little actual information. Dominion Kingdom setups can have a huge effect on win percentages, directly affecting the 1st player advantage. So when determining if Dominion is more/luck based, do you have to compare only a ... Jan 8, 2013 at 2:46
• particular kingdom setup? Do we need billions of rankings for Dominion because the game rules "change" for each kingdom setup? ... Does this answer actually answer the question, "how much skill is in this game?" Jan 8, 2013 at 2:47
• @user1873 - Re statistics: There is no threshold. The described method also does not work if you cannot guarantee that the player's overall skill levels are different. The reason is that game outcomes can prove a skill component, if they are statistically too unlikely to permit a "pure chance hypothesis" in a symmetrical game, but they can never prove a chance component. Jan 8, 2013 at 8:06

I do not have a full answer to this question but I have the beginnings of an answer, which I hope is supplanted by a better one.

This answer assumes that game rules are strictly followed, with no cheating or imperfect components. (In other words, no loaded dice, or imperfect dice whose imperfections can be observed after many thousands of rolls, etc.)

The question can be broken down into several simpler questions. The easiest is:

Q1: Is there a way to know for certain that a game has zero skill?

Answer: Yes. Games with zero choice such as Candyland or the card game of War have no choices. If you can't make a choice, it is impossible to apply skill.

Q2: Is there a way to test whether a game with choices has non-zero skill?

Answer: Yes, with caveats. As cited in the question, the game Fluxx has numerous random elements but also choices. A study proved skill exists in the game of Fluxx by playing 200 games where one opponent's actions were random, while the other followed a small set of pre-defined tactics. One can imagine a similar methodology applied using computer simulation to any game which is guaranteed to terminate in a reasonable amount of time.

However, all this can do is prove whether skill exists in a game. It cannot prove that skill does not exist, because if a tested set of tactics is poor, then it may appear that the particular set of tactics used does not produce a statistically significant advantage over random behavior. For most games it will be impossible to test every possible set of tactics, so it is therefore impossible to prove using this method that no skill exists in a game that has choices.

There exists a class of games where behaving randomly in some or all situations confers an advantage (i.e. Rock Paper Scissors). So testing whether skill exists should not just be against a purely random opponent, but also an opponent who automatically follows one simple rule (i.e. always chooses Rock in Rock Paper Scissors, or always follows the identical priority pattern when choosing cards in Fluxx). Then the second opponent can be programmed to detect patterns in the first opponent and react the detected patterns. If that confers an advantage, then skill exists.

There is also a trivial subset of games that can be mathematically proven to have no skill: games with choices where the choices are meaningless. For example, imagine a game whose board design is such that the first move is chosen (like which of the 1st 10 spaces to start on for a Candyland-like board), but every choice leads to exactly the same result - a victory (via different paths, depending on the choice). It would be easy to mathematically prove this game has no skill.

Q3: Is there a way to test whether a game has no luck?

Yes, with caveats. If you can program a computer so that it never loses (it gets to choose whether to go first or last to avoid the first/last issue encountered in games like Nim), then the game has no luck. However, the converse is not true. Just because a computer program has not yet been found that will always win a game, does not prove the game has luck - it could be that the program has not yet been discovered.

There exist classes of games where it is clear there is no luck by simple reasoning. One such class is games with no hidden information and no randomization mechanisms, such as Chess or Go. There may be other such classes as well, though I'm not clear as to whether games with hidden information such as Stratego can be considered to have no luck, because there is the luck of which types of information get revealed in which order.

So far, I've covered the very easiest edge cases and not tackled the much more difficult question:

Q4: For any pair of games, both of which (provably) have some luck and some skill, is there a testing methodology which will demonstrate which requires more skill?

I do not have a complete answer for this much more difficult question. I do not know whether it is even possible to answer, or whether there is academic literature on the subject. But I do have the beginnings of an idea:

How many lines of code does it take to program a computer to never lose? By this metric, Chess requires vastly more skill than Tic Tac Toe - which obviously matches intuition. This has issues such as the relative length of computer programs varying with the computer language chosen, and sequential vs parallel computing architectures (very relevant to pattern matching games like Chess or Go). But if this idea were further refined, it might actually lead to something, even if just a complexity classification system along the lines of Stephen Wolfram's work on complexity in cellular automata.

This is a tough question, and perhaps it won't be answered fully for hundreds of years, if ever. But if there's a better answer than this one, I'd love to see it. And I welcome feedback on how this answer can be improved, if there is anything faulty about the logic.

• Regarding Q3: I think absolutely no luck can often be determined based on chance elements and information. Go and Chess have no hidden information, and no random elements, thus they are 100% skill games (regardless of how well computers can play them). Stratego has hidden information but no chance elements. Backgammon has no hidden information, but lots of chance. However, in either of those games, I think a skilled player would defeat a random player 99.999% of the time. Jan 8, 2013 at 0:56
• You haven't defined what it means for a player to play with random actions. For example in poker it is important that a player's bids do not reveal their hand, and so they should bid 'randomly' using a carefully calculated distribution. Just because the best strategy is to play 'randomly' does not mean there is no skill in the game. Jan 8, 2013 at 10:23
• @tttppp You're absolutely right. An even simpler example is Rochambeau (Rock Paper Scissors). I'll figure out how to incorporate into my answer, though it may take some thinking. Jan 8, 2013 at 17:34
• @shujaa I agree with you and need to figure out a way to incorporate your comment into my answer. Jan 8, 2013 at 17:41

I think you're asking the wrong question.
Instead of asking about how some nebulous term (such as 'luck') applies to any particular game, what if we ask instead "how much effect can random chance have on the outcome of a game?" Now we suddenly have an answerable question!
A game like Chess or Go has no random factors in it, so the answer would be "none". This would, in your terminology, be an "all skill" game.
A game like Candyland or War is completely driven by random factors, so the answer would be "all". In your terminology again, an "all luck" game.
Obviously the in-between games (such as the oft-mentioned Fluxx) have some measure of both. The shuffled deck of cards, the roll of the dice, whatever is built into the game to incorporate the random factor. There are player decisions that are critical to determining the outcome, but the decisions to be made will have to involve the random factors. Sometimes your numbers come up in Catan, and sometimes they don't. What you do in each case, how you change your play to extract the most resources out of that random factor, that's a good portion of the skill involved. How can we measure how important the random factor is in any of these games?
Well, I think our best method of doing that is by applying a very powerful tool: gamers. It's the near-fanatical players of the game that have the knowledge and experience required to 'rate' any particular game that they've played sufficiently. In exactly the same way that it's the gamer that rates a game on the 1-10 scale, it's the gamer that would be able to rate a game on much much of a factor random chance has on the outcome of the game (I would envisage this as a 0%-100% scale) and via a dozen votes I'm sure a common average value would again emerge.

In summation, my answer on "How to measure luck VS skill in a game?" is this:
By adding a "Impact of randomness on game outcome" value to every game in the BGG database and waiting for the most powerful tool of all (us) to calculate the answer for each and every game.

And I really don't imagine that there's any other method that can work anywhere near as well. It's the power of parallel processing!

• Though BGG does not have a ratings category for either luck (randomness) or skill, I've seen many text descriptions. I am led to believe from the text descriptions that gamers, being human, are a poor judge of randomness. Classic experiment: ask 10 people to distribute themselves randomly throughout a large room. They spread themselves evenly apart, no clustering. Predictable. And very non-random. To site specific game example: Jan 9, 2013 at 16:42
• Stone Age is a game where numerous people who have played less than 20 games comment about how much luck it has. Yet highly skilled Stone Age players never lose to people who have played fewer than 20 times. So if you have 1000 people rating the game, only 50 of whom have achieved a high level of skill, their voices will be drowned out by the other 950 low skill players that aren't good enough to realize that the role luck plays in this game is small once you reach a certain skill level. The dice rolls and random cards/tiles ordering sure gives it the appearance of luck, though. Jan 9, 2013 at 16:45

I know exactly what you mean and I've wanted some sort of ranking/rating system as well, in part because how "mean" a game is depends in part on how much luck is involved (mean games being ones where non-winning players feel as though they lost due to choices targeted at them by other players; Diplomacy is probably just about the meanest game out there).

While a very granular or statistical study comparing and ranking games is probably more work than it is worth, I think that there is a rating scheme that would be useful for knowing more about a game before buying it. How about giving a rating to the existence of various elements in the game based on whether they are

• non-existent or flavor-only (-)
• balanced (according to the rule-book) variety-increasing elements (1)
• asymmetrical and capable of altering scoring position (2)
• the core mechanic for winning (3).

The elements would be

• A: Unchosen Flat Chance (single die, spinner with equal segments, etc.)
• B: Unknown Probabilistic Chance (dice under a cup, cards dealt face-down, etc.)
• C: Hidden or Simultaneous Choice (pre-selected cards, simultaneous signals, etc.)
• D: Known Probabilistic Chance (pair of dice, deck of known cards, etc.)
• E: Choice with Asymmetric Knowledge (pieces with values known to less than all parties, etc.)
• F: Fully-informed Choice (chess capture, etc.)

So, Chess and Go would be A-B-C-D-E-F3. Rock-Paper-Scissors would be A-B-C3D-E-F-. Stratego would be A-B-C2D-E3F2. Texas Hold'em would be A-B2C-D3E2F2. Candyland would be A3B-C-D-E-F-. Diplomacy would be A1B-C3D-E-F-.

This is a first draft, and I'm sure there's room for improvement, but I feel like you could look at the distribution of numbers by element to get a sense of the game. That said, my ordering or categorization seems lacking when it comes to Diplomacy.

An objective measure for Luck doesn't exist, so no such comparison can be made.

What is luck, in game terms? One definition is, the chance that you can win or lose a game, due to random chance and not due to decisions that you or your opponent made. A game of 100% pure luck, cannot have any decisions that you or your opponent make have any impact on who wins (i.e. You should be just as likely to win/lose by making random moves). A game of 100% pure skill, cannot have the result of who wins or loses determined by anything other than decisions that you or your opponent make, (i.e. Your decisions should cause you to win/lose more often than random moves).

Luck can be equated with randomness. An objective measure of luck would be:

The difference between how often you win a game compared to how often a random player would win a game.

While this is a completely objective, it isn't very useful. Some examples are in order:

## A very simple game

This game consists of a shuffled deck of 3 cards labeled 1, 2, and 3. The game is played over two rounds, with the deck shuffled between rounds. Each player chooses a number between 1-3, then the top card of the deck is revealed.

• If the card is >= to the highest guesses: the player with the highest guess scores the other players guess (if both are the highest, both score)
• If the card is < one player's guess: the other player scores the high guess.
• If the card is < both player's guesses: neither player scores.

Random Play: There are 27 possible outcomes in this game each round.

``````          ¦(1,1)¦(1,2)¦(1,3)¦(2,1)¦(2,2)¦(2,3)¦(3,1)¦(3,2)¦(3,3)¦Guess (Player1,Player2)
+-----+-----+-----+-----+-----+-----+-----+-----+-----+
R(1)¦ 1,1 ¦ 2,0 ¦ 3,0 ¦ 0,2 ¦ 0,0 ¦ 0,0 ¦ 0,3 ¦ 0,0 ¦ 0,0
R(2)¦ 1,1 ¦ 0,1 ¦ 3,0 ¦ 1,0 ¦ 2,2 ¦ 3,0 ¦ 0,3 ¦ 0,3 ¦ 0,0
R(3)¦ 1,1 ¦ 0,1 ¦ 0,1 ¦ 1,0 ¦ 2,2 ¦ 0,2 ¦ 1,0 ¦ 2,0 ¦ 3,3
``````

An expert player will never choose 3 for the first round. Doing so would result in their opponent scoring 3 points 33.3% of the time (ensuring a loss), drawing 4/9ths of the time, and only gaining points in 2/9 of the outcomes. An expert will never choose 3. If we compare a random player to an expert:

• Rnd1: E1,R1 = 3/9 Win, 2/9 Lose, 4/9 Draw
• Rnd1: E1,R2 = 5/9 Win, 4/9 Lose
• Rnd1: E1,R3 = 5/9 Win, 4/9 Lose

• Rnd1: E2,R1 = 14/27 Win + 6/27 Lose, 7/27 Draw

• Rnd1: E2,R2 = 3/9 Win, 2/9 Lose, 4/9 Draw
• Rnd1: E2,R3 = 14/27 Win, 8/27 Lose, 5/27 Draw

The expert player will open with Choose 2, and will win nearly twice as often as the random player. So, we will give this game a score of 60% skill (didn't work out the exact math for how often a random player draws, something close to 16%). So let us change the game. Instead of 3 cards, we will have 10 cards from 1-10. Now the random player has more bad choices. Instead of just 1/3rd of the choices being bad, now 4/10ths are bad. Is the game anymore skill based? An expert player will improve their win percentage against a random player, but what does that really mean?

We can make other rather trivial changes to the game like, not shuffling between rounds or removing several cards face-up from the shuffled deck (10-card version) that would increase an expert players win percentage versus a random player. These objective measures just don't have any meaningful value when comparing these trivial guessing games. If you take the fact that when trying to objectively measure the luck in these simple games, you can see the trouble of trying to apply these measurements to games with wildly variant forms of randomization.

• You've convincingly demonstrated how one particular objective measure is barely (or perhaps not at all) useful. However, this does not prove that all possible metrics are useless. One I've been pondering since I read your answer: how many lines of code does it take to program a computer to never lose? By this metric, Chess requires vastly more skill than the game you describe - which obviously matches our intuitions. But the 10 card version of your game vs. the 3 card likely requires the same amount of code or only slightly more, suggesting that the skill required has not increased. Jan 8, 2013 at 17:46
• @JoeGolton, It is impossible to prove a negative. Chess, and 3-in-a-row are both games of 100% skill, but trying to use this particular measure to determine how much more skill exists in that game than this game is useless. The only measure that is possible is a 3 number scale, all luck, all skill, or somewhere in-between. As to the 10-card/3-card game, I was only illustrating that trivial differences can make large changes in wins vs. random players (it still isn't clear what the optimum first move is to me 5, 6, or something else). I probably need more and better examples to show the jump. Jan 8, 2013 at 18:08
• Let's say it takes 500,000 lines of code to program a perfect chess player, vs. 20 lines of code to to program a perfect tic tac toe player. Why is this difference a useless measure? It may not be a truly great measure, because it probably doesn't say much if a game takes 20 vs 30 lines of code. But for wide swings of skill - doesn't several orders of magnitude more code strongly suggest more skill required? Jan 8, 2013 at 18:17
• For example, the win rate was about 30% for a random player in this 3-card game where Fluxx had a win percentage of 14%. So is this game more skill based then Fluxx? What if we add more players to Fluxx, or compare Fluxx to the 10-card game. As for lines of code, that is a terrible measure of complexity. Complexity itself might be a good measure of skill for a game, but now you have another problem. How do you measure complexity? Jan 8, 2013 at 18:32
• I agree with @user that lines of code is a terrible measure of complexity, but I think it's an even worse measure of luck. Jan 9, 2013 at 2:57

Consider the game of Risk. In a 1v1 game of Risk an expert player can beat random play very close to 100% of the time. If this were a measure of skill than risk would be 100% skill. But risk, obviously, isn't 100% skill.

Breaking down this question should be done differently. The question is how likely is it for a player who plays worse to beat a player who plays better(games with more than two results I'm ignoring for simplicity). This is conceptual and their is a lot of nuance in what exactly you are measuring but I think it really is what you are actually trying to get at.