4

I want to measure the difficulty of different layouts in Klondike-Solitaire. My question is, what are the parameters that determine the difficulty of a layout? enter image description here

By layout I mean the permutation of the cards. The difficulty is also a matter of the rules (deal-1 or deal-3 cards from the stock to waste (talon), assume the rules are fixed on one of the options)

At Microsoft Solitaire Collection, there are 6 difficulty levels. From my personal experience, the differences between the first 5 are minor compare to the huge difficulty difference between the second hardest (Master) and the hardest level (Grandmaster).

I thought about the following parameters:

  1. Solvable?
  2. number of moves till solution
  3. number of undo's
  4. time taken (for a given user/solver)
2
  • 1
    Knowing which percentage of folks who finished a level vs the population who tried that level would seem to give a good approximation of difficulty.
    – chicks
    Nov 26, 2020 at 23:19
  • 1
    @chicks, that might be the easiest and most accurate solution. However it needs many players to play that level. I want to get a difficulty estimation before letting players to play, then adjusting my estimation based on their results.
    – Cohensius
    Nov 29, 2020 at 16:49

2 Answers 2

7

From a human perspective I suppose one's interpretation of difficulty is quite subjective: what one person might find difficult another may find easy. If, for our purposes, we use a rough definition along the lines of, "how much time the average player spends on a layout", then there are a few interesting things we can say about the relationship between a layout's difficulty and its general features.

As your first point suggests, the solvability of the layout (i.e. does a sequence of legal moves exist that results in a win) is obviously key. Let's consider this case first:

Unsolvable Layouts

If a layout is not solvable then in one sense it's infinitely difficult. However, even for unsolvable layouts, the depth of the search tree can reflect a kind of difficulty.

In the extreme case, if to begin with there are no legal moves available in the tableau (main cards) or the stock, then although the layout isn't solvable, in a sense it is easy because you can immediately give up. Alternatively, a layout may have a huge number of promising moves, only to turn out to be unsolvable in the end.

Using a tool like Solvitaire (full disclosure: I am one of the authors of Solvitaire) can identify unsolvable layouts and record the number of unique states that needed to be searched to prove that there's no solution (often a very large number of states!). Another good solver is Klondike-Solver, although I'm not aware of what metrics it reports.

This isn't a perfect measure by any means. Humans aren't computers, and our brains probably aren't using depth-first search in the way a solver might. Without looking at human gameplay data though, this is probably the best heuristic of difficulty for unsolvable layouts.

Solvable Layouts

As was the case for unsolvable layouts, if one has access to human gameplay data then that's almost certainly going to be the best source of understanding how difficult humans will find a layout.

Let's assume though, that we don't have this data available, but we do have a computerised solver. Just looking at the static cards in the starting layout is unlikely to tell us much, but by running the solver we can deduce more about a deal's difficulty (for a human).

Here are some heuristics one could use instead:

Size of Search Tree

For a computer this is important, but in the solvable case it may not tell us much about human play. For instance, the search tree for a layout could be huge, but have one solution that's very obvious to a human. We would not want to classify such a layout as difficult just because it has a large search tree.

Depth of Shallowest / Best Solution

Based on the above logic, the depth of the shallowest (i.e. "best") solution in the search tree is perhaps a better measure. However, this still may not be ideal. We might label a layout as easy because it has a shallow solution, but if this solution is extremely hard for a human to spot then our heuristic will be misleading.

Number of different solutions

This heuristic could be quite accurate, although it's hard to say. Having a lot of different solutions in the search tree would suggest a human player is likely to come across a solution sooner, but there is no guarantee. This would certainly be something worth exploring further.

Number of states searched until first solution

This is probably the most obvious heuristic: how many states did it take before the solver came across the first solution? However, this measure can be deeply flawed. There is no guarantee that computerised solvers search in anything like the kind of order a human would search in (Solvitaire certainly doesn't).

Say move "A" obviously leads to a solution in a few moves, whereas move "B" leads in a very different direction. There is no guarantee the solver picks "A" first; it might instead pick "B" and try thousands or even millions more moves before it backtracks to the point where it thinks to try "A".

To make this kind of method work one needs a policy for which moves to try. For Solvitaire, which just wants to explore all possible moves, this doesn't matter, but for determining difficulty this is very important! If one could come up with a policy which reflected closely how an average human would play, then the number of states searched until the first solution would be an excellent metric. But coming up with such a metric is hard, complex work.

What Is A Move: Dominances and K+

In all of these discussions we've talked about moves and states. But how we define a move in Klondike is actually not trivial.

Firstly, consider the stock. The standard rules require we turn over 3 cards in one go. A consequence of this is that at different times there are different groups of cards that are effectively "available" to us.

Bjarnason et al. use a solver which can move any of the available stock cards into the tableau in a single move (they call this the K+ representation). For a human, this is like saying "I remember there's a 5H in there, so I'll just loop back over the stock to get it". From the point of considering difficulty, one may wish to consider this sort of thing to be a single move, rather than several separate moves.

Finally, in our paper on Solvitaire we consider something called dominances. These are cases in which we can prove that one of the available moves is guaranteed to be a good move. For instance, sometimes it can clearly be shown that "putting up" say an ace, is always the right thing to do. This is often obvious to a human player and one might not wish to count this as a move (Solvitaire doesn't count this as an extra move).

In summary:

  • One must first consider solvability
  • Then if a layout is solvable, one can begin with easier, but probably less accurate methods like the number of possible solutions, or the depth of the best solution
  • Finally, and only if one can craft a good (human-like) policy for moves to try at each search step, a better approach may be to measure the number of states the solver searches until the fist solution
1

Easiest solution: Let people play those layouts

The best estimator for how human precept difficulty is, well... humans. If you can let human players play then their win-rate[%] along with time taken and backtracks should produce a very accurate difficulty function.

Reverse engineering spots the hardest difficulty level

I manually solved 3 layouts from each of the 6 difficulty levels and store their initial card layouts. Then used Charlie's Solvitaire to solve those layouts. I Hoped to be able to distinguish between all 6 difficulties using the output of the solver. However only the Grandmaster level produce distinct outcomes. The differences between the others 5 levels in terms of Solvitaire output were somewhat ambiguous.

Second attempt: Compare optimal solutions

Solvitaire finds a solution, I thought that a better comparison will be to compare the optimal solution, by optimal I mean least number of moves. So I adjusted Solvitaire from DFS to search ID-DFS (start with a solution from DFS, then use bounded-DFS that search for a solution with less steps. Repeated with smaller and smaller bounds until there is no solution, which means that the last solution is optimal).

Comparing optimal solutions did not produce better separating results.
However I think that this process could work if it uses more data and better difficulty function.

Solvitaire output is the following parameters:

  1. is solve-able?
  2. States searched
  3. Unique states searched
  4. Backtracks
  5. Dominance moves
  6. Final stats in cache
  7. final buckets in cache
  8. maximum search depth
  9. final search depth
  10. time taken

I used parameters #1,2,3,5,9,10 from the DFS solution and #1,2,3,5,9 from the ID-DFS

Solvitaire (DFS) enter image description here

I modified Solitaire to find the optimal solution using iterative-deepening (IDDFS) enter image description here

Made a difficulty function using the marked parameters: enter image description here

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .