I´m hooked on Carcassonne but feel that after you learned the basic tactics it is mostly a game of luck. This has me wondering, when designing board games how would one go about calculating the luck factor? Are there any known good methods?
A reasonable empirical measure of luck is the probability that the best player can beat a good player in a given game. So if you can get data about a large player population (I have done this with Race for the Galaxy), you can get an implicit measure by looking at the Elo rating difference at various skill percentiles.
On the other hand, this only tells you about the amount of luck in the game as a whole, not in a particular game.
It would depend on the game. Certainly for games that involve rolling dice, the probability of certain events happening can be easily calculated. For games where dice are thrown or cards or drawn, there are some games where some outcomes can really change the flow of the game. There are also games where random events actually have minimal affect on the outcome of the game.
Monopoly has a strong aspect of how luck factors into the game. Where you pieces land, makes or breaks the game for you. In Settlers of Catan, luck exists, but good players do their best to maximize their odds so the luck of the dice is less of a factor for them. Chess, is completely a game of skill, there is no random factors are all.
For games that combine luck and skill, experienced players make moves that are likely to benefit them the most. Thus, their skill helps them depend less on luck to win. That doesn't mean bad luck still can't keep them from winning. You have to analyze how randomness created by dice or card drawing helps or hinders players when designing the game in order to determine how much luck affects the game.
There are books on the subject of game design you can look into and an entire course on the subject matter at one university.
Some books I've seen on the subject are:
Rob's answer on using a large population database to evaluate the roll of luck is absolutely the way to go--assuming this data is available. But the question also seems interested in how to evaluate the roll of luck during game design, when no such data is available. In this context, the question is about how to model or simulate a game to test the mechanics. The answer depends on the type of game being evaluated.
Trial and Error
The least satisfying method because it's so slow. Basically this means play testing to see if something rewards luck too much (based on the designers desire for luck/skill balance). Unfortunately since we're talking about luck you might have to play a lot to get a good feel for the balance. Basically you're generating Rob's database yourself. :(
This is more appropriate for modeling portions of games that involve dice--or a similar randomization mechanism. In a game like Arkham Horror (or even Risk) it's easy to test out individual battles (or write a computer program to do it) and quantify the impact of a change to the rules. This might be useful for making sure a custom Arkham character isn't too weak or too powerful, or if you were experimenting with a different Risk combat engine.
The easiest to implement computer-based method is Monte Carlo simulation: use a random number generator to simulate a single battle, then re-seed the generator and run it again; repeat tons of time and calculate statistics on the results. Such tools exist for games like Axis and Allies, but if you were savvy you could code one up yourself.
I think I just made this term up, but it's supposed to explain looking at a certain game situation and determining to what extent luck could influence the outcome--usually by bounding the best and worst case outcomes and seeing how large the variance is. Similarly, you can evaluate the extent to which the player can tailor their game play to insulate themselves from risk. Games that enable or encourage conservative play and that ultimately reward the more skilled player can greatly reduce the influence of luck.
For example, in a game of Axis and Allies where multiple aircraft are being shot at by an AA gun, the swing between best- and worst-case outcomes is huge: its varies from total annihilation to no damage. Obviously in this case luck can play a huge factor. That said, an experienced player over-matched verses their opponent probably shouldn't use aircraft to attack territories defended with AA guns. They know if they wait it out they'll win eventually and shouldn't take the risk of letting the dice even the playing field.
Back to the root of your question, one way I've found to diminish the role of luck in Carcassonne is to have all players draw a "hand" of tiles (3 to 5 seems to work well), and then on each turn you draw a tile and then play any tile from your hand. When there are no more tiles to draw at the end of the game, you just play one of the tiles from your hand until everyone has played all of their tiles.
Luck will still play a part, but this gives the players a little more control over which tile they play at a given time.
Some other variants that affect the luck factor in Carcassonne are to deal out all of the tiles at the beginning of the game, play with the remaining tiles face up so you can choose exactly the tile you want as long as it's still available, or begin the game with a "draft" where you pick the tiles you want to have during the game.