Are there any open source programming libraries, perhaps in Python or Lua for example, for modeling table top games to help with design and testing work?
Currently, there are none.
You might be looking for Monte Carlo simulation tools, but this might be a very, very broad description of tools for your purpose.
I've been thinking about the subject for a while now, and I've come to believe that the tools that are the best fit to the job are the business process management (BPM) tools, in particular the BPM modeling and simulation tools. They allow you to describe a turn as a processes (i.e., a series of steps that a player should take when he is called to act) and simulate many matches to collect and analyse critical indicators. You should also take a look on business rules management systems (BRMS) that allow you to easily set rules that might help you emulate smart decision making.
A decent open source BPM tool is jBPM for BPM and Drools as the BRMS. There is also Activiti. Sadly, both tools lack simulation capabilities. If you use BPMN 2.0 to model your game as a process, you can try simulating with bpmn-simulator, but then you lose the BRMS integration that is important to simulate player decision making.
Anyway, I must emphasize that this is the best approach I could think of with today's open source software. This does not means that this is a good approach. In fact, I haven't tried it myself yet. Still it makes sense to me to model a game as a very complex process filled with rules that could be simulated with adequate tools.
If you're interested in testing the aesthetics and flow of the game (rather than fully automated simulation for balance purposes), then I'd look at Tabletop Simulator.
From their site:
Tabletop Simulator is the only simulator where you can let your aggression out by flipping the table! There are no rules to follow: just you, a physics sandbox, and your friends. Make your own games and play how YOU want! Unlimited gaming possibilities!
For what it's worth, this is actually a pretty difficult problem to solve. You might have a look at General game playing on wikipedia, and the things linked from there. There are a few engines that people have built that you could look into. For your purposes you don't need the AI to find optimal strategies, of course, but even just doing well enough to come close enough to human play to measure what you want to measure is tough - I'm not sure whether or not they'll be good enough.
I wouldn't count on much of anything existing beyond what you see there, and find searching for similar things. There's a reason they're still holding contests!
You can model subsystems of a game for calculating probabilities using the Lea library for Python.