I don't think that there's any single thing designed for this; it's quite a small niche. However, you might be able to piece something together.
To me, a wiki sounds like a very natural fit for hosting and editing the rules for your game. But the general idea seems solid: you can have a page or many, organized how you need, for the rules, include pictures when necessary. You can easily create offshoot discussion pages, linked to as appropriate, group them into categories/listings, put boilerplate at the top to indicate status of the discussions, and so on. An alternative with a steeper learning curve and and startup effort for contributors would be to simply keep your rules in text format in a hosted version control repository and then use other features of the hosting (particularly issue tracker) to do the other things.
You'll probably want to do a lot of iteration and playtesting. I've been pretty impressed with what I've seen people pull off with Vassal modules. Vassal is a generic game engine, which lets you create boards, pieces, cards, tiles, and so on. You can play the games with others online. There's a designers guide as well as a FAQ page on creating modules. This would give you the platform you need to test things out. You'd probably want to store the module in some sort of version control; unfortunately it's an all-in-one file, edited within vassal, so you won't be able to much in the way of branching and merging, if there are many collaborators. Depending on the scale of your project, it might be sufficient to simply have file hosting, and store numbered versions, or you might want more.
Note that SourceForge might actually be a great candidate for all of this! It supports project wikis, file hosting (for your vassal modules), and revision control repositories (perhaps for your rules, if you don't want them on a wiki). You just have to decide how you're going to use it. Just think: a board game isn't that different from a computer game. It's just written in a "programming" language that humans, not computers, understand!
(I know this isn't a terribly complete or specific answer; perhaps others can contribute more.)