At first, the additional complexity definitely feels like a downside. But once you play enough, the decisions present in the family version become much more intuitive and faster. At that point, the additional complexity becomes a good thing: deciding which minor improvements and occupations to use (and when) becomes a big, interesting decision. And it's not just wasted time making decisions; using them well has a large effect on your score. In fact, they can be much more meaningful than the worker placement decisions you might agonize over in the base game.
So in addition to making the game more interesting to play, they'll also make it a more interesting competition. Once everyone's pretty good at the family game, you'll start to lose control of your destiny: everyone will be able to pick their actions and plan fairly well, so it'll get harder to win by virtue of good strategy. But throw the minor improvements and occupations, and suddenly there's again a lot of room for good (or bad) strategic decisions.
On top of that, they also add a lot of variety and replayability. Since you don't see the same set of cards every game, you're forced to adapt your strategy to suit the cards you have, giving you different decisions to make each game, not just more decisions.
So in a way, the things you dislike about them now are some of the good things that they add to the game: they add additional decisions, making for a deeper game. Right now that's a bit overwhelming, and you're not able to take full advantage of it, but if you play enough it'll become a positive, defining part of the game.