I feel like the current answers don't touch on the most important aspects of lifegain & its role in deck-building, so I'm adding this answer.
First: Most Magic decks win by reducing the opponent's life to 0. This means lifegain is always helpful at preventing them from winning. However, it does not mean lifegain is effective.
To illustrate, consider this deck. It's a modern Jund deck and the way it kills the opponent is via some combination of creature beatdown and possibly burn spells. Each of their creatures can attack for a lot of damage - Kroxa for example is 9 damage a turn if you can't discard a non-land card to it. This means that if you want to beat Jund using lifegain, you need to gain a lot more than 9 life a turn. This is not really plausible. It's why pure lifegain cards like Stream of Life are generally considered as bad.
However (and this is worth emphasizing), if you could gain ~40 life a turn,* then you will beat this Jund deck. They can't do enough damage to kill you.
The problem is, of course, how are you going to gain 40 life a turn? You need it to be repeatable as well, since you cannot count on drawing a lifegain card every turn. There are ways to do it (see example later), but they generally don't arise incidentally. You build your deck explicitly to gain so much life that most decks can't kill you.
*This is >9 because the Jund player will be attacking with more than just Kroxa every turn.
Second: Jund is a creature deck. There are decks that win by damage, but are not creature decks. Here is an example. This deck is a burn deck, which wins with haste creatures followed by direct damage spells aimed at the opponent.
The big difference between burn and Jund is that burn's damage is not repeatable. The burn spells are one-shot effects. Most of them deal 3 damage, which means that if you can gain 3 life a turn, they can't kill you. It also means that if you gain 6 life at some point in the game, you effectively gain two extra turns to kill your opponent with.** Lifegain is at its most effective against burn decks for this reason. Your one Feed the Clan is worth 2-3 of the burn player's cards. It is why people play pure lifegain in their sideboards when they expect to play against burn.
**This is because they draw ~3 damage worth of burn every turn, so if you gain six life, they need two turns to draw enough burn to negate your lifegain.
Third: In specific situations near the end of a game, lifegain can be the difference between winning and losing. For example, say you have a 10/10 creature and opponent has five 1/1 tokens. You have control of the board, but life totals matter more. If you're on 1-4 life, you're in critical danger of dying. Similar to this is when your opponent can no longer attack, but they might still have some specific cards that win the game. Examples of such cards are Sleep and Banefire. Lifegain goes a long way towards making the game go from winning to won.
Based on the above we can answer all your questions:
Are there any decks where cards whose primary purpose is life gain really shine?
Yes. Some examples are already given in other answers. Martyr proc is such a deck. So is Soul Sisters. The first deck is an extreme "gain so much life a turn opponent cannot kill you" kind of deck. The second uses lifegain as a way to buff its creatures like (Ajani's Pridemate). Lifegain is not its main strategy, beating down with big creatures is, but it is an incidental benefit.
What combos or strategies upgrade them from average cards to truly great ones?
If the primary purpose is lifegain, then you need to gain in the vicinity of 40 life a turn. Martyr proc is the only strategy I'm aware of that aims to gain that much life a turn. Most other pure lifegain strategies instead go for "infinite life". There are a variety of infinite life combos in Magic, which you can Google for. I'll only mention one example: if you have Heliod, Sun-Crowned and Spike Feeder in play, then you can activate Spike Feeder, gain life, and use Heliod to put the counter back on Spike Feeder. Loop this enough and you have infinite life.
How can I tell if I should add some life gain cards to my deck?
If your deck isn't built to take advantage of life gain - either by incidental synergies ala Soul Sisters, or by directly gaining enough life to never die ala Martyr proc - then you probably should not add life gain cards to your deck. The exceptions are:
- You are playing against a burn deck. In this case your lifegain cards trade directly against their damage cards, and if your lifegain cards are more efficient (they probably are), then you come out ahead.
- Another case is if your lifegain card does something else while possibly gaining life. Examples are Cling to Dust and Huntmaster of the Fells. The first card is usually used as graveyard hate (i.e., a card that counters specific graveyard-dependent cards from the opponent) that also acts as a late-game mana sink. It is incidentally able to gain you 3 life if you need it. The second card is just a generically good card - it's a 4/4 for 4 mana with upside - that is capable of incidentally gaining you life.
- The final case is if you are the defensive deck and are playing against a deck that still has a way to win even after losing control of the board (most well-built aggro decks will be able to threaten this). Ideally the cards that stabilize your board are also capable of gaining life. Examples of cards in this category are Lyra Dawnbringer and Tolsimir, Friend to Wolves.
(Note even in the last two cases, the cards do things other than "gain life".)