Ragavan's ability does not use the word "target", so it doesn't actually target any player.
But I think what you meant to ask is whose library you get to exile a card from. When Ragavan (is unblocked and) deals damage, he deals it to one player: the player who he was attacking. This is locked in when you declare Ragavan as an attacker.
810.9. Damage, loss of life, and gaining life happen to each player individually. The result is applied to the team’s shared life total.
The effect of the damage is that the team's shared life total decreases, but that doesn't mean the damage was dealt to the team, or to both players. The damage was still dealt to only the player who was attacked.
120.3. Damage may have one or more of the following results, depending on whether the recipient of the damage is a player or permanent, the characteristics of the damage’s source, and the characteristics of the damage’s recipient (if it’s a permanent).
120.3a Damage dealt to a player by a source without infect causes that player to lose that much life.
In a two-player game, technically something similar happens: an unblocked attacking creature deals damage to a player, and the effect of that damage is that the player's life total decreases. (With some exceptions as listed in the rest of rule 120.3, e.g. if the creature has infect, the effect of the damage is that the player gets that many poison counters instead of their life total decreasing.) You just don't typically think about the difference between the damage and the effect of the damage.