If I cast a spell which then turns out to be un-playable, what happens to my spell? E.G.: Casting a spell on something that has Hexproof and can't be targeted in the first place. Do I simply place my spell back in my hand? Do I untap the lands used to pay for that spell?
If The Spell Never Had Any Legal Targets
You cannot cast a spell that requires a target unless there is a legal target for the spell.
If you try to cast a spell that requires a target, and discover that there is no legal target for that spell, it is considered an illegal action.
The comprehensive rules on handling illegal actions:
So, you would essentially "rewind" the game state, untapping your lands and returning the card to your hand. Unfortunately, this misplay does mean that your opponent has earned some free information (i.e. they now know one of the cards you have in your hand).
If The Spell's Target Becomes Illegal After Casting It
In this case, your spell has already been cast, because you chose its target while casting it and at that time the spell had a legal target, but during the resolution of the spell, that target is discovered to no longer be legal. This can happen, for instance, if the card Ranger's Guile is cast in response to your spell, giving the target of your spell Hexproof.
When your spell is resolving, it checks again to see if the target is still legal. In this case, since the creature now has Hexproof, it will be determined that the target is not legal, and the spell will go to the graveyard and have no effect. You will not untap your lands. You simply lose that mana and that spell's effect; your spell was countered.
The relevant section of the comprehensive rules states (under the heading "Resolving Spells and Abilities"):
As others have quoted, rule 717.1 applies:
But unlike what callahan09 says, if there are other legal targets you are not forced to name one of them. This post from a Level 2 judge agrees. (Note that the O-ring case is different because Oblivion Ring itself doens't target, only the ability, and the ability is mandatory.)
pretty clear quote for your situation taken from the comprehensive rules.
protected by ire_and_curses♦ May 30 at 1:39
Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.
Would you like to answer one of these unanswered questions instead?