So if Jace, Telepath Unbound is returned to your hand, does it turn back into Jace, Vryn's Prodigy to be recast or does it in effect go to your graveyard because Jace, Telepath Unbound cant be recast?
When it enters a new zone, it resets back to its default characteristics. Namely as a creature card. So, you can then recast it as Jace, Vryn's Prodigy.
This is based on two rulings on Gatherer:
While a double-faced card isn’t on the battlefield, consider only the characteristics of its front face.
A double-faced card enters the battlefield with its front face up by default, unless a spell or ability instructs you to put it onto the battlefield transformed, in which case it enters with its back face up.