Unearth seems to replace where it goes when it leaves the battlefield, but it's already going into exile.
Yes, Nicol Bolas will be exiled and return transformed. The Unearth effect does not apply to that event.
Sedris, the Traitor King has the following ruling:
If a creature returned to the battlefield with unearth would leave the battlefield for any reason, it’s exiled instead — unless the spell or ability that’s causing the creature to leave the battlefield is actually trying to exile it! In that case, it succeeds at exiling it. If it later returns the creature card to the battlefield (as Oblivion Ring or Flickerwisp might, for example), the creature card will return to the battlefield as a new object with no relation to its previous existence. The unearth effect will no longer apply to it.
The exact definition of Unearth is in rule 702.83a:
Unearth is an activated ability that functions while the card with unearth is in a graveyard. “Unearth [cost]” means “[Cost]: Return this card from your graveyard to the battlefield. It gains haste. Exile it at the beginning of the next end step. If it would leave the battlefield, exile it instead of putting it anywhere else. Activate this ability only any time you could cast a sorcery.”
It is not going "anywhere else", so the Unearth replacement effect does not apply.
Nicol Bolas will be exiled and not return. Unearth doesn't care where the creature is going.
Unearth is a replacement effect. The event it replaces never happens:
614.1a Effects that use the word “instead” are replacement effects. Most replacement effects use the word “instead” to indicate what events will be replaced with other events.
614.6. If an event is replaced, it never happens. [..]
The event to be replaced is "Exile Nicol Bolas, then return him to the battlefield transformed". Exiling NB would cause him to leave the battlefield, so Unearth replaces the whole event and what happens instead is simply "Exile Nicol Bolas". Exiling Nicol Bolas this way does not return him, because the "return NB to the battlefield transformed" effect doesn't come from a triggered ability, it was part of the event that was replaced.