The text specifies that I tribute the target of soul exchange "as if [I] controlled it". When I tribute a monster, it normally goes to my graveyard. Is that because I control it?
To answer to the question in your title, all tributed cards go to their respective owner's graveyard. Now,
When I tribute a monster, it normally goes to my graveyard. Is that because I control it?
Yes, it goes to your graveyard, but it is because you own it.
Two key terms here:
Owner: The player who owns the card, regardless where that card may be.
Controller: The player who has a specific card under their control. In other words in which side of the field that card is.
When a card is going to be destroyed, tributed, sent, returned, etc., it goes back to it's owner's possession and to the respective destination the card was redirected (owner's GY, deck, hand, etc.)