Looking at rules here I think your interpretation is correct. Although I suspect from the way you say "its a brilliant move, and I don't see anything specifically preventing this" makes me think whoever you did this against didn't agree!
anyway the rules quotes....
On a players turn one of there options is
Pass and discard 1-3 face-down cards from your hand..... then draw 1-3 cards.
The note underneath clarifies
When the pile is used up, no cards are drawn.
So if there were a situation where a player is to draw a card and can't they simply don't get a card. So yes you can discard cards knowing you can't draw back and deliberately shrinking your hand size.
The rules for swapping state :-
The player who plays this card chooses one other player and exchanges hands with them. Discard the action card, then the other player gets to draw 1 card!
So you swap your remaining hand of zero cards with the other player. They are unable to draw a card as non left.
So yes the move seems entirely legal based on that.