The rules are not definitively on your side but you definitely should have appealed the ruling. According to the MTG Infraction Procedure Guide section 2.3, the default penalty for drawing an extra card is a game loss. However, it also says:
If the player received confirmation from his or her opponent before drawing the card (including confirming the number of cards when greater than one), the infraction is not Drawing Extra Cards.
and your opponent's saying "Draw" and pointing to you arguably counts as confirmation that you should draw for your next turn. This argument might be difficult to make though because "Draw" is not listed explicitly in the Tournament Rules list of shortcuts (section 4.2) for passing priority until your opponent's turn.
However, there is another argument to be made that you should not have lost. The Infraction Procedure Guide Drawing Extra Cards section says
A player illegally puts one or more cards into his or her hand and, at the moment before he or she began the instruction or action that put a card into his or her hand, no other Game Play Error or Communication Policy Violation had been committed.
One relevant piece of free information (as specified in section 4.1 of the Tournament Rules) is "The current step and/or phase and which player(s) are active", and the Infraction Procedure guide says in the Communication Policy Violation section:
Players may not represent derived or free information incorrectly
I would argue that your opponent did just that when they said "Draw", which means that the judge should have followed the procedure in this section instead:
If the situation is simple enough to safely back up without too much disruption to the course of the game, the judge may get permission from the Head Judge to back up the game to the point of the incorrect information. Each action taken is undone until the game reaches the point immediately prior to the error. Cards incorrectly placed in hand are returned to the location in the zone from which they were moved (if the identity of the incorrectly drawn card is not known to all players, a random card is returned instead). Once the game is backed up, it continues from that point.