As you noticed, rule 717 seems to be applicable here.
717.1. If a player realizes that he or she can’t legally take an action after starting to do so, the entire action is reversed and any payments already made are canceled.... The player may also reverse any legal mana abilities activated while making the illegal play, unless mana from them or from any triggered mana abilities they triggered was spent on another mana ability that wasn't reversed.
But I'm not sure it is, exactly. I'm no judge, but this rule states that it applies only to illegal actions, i.e. those which cannot be completed as declared. In your case, one might argue that there's nothing illegal about your casting of the Spine, and so this rule doesn't apply.
So we have to look to another rule. The one I find that seems to apply is
601.2. To cast a spell is to take it from where it is (usually the hand), put it on the stack, and pay its costs, so that it will eventually resolve and have its effect. Casting a spell follows the steps listed below, in order. If, at any point during the casting of a spell, a player is unable to comply with any of the steps listed below, the casting of the spell is illegal; the game returns to the moment before that spell started to be cast (see rule 717, “Handling Illegal Actions”). Announcements and payments can’t be altered after they've been made.
(emphasis added) This suggests that your friend was right and that you cannot stop casting a spell after having announced it, unless you are actually unable to perform the steps necessary to cast it. Note that the fact that you started paying for the spell is irrelevant; it's the announcement of the spell that locks it in.
In practice, most people don't adhere strictly to these clauses in the rules. I've known players to be allowed to take back spells in the middle of casting them - even after casting them in some cases - in regular REL tournaments, occasionally, and quite often in casual games. Since EDH is supposed to be a fun, casual format, as described by the official EDH rules,
Commander is designed to promote social games of magic.
It is played in a variety of ways, depending on player preference, but a common vision ties together the global community to help them enjoy a different kind of magic. That vision is predicated on a social contract: a gentleman's agreement which goes beyond these rules to includes a degree of interactivity between players. Players should aim to interact both during the game and before it begins, discussing with other players what they expect/want from the game.
House rules or "fair play" exceptions are always encouraged if they result in more fun for the local community.
I would think taking back a partially cast spell should be allowed there as well.
In light of this information I believe the technical term for what your friend is doing is "being a dick."
For what it's worth, Magic Online does allow you to take back a partially cast spell, at any point up until you finish selecting targets and paying the associated costs. But that situation is a little different because MTGO doesn't even reveal to your opponent that you intend to cast the spell until you have finished all the steps.