What happens if you cast a modal spell but choose illegal modes?

Question

For example, Izzet Charm is a spell with three modes, one of which is "counter target noncreature spell unless opponent pays {2}".

Suppose I cast Grizzly Bears and opponent responds with Izzet Charm attempting to counter the Grizzly Bears. That's illegal, of course. Which of the following is the correct result?

• Opponent gets to take back Izzet Charm and untap their lands.
• Opponent must pick one of the other two modes of Izzet Charm (which can legally be cast).

Bonus question: what if opponent went "Izzet Charm to counter your Grizzly Bears, in response do X"?

I'm assuming the answer varies based on REL, hence tagging with all RELs.

Answer 1

The opponent gets to take back Izzet Charm; see below for untapping lands.

The important rule here is 601.2 which states (in part):

If a player is unable to comply with the requirements of a step listed below while performing that step, the casting of the spell is illegal; the game returns to the moment before the casting of that spell was proposed

The steps listed below are:

• 601.2a: moving the card to the stack. This succeeds.
• 601.2b: making modal choices (and potentially other things). This succeeds, because the player hasn't named a target yet.
• 601.2c: Selecting targets. At this point, the player realises they have a problem - but nothing in the rules allows a "partial unwinding" at this point; if the player cannot name a valid target, the casting of the spell is illegal and 601.2 means we unwind to the moment before the casting of the spell, not to step 601.2b.

As for untapping lands: going through the strict sequencing, no lands needed to have been tapped at this point; they would be done in step 601.2g. If the player had tapped lands before they proposed casting the spell, they would not be able to untap them at this point.

I'm not enough of an expert to say what the penalties would be for this at any REL, other than the usual enforcing anything at regular REL is for exceptional cases only.

Answer 2

Phillip ia right about what happens in the game, the spell is illegal and couldn't be cast that way, so the game rewinds what it can to before it was cast if possible (some mana sources cant be rewound like Selvala, Explorer Returned so mana from those sources is just left in the pool), this includes any other actions taken on stack with the illegal spell, they are undone too.

As for the tournament/REL based actions, a player is generally just reminded to be more careful if the error is believed to be an honest mistake, particularly at lower rules enforcement levels. For your bonus question in particular this could lead to disqualification, casting a spell illegally and then immediately responding to it makes it seem you know the spell was illegal and you're trying to keep your opponent from noticing, that looks like cheating and at any rules level cheating is grounds for disqualification from the event.

Answer 3

The answer is pretty much the same for casual games, Regular REL, Competitive REL, and Professional REL, but there are minor differences.

For casual games, the rules simply state that the action is reversed. That means, you undo the action.

CR 729.1 If a player takes an illegal action or starts to take an action but can’t legally complete it, the entire action is reversed and any payments already made are canceled. No abilities trigger and no effects apply as a result of an undone action. If the action was casting a spell, the spell returns to the zone it came from. Each player may also reverse any legal mana abilities that player activated while making the illegal play, unless mana from those abilities or from any triggered mana abilities they caused to trigger was spent on another mana ability that wasn’t reversed. [...]

At Regular REL, the judge rewinds the game to just before the spell was cast. From the JAR:

A player makes an in-game error not mentioned above
[...]
If the error was caught quickly and backing up is relatively easy, you may undo all the actions back to the point that the illegal action happened.

At Competitive and Professional RELs, the casting of the spell is reversed by "simple backup". Your opponent (the player who made the error) receives a Game Rule Violation. If you fail to catch the error in a reasonable time then you receive a Failure to Maintain Game State Violation. You didn't say how much time passed, but I'm going to assume you realized within a few seconds, and so you would not receive a violation. From the IPG:

IPG Section 2.5, Game Play Error — Game Rule Violation

Additional Remedy
First consider a simple back up (see section 1.4)

IPG Section 1.4, Backing Up

Some remedies state a simple backup may be performed. A simple backup is backing up the last action completed

As for your "bonus question", it does not change the situation at any level of play. Even though your opponent has taken two actions now, none of the other remedies for Game Rule Violation are more suitable than a full backup of both actions.