Odds for non-Brawl using the pre-Core 2020 mulligan rules (according to Ikegami's editing)
The probability of having a desired card in the hand,at the first draw of seven cards, is, by removing the commander, (7/59) * 100 = 11.8644%
If, at the first draw, the desired card has not arrived, then, according to what is stated in the question, you will draw again using the mulligan; and then the probability of reaching this point, which is obviously greater than 7/59, is as follows -
first of all, this probability is due to the probability of not drawing the desired card to the first set of seven cards, which is: (52/59) * 100 = 88.1356%
this value is then multiplied by the probability of drawing the desired card now, which, having returned the 7 first cards in the deck, but now having to draw 6, is (6/59) * 100 = 10.1695%
Consequently, the probability of one mulligan, and then of finding the card sought in the six cards of the second draw is (52/59) * (6/59) * 100 = 8.9629%
If the card you are looking for hasn't arrived at this point, then, as specified in the question, you will mulligan for the second time.
So, this probability is obtained by multiplying the value of the probability of not drawing the card among the first seven (52/59), multiplied by the probability of not drawing the card even the second time (53/59), multiplied by the probability of draw the desired card now, but drawing only 5, or 5/59.
Consequently, the probability of mulligan twice, and then of finding the card sought among the five cards of the third draw is: (52/59) * (53/59) * (5/59) = 6.7%
At this point, the total probability of finding a desired "key-card", being able to make two mulligans, is given by the sum of the three possibilities of finding the card:
(possibility of finding the card after the first draw) + (possibility of finding the card after the first mulligan) + (possibility of finding the card after the second mulligan) =
11.8644+ 8.9629+ 6.7 = 27.5273%
Well ... that's all ... at least I think.
I hope there are no typing and/or math errors !!!
Odds for Brawl using the Core 2020 mulligan rules
The result shown above works only if, in making Mulligan the first time, the player draws 6 cards (not 7), and, in making Mulligan the second time, if the player draws only 5.
In addition, the result is valid only without looking at the cards that are placed at the bottom of the deck.
But, as I suspected, this does not accord with the current Mulligan rule...
I think I will leave this answer here just the same, also because, even if it refers to a mulligan rule that is not now applied, its procedure is correct.
To prove it, I will verify that his calculations are correct, rewriting them taking into account my own procedure, but considering the correct Mulligan rule.
The probability of having a desired card in the hand, at the first draw of seven cards, is, by removing the Commander, (7/59) * 100 = 11.8644%.
The probability of having the card at the first mulligan is:
(52/59)*(7/59)*100 = 10.4567652973%.
The probability of having the card at the second mulligan is:
(52/59)(52/59)(7/59)*100 =9.21613212646%.
(This means that the player is able to, according to the current Mulligan rules, look at 7 cards, and choose for himself the one that will be placed at the bottom of the deck, according to his needs. And it is always for this reason that the calculations agree based on the fact that, making Mulligan, for the purpose of searching for a "key card", it is as if the player was still drawing seven cards).
The probability of having the card at the third Mulligan -
(or when, granting the answer to the fact that the asker claims to want to stay,at most,with 5 cards in hand, after the Mulligans, which, however, are 3, and therefore not 2) -
is:
(52/59)(52/59)(52/59)*(7/59)*100=8.12269272162%
(And this also means that the player is able to, according to the current Mulligan rules, look at 7 cards, and choose for himself the two that will be placed at the bottom of the deck, according to his needs).
At this point, the total probability of finding a desired "key-card", being able to make three mulligans, is given by the sum of the four possibilities of finding the card. And is:
11.8644% + 10.4567652973% + 9.21613212646% + 8.12269272162%= 39.6590927216%.
This result is absolutely close to the correct one, already calculated in the right answer, higher.