The bigger problem here isn't the auction. It's that your brother didn't have his money in view, so you couldn't even guess how much he had. Tournament rules disallow this, though they don't require the player to disclose the exact amount of cash they have, nor to organize their cash in a way that this is easy to determine.
I could add that technically you would have been able to figure out exactly how much money he had by keeping track of all his transactions, but I don't blame you for not doing this. I certainly would not be able to do it. Still, you can often estimate it; if they mortgaged properly recently or sold a house, you know they're not exactly flush with cash. And if you can see their properties, you can add up their values and get a rough idea of how much the player has spent over the course of the game.
But I think the best solution is to agree to keep all money in view and to allow overbidding. If your opponent tricks you by overbidding when you can see more or less how much money they have, that's on you. Instead, turn the tables and make them pay an enormous debt!