I've played this game several times, but I am still confused when it comes to supply and the amount of armies you can have. Are you allowed to go over your army limit, but when a supply card comes up, you just have to diminish your armies? For example, if I am playing as Lannister and I start the game with 2 supply which means I can have one army of 3 and two armies of 2, am I allowed to have an army of 4, but go down to three if the supply card comes up?
No. The supply cards are when you adjust your position on the Supply track, but your armies must always respect your supply. This is called out specifically in the supply section of the rulebook on page 8:
A player is never allowed to take any action in the game that would cause him to exceed his actual supply limit as dictated by his position on the Supply track (such as mustering, marching, or retreating).
The mustering section on page 9 has a blurb that reinforces this idea:
A player may never muster a unit that would create or expand an army beyond his actual supply limit. In other words, if mustering a new unit would cause a player to have more (or larger) armies than allowed by his position on the Supply track, that unit cannot be mustered.
Thus, in your example, you cannot have an army of 4 while you remain at position 2 on the supply track. This also prevents you from swapping out say a siege engine for a footmen - once you recruit a unit, you're stuck with it until it dies.