That is correct: The army limit is based on your position on the Supply track, but the Supply track is not automatically adjusted during any of the game phases. You would only adjust the Supply track (and thus your armies) when the a game effect explicitly commands it (i.e. a Supply card is drawn).
From page 8 of the official rules (second edition, but it plays the same as first edition in this case):
- Though a player may gain or lose Supply icons on the game
board throughout one or more Action Phases, his armies
are reconciled only when the Supply track is adjusted (via a
“Supply” Westeros card, or other game effects).
(Note: The rules also calls out "other game effects" but off-hand I have no clue what those are, or if they're second-edition-exclusive. From first edition, I don't recall ever needing to adjust the Supply track except from a card explicitly commanding me to.)
The "Supply Example" on that same page also calls this out more clearly (emphasis mine):
Lannister has recently lost control of Riverrun and
Seagard to Greyjoy (each of which contains a single Supply
icon). During a later Westeros Phase, a “Supply” card is
revealed. Due to the loss of these areas, Lannister must now
adjust his actual Supply from 5 to 3.
Before the “Supply” card is drawn, Lannister has four armies
of 4, 3, 2, and 2 units.
Of note, the bolded text mentions that Lannister still had four armies before the Supply card is drawn (despite the territories being lost before then, given that the Supply card was drawn during "a later Westeros Phase").