You are correct, when you choose to decline your current race, the previous markers of the previous race in decline are immediately removed and not scored (Unless of course the previous race in decline has the Spirit special power.)
See Entering in Decline on page 6 of the rulebook. Emphasis mine.
Each player can only have a single race In Decline on the map
at any given time. If the player still has tokens from an earlier
In Decline race left on the map, those are all immediately
removed from the map and placed back in the storage tray,
before flipping the new tokens into Decline.
The Race banner of the now vanished race is placed at the
bottom of the stack of Race banners, or in the lowest empty
slot in the banner column, if any. The same is also done when
the last token of a race In Decline is wiped off the map as the
result of their last Region being conquered.
The player can make no conquests during the turn his race
goes into decline; his turn ends immediately after scoring! He
gains 1 Victory coin for each Region his newly In Decline tokens
occupy, but unless stated otherwise, he scores no Victory coins
from his now In Decline Race banner power or discarded Special