No, your Fortress does not act as a unit, and the foriegn unit does not need to defeat the Fortress. They only must survive the first round combat. The rules for Fortress (page 6) tell you, "During the first combat roll of a battle: +1 die for combat and Cancel one hit." So, your Fortress allows you to roll a single die, and if that defeats the foreign Army unit, combat ends with no winner or loser. If not, the city is conquered and any Settlers are defeated. (remaining rules page 12)
Land Combat - Battles typically occur during a "Move" action when an Army moves into a space with one or more foreign umits or a foreign city [...]
Combat Roll and Combat Value - Battles are resolved through combat rounds. [...] If one side loses all his units, he has lost the battle. If both sides lose all their units, combat ends with no winner or loser. [...]
Settlers in Combat - [...] If Settlers end up in a space alone with a foreign Army unit, a battle occurs in which any Settlers are defeated automatically (no cards are played or dice rolled). [...]