Since one can visit other sites such as Ruins and Dens and choose not to enter or engage, if you fail in battle against enemies here, I believe you are not forced off the space. Are Keeps and Mage Towers then the only space types where losing a battle forces you to leave the space?
Fortified sites are the only spaces that require you to withdraw to the space you attacked from when you lose.
On page 9 of the rulebook, under section 3 of Combat Outcomes:
d. If you failed to defeat all the defenders of a fortified site, you have to withdraw back to the space you were attacking from.
The rules regarding ruins and monster dens are different. The Combat Outcomes section mentions leaving any revealed enemies on the space, but doesn't explicitly mention having to withdraw. We can use some sections from other parts of the rulebook to clarify. On page 6, section 4 under Movement (emphasis mine):
d. Entering a space with an adventure site (ruins, monster den, spawning grounds, dungeons or tombs) does not necessarily end your movement; you may ignore it and treat it as an empty space, even if there are enemy tokens.
And on page 7, under section 1 of Combat with Enemies (emphasis mine):
b. If you are at an adventure site containing enemies (dungeon or tomb, unconquered monster den or spawning grounds, or ruins with enemies), you may announce that you are entering the site as your action. This initiates combat with all enemies in the site.
Given these conditions, these spaces are treated as safe spaces (page 6; rule 7d) and don't require a player to withdraw after a loss.
The main purpose of moving back to the attacking space when losing to a fortified enemy is to ensure that Forced Withdraw rules (page 9; rule 2) don't apply immediately. It prevents players from having to take additional wounds when they've already lost and likely taken wounds. This isn't a concern with adventure sites because they are already considered safe spaces.