The Colony is a location with six entrances. So for instance Raiding Party says add two zombies to every location. This includes the colony which would get one zombie in Entrance 1 and one zombie in Entrance 2
In your specified case when taking away zombies you would again take only two from the colony not two from every entrance
From the manual page 5:
The game board is made up of 7 different locations, including the colony board and 6 location cards.
Page 12; note the specific phrasing indicating that other locations are "non-colony locations" thus including the colony as one location just a special location.
Add 1 zombie to the colony for every 2 survivors (including helpless survivors) present there (rounded up). Add 1 zombie to each non-colony location for every 1 survivor present at that location.
thus the colony IS a location. a single location. The rest of it is a combination of cards saying to add/remove zombies from locations and not entrances.