I think there's two types of answer to this depending on whether you mind longer putting-away times or not.
If you're happy to put some extra effort into packing the pieces away, then a huge amount of time can be saved by storing the components in bags related to the number of players. For example a "3 player" victory point bag with 75 VPs in, a "4 player" victory point bag with 25 VPs in (100-75), and a "5 player" victory point bag with 22 VPs in (122-100). To play with 4 players simple get out the bags for three and four players.
The same trick can be applied to the colonists. There is an example of this being done very neatly here: http://boardgamegeek.com/image/841793/puerto-rico
Assuming instead that you want to save time overall then I know of a few other things that help.
The same idea behind having separate bags can be applied, by marking some of the victory point chips and colonists. For example it wouldn't affect the gameplay to have colonists painted in three different shades of brown or the front of the victory point chips marked with small dots.
Alternatively colonists can be counted very quickly when they are flat on a surface by using two straight edges. The trick is to push them into a triangular shape and then select colonists by the number of rows. See this image, and the comments below it, for details.
A final obvious, but sometimes overlooked, way to speed up the game set up, is to distribute the tasks. I usually end up doing this as otherwise some players have a tendency to stand back until the game is ready to begin.