The jump from 4 players to 5 players seems to makes the game tougher. While stronger monster surges start at 3 players, it's easier to cover the entire world with a 4 player team. I find gates spawning is the most dangerous issue that needs to be dealt with since leaving gates open will help advance doom. Having extra monsters is also bad, but some monsters will sit quietly until you go deal with them. A few cards look at how many monsters are on the map, so you don't want to let the total number of monsters to get to hight. That being said, monsters that move to expedition locations often don't have retribution effects and tend to be ignored in many of the games I've played.
If you have Akachi as one of your investigators, she can control where the next gate spawns by using her ability to look at the top two gate locations and putting one of them on top of the stack and one on the bottom. I'm not sure how powerful this strategy is, but it certainly is an option in a four player game. Once you get that second gate spawning, it's no longer an effective strategy.
If you want to alter the difficulty of the game and make it easier, I suggest altering the mythos deck and removing the hard mythos cards that have tentacles. Then build your mythos deck from the remaining cards. To make the game harder, you can remove all of the easy mythos cards. The mythos cards that come up heavily effect the difficulty of the game.