It says in the rule book that you can only take 2 gems of the same kind only if there'll be at least 4 chips left after you take the 2. Since you only have 4 chips for two players and 5 chips for three players, is the ability to take 2 gems of the same kind reserved for four players only?
You are reading the rules incorrectly, they says when there are 4 chips at the start of your turn not after you take them. The rules don't care how many gems are in a stack after you have taken them just beforehand.
Collect Gem Chips – Either:Take 1 Gem from each of 3 different stacks (not gold)
1: Take 2 Gems from one stack, as long as there are 4 or more chips in that stack
In a 4 player game there can be 3 chips missing at the start of your turn, 3 player 1 chip missing and 2 player no chips missing.
Testing on the iOs version of the game lets me draw two chips of the same color from a stack when there is exactly 4 chips on it in a two player game.