After a piece and/or group is completely out of liberties, it is removed immediately.
The only rule that would really prevent you from placing a stone back on the captured space is the ko rule, in which you can't re-place exactly the same stone that was just captured in such a way that it completely re-creates the board position from the previous turn (otherwise, you'd basically be stuck in an infinite loop of capturing and re-capturing the same stone ad nauseum and that ain't no fun).
If re-placing your stone doesn't re-create the board position, it's fine.
For example:
$$ A simple ko
$$ . . . . . .
$$ . . O X . .
$$ . O 1 a X .
$$ . . O X . .
$$ . . . . . .
When white plays at a, the black stone at 1 is immediately captured. Black can't then play again at 1 immediately since this would re-create the entire board position (black would be capturing exactly the same stone white just played to capture the original black stone). Black needs to play somewhere else first.
However:
$$ B Not a ko
$$ . . . . . . .
$$ . . X O O . .
$$ . X 2 a 1 O .
$$ . . X O O . .
$$ . . . . . . .
The black stone at a will capture the white stone at 2, however in this case white can play back at 2 immediately since that move will capture both the black stones at 1 and a: This is not re-creating the board position so it's completely legal.
The ko rule only applies to the turn immediately after the capture; once you've played somewhere else, and your opponent has played somewhere else, you can go back and re-capture the stone (if he hasn't defended it) by playing in your original spot just like normal.
Of note, there is another related rule known as the superko rule, which is basically the same thing except it's not limited to the immediate next move. For example, any move that results in the exact same board position being re-created, even if that takes place two or more turns later, could also be prohibited. However, the superko rule isn't as universal as a regular ko, and the exact implementation (or whether it's implemented at all) would depend on your particular ruleset. In casual play, it rarely if ever comes up.