The instructions don't say (from what I see) and I need to know for the game.
As many times as you want. Almanac, page 9:
Example: Nick, the orange player, built a settlement at the ore special harbor. Nick may exchange 2 ore cards for any 1 other Resource Card. He can also trade 4 ore cards for any 2 other cards.