The now-defunct magazine The Games Journal has this to say about the Library building in San Juan:
Simply put, this card is too powerful, even in a four-player game. Even with a cost of 5, you should devote all your energies to building it the moment you draw one. Although it is (usually) not that useful when choosing Producer or Trader, the fact that it gives a 2-card advantage when choosing Builder or Prospector means that the other players must altering their actions to counter this advantage, much moreso than with any other card. In fact, the best way to address this, is to build your own Library but this requires that you actually draw one. Ultimately, this means that there's a great deal of luck in whether you draw one of them or not.
Is the library really this strong? You give up six cards to get the Library (the Library card itself plus the five other cards it costs to build it). If it gets you one extra card every turn, as you choose prospector or builder, it seems to me that it's generally going to take much of the rest of the game to pay for itself. I think that's not so different from other cards. For example, the Carpenter and Quarry cost three and four respectively and both give you one more card every time you build a violet building (in slightly different ways, much as the Library gives you one extra card in slightly different ways as Prospecter/Builder). Is the Library going to pay for itself so much more handsomely than those cards (or many other cards)?
For reference the rules are available here.