You lose when you draw a card in an empty library due to rule 104.3c:
If a player is required to draw more cards than are left in his or her library, he or she draws the remaining cards, and then loses the game the next time a player would receive priority. (This is a state-based action. See rule 704.)
Scrying doesn't cause you to draw cards. It only allows you to look at the top X cards of your library and put them on the top/bottom of your library. See rule 701.17a:
To “scry N” means to look at the top N cards of your library, then put any number of them on the bottom of your library in any order and the rest on top of your library in any order.
emphasis mine
If scry made you draw it would say draw and not look.
source 1 2