I noticed gobans are not square, but rectangular. Why is that, and what is the correct orientation of the board when playing?
The design of Go components is all about symmetry. Because perspective shortens the vertical height of the board, the boards are made taller than they are wide. This way they appear to be square when you're playing. Another similar issue occurs with the stones. Black stones appear to be smaller than white stones, so the black stones are actually made slightly larger than the white stones to counter this effect.
Related and awesome: How many squares are there on a Go Board?
My japanese goban is exactly like that (45x42 cm²) and the effect it's actually noticeable after you have been playing for a while.
However, I also own a korean folding goban which is a bit smaller (understandably as it is supposed to be easier to carry around, with accordingly slightly smaller nice glass stones) which is exactly 38.5x38.5 cm², and although I cannot now find the reference (looked over the Internet and my books) I actually recall reading that Korean boards are usually squared instead of rectangular; maybe someone can find a source for this.
Anecdotically, our go club Sensei (a Japanese ex-insei) once really called us out in the middle of a game cause we had inadvertently put the goban the wrong way. It was amusing and embarrasing. So much, that in the end we called it a jigo, turned the board 90º and started a new game!
He actually nodded with his head and smiled the next time he came by to have a look.
Indeed it has to do something with perspective, but a full compensation for perpective would require a much more stretched board.
I believe the squares are stretched just enough so that vertically adjacent stones appear to be touching and not overlapping (consistent with horizontally adjacent stones). In that way, thicker stones require a deeper board than thinner stones.