When Yu-Gi-Oh! first started, an unlimited deck size was allowed. Now there is a limit of 60 cards. It may not always be advantageous to play more cards (see Why would one want more than 40 cards in their deck?), but why was this restriction added? After all, Magic the Gathering has cards like Battle of Wits that rely on decks of over 200 and they seem to be doing fine.
If Reddit is to be believed, there was a person at a German regional tournament who thoroughly abused this by bringing a deck with 2222 cards (enough for about 740 cards with 3 duplicates each). To make matters worse, as many cards as possible were added so that the deck had to be shuffled during the game as often as possible. While it was definitely poor sportsmanship, is was unquestionably legal, so Konami instituted the maximum deck size shortly thereafter.
It was originally 40+. There was no limit. I'm pretty sure the 60 card limit was introduced because some guy brought a deck that was outrageously large to a tourney.
I think at one event long ago in Germany some dude brought a 2k card deck or something? I guess Konami decided to set an actual limit then.
This story has appeared in other places across the web, with this forum post including a picture of the deck purportedly being carried in (and a several dozen page deck list).