Many sources (including here, here, here, and here) explicitly list Royal Flush as the highest poker hand. This seems unnecessary - it's the highest hand anyway as the highest Straight Flush. When designing a ruleset, unnecessary is harmful (e.g. it gives new players more to remember). So is there a good historical or game-mechanics reason why we treat the Royal Flush as a special case?
There are a number of different hands that have a specific name. For example, A-2-3-4-5 is called a Wheel. The Royal Flush obviously gets its name from being the highest hand, and specifically having royalty in it (although one could argue so does a King high.)
As for the mistake of ranking of Straight Flush being lower than 4 of a kind, this is easily dispelled by looking at the math behind them. There exist only 40 straight flushes (of which 4 are Royal) and there are 624 unique hands that rank as four-of-a-kind.
One reason for it persisting would be in Video Poker where a Royal Flush tends to pay out significantly higher than the other Straight Flushes. This doesn't answer the question about how it managed to still exist before Video Poker was around, though.
I have had it conveyed to me recently that it's listed separately potentially due to wild-cards and how they affect the math - that perhaps a straight flush is not as difficult to acquire anymore where a Royal flush is. However I have not had an opportunity to run the math on it. I would welcome an addendum if someone does.
I would think this has something to do with the fact that a straight flush has multiple card combinations per suit to achieve it, making it more common (2, 3, 4, 5, 6/ 8, 9, 10, J, Q etc...) and the Royal Flush only has one card combination per suit (A, K, Q, J, 10).
The Royal Flush could be a straight flush also in theory, (I understand this is where your confusion lies) but the fact it has all of the Royalty cards in it, makes it a one of a kind "straight flush" in the suit. It also stands to reason that it is a lot more valuable of a hand than a normal straight flush as it has all of the highest value cards in it.
The only real supporting evidence for the origin of the name Royal Flush is literally only in the dictionary. It states that a Royal Flush is made up of the top 5 "honors" of any suit. To me the word Honor further suggests Royalty. The fact that Royal figures (King and Queen) are in the line up of a Royal Flush is the best suggested evidence of its origin.
It is so the James Bond character player by Sean Connery in the 60's can arrogantly say Royal... Strait... Flush... as he is slowly laying his hand down while raising his eyebrow & lighting his cigarette with his smug look on his face & then say Bond... James Bond so the camera can then flash to the villain who is stunned by the fact that he lost even though he cheated for the camera to then flash to the "femme" fatal whom is giving off this "Je ne sais quoi" look of admiration & seduction. Yeah that's it... Bond James Bond! That's why the Royal Strait Flush is differentiated... Because Hollywood did it!