Achieving the statistically perfect komi is interesting, but generally the difference between winning by 1/2 a point or winning by 1/3 a point does not change a game result. We play in units of one stone (i.e., one point in area scoring), not 1/3 stone.
I don't have my copy of Go Players Almanac so I can't readily check dates -- I'll try to update later -- but my memory is that before komi was introduced, jigo (a draw) was possible; but in the Japanese pro tournaments at least, a jigo counted in White's favor (though not as strongly as a win), since it was known that Black had a significant advantage.
The Japanese introduced komi in the early 1900s, at a value of 2. (Komi wasn't standardized for a time, so there are arguments for "introduced" at earlier dates -- see John Fairbairn's discussion at https://senseis.xmp.net/?HistoryOfKomi for examples -- but I think "early 1900s" is good enough for our purpose here.) 2 was pretty quickly seen as insufficient, and by 1934, komi had been raised to 4.5, where it stayed until the late 1950s / mid 1960s, when it was raised to 5.5.
When I began playing in the late 1980s, players indeed talked about how komi had "recently" been increased to 5.5. (Well, when the game is 2000 years old, I guess 20 or 25 years seems "recent".) When I began playing in amateur tournaments in the late 1990s, the komi could vary, depending on the rules in use. AGA tournaments tended to follow the lead of the Nihon Ki-in, and used komi of 5.5; but there was strong material support from the Ing Foundation for tournaments, with the caveat that the tournament had to be played using Ing's idiosyncratic rules (intended to make the game more accessible to children and beginners) -- one of which was komi set at effectively 7.5.
Sometime in the early 2000s, Chinese rules switched komi from 5.5 to 7.5. About the same time, the Nihon Ki-in switched from 5.5 to 6.5. The AGA switched from 5.5 to 7.5 effective 2005.
There was discussion about whether the AGA should make it 6.5 instead, but changing from 5.5 to 6.5 would affect only the margin of victory in the vast majority of AGA games, not the game result. (It's theoretically possible to have a game with weird seki formations where that may not be true, but it's very rare in actual play.) So for example, if Black with 5.5 komi wins by 1.5 points, then with 6.5 komi Black still wins, but by 0.5 points. If the reason for komi is to give both players an even chance, and 5.5 isn't enough, then 6.5 also isn't enough. So the AGA changed to 7.5 not 6.5.
You may be thinking, "what about the games that Black with 5.5 komi would win by 0.5 points? Doesn't shifting to 6.5 komi mean that instead White would win by 0.5?" The answer is that, with 5.5 komi, Black would not win by 0.5 (except in the unusual seki situations that rarely happen). This AGA article describes why:
https://www.usgo.org/news/2013/09/only-a-passing-matter/
(I read a better explanation, but I'm not finding it now. Will update if I find it.)
A reasonable question is, why did the Japanese go with 6.5? I don't have a definite answer, but I believe it's mostly because they have a slightly different ruleset. In AGA rules, White must play last, which would penalize White under the Japanese rules. In Chinese rules, you get a point for each intersection you surround, and you get a point for each stone you have alive at the end of the game -- there is no penalty for filling in a point of your own territory, as there is with Japanese "territory" scoring. (There are also differences in how prisoners are handled -- they matter in Japanese "territory" scoring, but not so much in Chinese "area" scoring.)
The mathematician Elwyn Berlekamp proposed a system of bidding for komi. For example, on the first move, Black could play and pay 20 points komi, or pass. Next, White could play and pay 19 points, or pass. Eventually, one of the players moves, and pays the komi to the other player. Berlekamp arranged a short series of games between two pros living in America. IIRC, the komi paid tended be surprisingly high -- 9 or 10 points. (Thanks for reminding me of this -- I'll have to dig up the details.)
