One of the authors of the paper here. It might shock you to find that I believe the paper was not "simply wrong". I do however think that we did a poor job explaining our evaluation setting, and we plan to rectify that in a paper revision soon. This rule set issue has tripped a lot of people up.
First, I want to clarify what we were trying to investigate in the paper was whether AI systems that in typical situations can outperform humans can still suffer from surprising and non-human failure modes. As another answer to this post notes, machine learning systems in general are often vulnerable to adversarial attack. But in many cases these machine learning systems are vulnerable to, well, a lot of other things: e.g. image classifiers often fail when you change the camera angle, lighting, etc. So it's natural to wonder whether these failures just reflect the system not really being "smart enough" and that it'd go away with more data, bigger models, etc.
We picked Go as an evaluation environment because there are very strong Go-playing AI systems, that can beat the best human players. Moreover, these AI systems haven't just been strong in "the lab" in a controlled tournament setting. They're also used in the real world by many thousands of Go afficionados as a teaching aid, a fun opponent to play against, etc. As any software developer can tell you, a large user base is an excellent (if sometimes painful) way of discovering bugs in your system. So, it's notable that people have for the most part not found many ways of beating these AI systems by hand -- although there are some, e.g. one of the authors of KataGo in his excellent response on Reddit describes "Mi Yuting's flying dagger" opening pattern that some earlier otherwise very strong Go AI systems struggled to handle.
This preamble is to say: our primary focus was to learn something about machine learning, not about Go. Because of this, we aimed to pick an evaluation setting that is fair for KataGo, but not one that necessarily corresponds to human play. This I think is the root of the confusion you and many others experienced. In retrospect we should have expected this, and we're actually working on a follow-up attack now that also wins under standard Chinese/Japanese rule sets.
With that out of the way, what is our evaluation setting, and why do we think it's a reasonable thing to do? We used Tromp-Taylor rules, modified to remove opponent stones from within groups that can be shown to be unconditionally alive via Benson's algorithm. This is the same as "Tromp-Taylor Rules" with "SelfPlay Opts" turned on at the KataGo rules page.
We picked this ruleset as it was the one used during KataGo's original training run reported on in their paper. Our understanding is that later training runs randomized the ruleset to enable KataGo to better transfer to play under human rules, but it still made up a significant fraction of later training data. Crucially, KataGo does know the ruleset it's playing under: it's a configuration option.
A human player might wonder why these strange AI researchers are obsessed with Tromp-Taylor rules. They're often called the "logical rules of Go" because they're simple, which is appealing to a certain kind of person obsessed with Kolmogorov complexity. But the main reason is that it allows for automatic scoring of board games.
Under typical human play, players will decide which stones are dead or alive at the end of the game. That works well for players who know what they're doing. But systems like KataGo learn to play Go from scratch. What does a randomly initialized neural network do here? It'll make completely random predictions about which stones are dead or alive. If we then use these nonsense predictions to train the system -- well, garbage in, garbage out. So, this human convention has a serious bootstrapping problem.
Of course, once the system has learned something about Go, you could switch to a more human style of play. And indeed KataGo configured to play against humans does do this -- it has an auxiliary head of the network that predicts what stones are dead or alive, that it's learned from experience over many games. But it's still risky to use it as an optimization target. If KataGo has a particular blindspot in its dead/alive evaluation, then that blindspot will just get reinforced as it plays against itself.
Tromp-Taylor avoids this problem by, well, just forcing the game to be played out to the end. That'd be annoying to a human, but AI's are quite good at not getting bored. It is a waste of compute and slows down training, which is where KataGo's SelfPlayOpts kicks in. If it can show (not predict, but actually have a guarantee) that a particular group of stones is unconditionally alive (under any sequence of future legal moves) then it'll remove opposing stones within that group. These are stones that are provably dead.
Let's look at the game from the ArsTechnica article quoted in the OP:
Under Tromp-Taylor rules (whether the original, or the "modified" version used by KataGo) this is a win for black, the adversary. But if they were playing in any human tournament, it'd surely be called a win for white. What's the difference here?
Well, the black stones in the lower-left are clearly very weak. They could be captured by white with little work. Black's territory in the top-right looks fairly secure, but it's pretty small. So, a polite pair of human players would agree the black stones in the lower-left are dead, and declare a win for white. If they really couldn't agree, a referee might be called, or the match might restart.
However, the white territory in the lower-left isn't "unconditionally alive" in the sense of Benson's algorithm. If the white player plays sufficiently badly (you can imagine it's trying to lose, even), then it could lose that territory. So, the black stones in the lower-left territory don't get removed under KataGo's "SelfPlayOpts".
Scoring then proceeds with regular Tromp-Taylor rules. This scoring rule gives points for each stone of the color (which is about even between black and white), plus the number of empty points that reach only that color (this is none for white, but black gets all of the top-right). So, black wins.
Now this does all feel rather contrived from a human perspective. But remember, KataGo was trained with this rule set, and configured to play with it. It doesn't know that the "human" rules of Go are any more important than Tromp-Taylor. In general, we want AI systems to do well at the thing we trained them to do: if they fail at that, but do great at some unrelated task, that's interesting, but not very reassuring.
However, it's an interesting question whether we can find attacks that work under human rules, too. After all, our adversary wasn't trying to win under human rules -- what if we just train it do so? Our provisional results suggest we can, with around a 98% win rate (a bit lower than the 99% in the original paper, but not too bad!) Here's a sneak peek of one game (white is victim, black is adversary). We definitely need to dig more into this before we're confident in the results, but wanted to share it as it may be more interesting from a Go perspective.
Although I can't quite argue for this paper being simply wrong, I do think there is one major limitation: the problem we discover disappears if the victim does enough search. Now even without search KataGo is very strong (we estimate top-1000 professional). So, it's interesting an AI can be so strong and so weak at the same time. But, we've certainly not broken all of computer Go, although we plan to see how far we can push the attack against victims with search (they may still be vulnerable, but it just be harder to find the attack).