THE GENDER GAP IN CHESS: A CASE STUDY IN STATISTICAL RESEARCH

The following is an introduction to a series of posts about some of the difficulties with conducting and interpreting statistical research, with links to the rest of the series at the end of this post.

Bobby Fischer once said he could beat any woman in the world giving them knight odds* (the full quote, in true Fischer fashion, is worse). Mikhail Tal famously responded, "Fischer is Fischer, but a knight is a knight!"

*Knight odds means the player giving odds starts the game with one knight already off the board.

Tal was correct, of course. In 2008, a master player named John Meyer, rated 2284 (grandmasters are rated 2500+, with the top GMs well over 2700 or even 2800), played a match against the computer program Rybka with knight odds. By that time, computers had far surpassed humans in chess. Rybka could have easily defeated the world champion in a non-handicapped match. With knight odds, Meyer won the match 4-0. There were women in Fischer's generation much stronger than Meyer who would have had no problem beating Fischer given such a handicap.

Still, chess remains a largely male-dominated profession. Currently, there are just two women in the top 100 rated players in the world, and one (Judit Polgar) is retired and will fall out of the active rankings later this year. Theoretically, chess should be among the most gender-neutral competitive disciplines, but the overwhelming majority of players are male. In fact, the predominance of male players is so strong that the URL for FIDE's top 100 overall list actually ends with "...?list=men", even though there are women on the list.

The question of why this is and what can (or should) be done about it has long been a point of discussion in the game, but this discussion reached the mainstream media last month due to controversy over an article written by British Grandmaster Nigel Short in the magazine New in Chess.

If you don't know Short (and you probably don't unless you particularly follow chess or remember his highly publicized World Championship match with Garry Kasparov in 1993), he's...well he's not really the best representative to speak about anything, really. When asked to write an obituary in his newspaper chess column for fellow British Grandmaster Tony Miles, he pretended to write a proper obituary for a few paragraphs before descending into a long-winded rant about why he didn't like Tony Miles, culminating with the line "I obtained a measure of revenge not only by eclipsing Tony in terms of chess performance, but also by sleeping with his girlfriend, which was definitely satisfying but perhaps not entirely gentlemanly." Nigel Short, everyone.

So it's no surprise that Short set off some fuses when asked to write about this topic (by the time he gets to the part about how he has to "manoeuvre the car out of our narrow garage" for his wife, you kind of get the sense that he's just doing this on purpose--which, in a media environment where controversy equals views equals money, he may well be.)

In the midst of his rambling, though, Short actually does cite an academic paper by Robert Howard (actually, a synopsis of the study posted by Howard to the chess website Chessbase.com):

"Nevertheless, my gut feeling was that female chess players are both stronger and more numerous than they were when I first began competing. The latter is certainly true, but an excellent article by the Australian Robert Howard on the chessbase.com website last year demonstrated that, despite the enormous societal changes over 40 years, the gap between the leading males and females has remained fairly constant at nearly 250 Elo points – a yawning chasm in ability. That women seem stronger has more to do with universally higher standards, due to the ubiquity of computers, than any closing of the gender gap."

Unfortunately, Short's citation comes with a clear agenda, as is evident in how he presents a second academic study which reached different conclusions:

"Howard also subtly critiques the most absurd theory to gain prominence in recent years, by Bilalić, Smallbone, McLeod and Gobet (which was submitted to the prestigious Royal Society, no less), that the rating sex difference is almost entirely attributable to participatory numbers (they comprise just 1% of the readership of this magazine). With the aid of a couple of bell curves this foursome neatly solve the eternal chess conundrum of why women lag behind their male counterparts, while simultaneously satisfying that irritating modern psychological urge to prove all of us, everywhere, are equal. Only a bunch of academics could come up with such a preposterous conclusion which flies in the face of observation, common sense and an enormous amount of empirical evidence too. Howard debunks this by showing that in countries like Georgia, where female participation is substantially higher than average, the gender gap actually increases – which is, of course, the exact opposite of what one would expect were the participatory hypothesis true."

The problem is partially that Short probably has no idea what the studies are doing (for example, Short seems unaware that Howard found the gender gap did decrease in Georgia compared to the rest of the world, makes up the term "enormous amount of empirical evidence" without justification, and I don't get the impression he's even read the Bilalić, et al study), but in this case, the blame doesn't lie entirely with Short. Howard's synopsis itself is largely responsible. It appears to misrepresent Howard's own work, as well as point to some potential critical issues with the study.

That being the case, I'd like to use this as an opportunity to cover some of the potential pitfalls in running this type of statistical analysis.

PART 1: MEASURING THE GENDER GAP
PART 2: ELO RATINGS
PART 3: CAUSE AND EFFECT, THE BILALIĆ, SMALLBONE, MCLEOD AND GOBET STUDY
PART 4: MISREPRESENTING THE DATA