One Last Mention of the CSWA Survey for 2016

[Update: this post deals, in part, with a specific error in an article that was corrected after I pointed it out. The 61% figure has now been corrected to 32%. Sarah Scoles has confirmed that my hunch was right, and the error resulted from a misinterpretation of the CSWA survey. The article, however, still does not provide that source, nor break the 32% down more informatively into "rarely", "sometimes" and "often". Nor is the reader told that the survey remains unpublished and its methods unchecked by peers.]

"Next time someone sends you a press release and you’re thinking of running the story, first contact the organization and ask to see the written report. If they say they don’t have a report, it’s simple: Either don’t run the study, or run a report that is appropriately dripping with skepticism, including the phrase "for which the organization refused to supply a written report" as many times as possible." (Andrew Gelman)

I had actually half expected not to hear any more about the CSWA survey of harassment in astronomy. I've written about its many problems before and explained why I think it should only be invoked with Andrew's "dripping skepticism". It only exists in the public domain as a set of slides from a conference presentation. Indeed, not only has it not yet passed peer-review, I gather from reports on Twitter that it has been rejected by at least one journal.

That has apparently not stopped Sarah Scoles at Wired from taking its results at face value. Indeed, as we shall see, she goes a little further than that in her review of "science's sexual harassment problem" in 2016. She begins by saying that this was the year that the science community finally caught on to the problem of sexual harassment, mostly through high profile cases. But, she tells us, it's not just anecdotes and stories.

The numbers agree: A 2014 study found that 71 percent of female scientists had been sexually harassed while out in the field, and 26 percent had been sexually assaulted.* In a 2015 survey of astronomers, 61 percent of respondents reported experience of verbal harassment in their current job, for gender, gender identity, sexual orientation, race, religion or disability status. Around 11 percent reported physical harassment.

Like Andrew Gelman, I think it’s very important to provide a full citation for such studies so that readers can check out the details themselves. Since I’ve been following the #astroSH story myself very closely this year, I was surprised not to recognize the 61% figure. After all, in her piece for The Atlantic in early January, she had written, as others did at the time, that "fifty-seven percent said that they had been verbally harassed because of their gender, while nine percent said they had been physically harassed." The 57% figure turned out to be wrong and has since been corrected to 32%. I.e., still not 61%. I went back to Christina Richey's slides to look for a 61% and couldn't find one. So I had a closer look at slide 5, with the relevant results:

Scoles seems to have simply added up the percentages. This is probably not an accurate interpretation of the data and not one that I have heard or read either Christina Richey or Kate Clancy (two of the study's authors) mention. That's no doubt because the categories overlap in principle. Indeed, most researchers and activists in this area insist that they overlap in practice. If there is anything to “intersectional” feminism, then some of the respondents (and probably most) would have reported harassment in multiple categories. Indeed, "male privilege" would, in theory, be able to protect people who aren't part of the 32% from, say, religious discrimination.

Also, do please note that Scoles, following the author's lead, simply ignores that the vast majority of respondents say that verbal harassment happens only "rarely" or "sometimes". She simply lumps them in with "often" to get the highest possible figure in each category. Then she adds the categories together as though there's no overlap. If she had applied the same procedure to slide 3 she would have found that 429% of respondents reported hearing problematic language of various kinds.

I have to say that "61 percent of respondents reported experience of verbal harassment" is only probably a misinterpretation because Richey and Clancy have provided no write-up of their results yet, and have in fact refused to discuss their methods and analysis with me (or anyone else that I know of). In my opinion, until a detailed written report is made available, the CSWA survey should be passed over in silence or mentioned only, as Andrew puts it, in phrases dripping with skepticism. Certainly, entirely novel results (i.e., results not yet announced publicly by the researchers) should not be concocted simply by adding up percentages on a chart, and then reported without even providing the source!

___________
*This is presumably Kate Clancy's SAFE study, which I've written about here. [Update: I want to add a few words about describing the SAFE survey as "a 2014 study [that] found that 71 percent of female scientists had been sexually harassed while out in the field, and 26 percent had been sexually assaulted." SAFE was not a study of all "scientists", but a study of anthropologists at field work sites all over the world, which are probably some of the more dangerous places for scientists in terms of sexual harassment and assault. Moreover, Scoles makes the study sound representative of all scientists, while the paper that reported the survey makes clear that there was a self-selection bias, so that "these survey data neither allow us to estimate the rate of these experiences among our trainees and colleagues, nor do they allow any estimation of the prevalence of field sites with a hostile work environment and/or systematic abuse." Finally, while Clancy herself has presented the 26% as reported rate of "assault", the relevant survey question uses a rather broad definition, on which even people who did not experience the encounter as an assault might well answer yes: "Have you ever experienced physical sexual harassment, unwanted sexual contact, or sexual contact in which you could not or did not give consent or felt it would be unsafe to fight back or not give your consent at an anthropological field site?" In short, think we have to write much more carefully about these studies in the future so as not to give readers the wrong impression about the scope and nature of the problem.]