A Critique of the SAFE13 Study

[This post was edited for clarity on 05/05/17 at 14:30]

SAFE13 is widely regarded as a ground-breaking study of sexual harassment in the sciences. It was conducted by Kate Clancy, Robin Nelson, Julienne Rutherford, and Katie Hinde in 2013 and published in PLOS ONE in 2014. The data set comprises 666 survey responses and 26 interviews with field scientists from 32 different disciplines in life, physical, and social sciences. In a press conference, Clancy summarized the result as follows:

In our sample, 71% of women and 41% of men reported experiencing sexual harassment. 26% of women and 6% of men in our sample reported experiencing sexual assault including rape.

In this post I want to look at what these claims mean and how they are supported by the data. Since the study is, indeed, altogether likely to be breaking new ground, inspiring similar studies in other fields, I think it is important to take a critical look at its methodology. Does it really "reveal" what it says it does? My conclusions are, let us say, skeptical.

Part of my skepticism comes from the authors' attitude to criticism. PLOS ONE normally publishes papers only on the condition that data be made available on request, and that authors will provide information about metadata and methods. In this spirit, I have been trying to contact Clancy to discuss the issues in this post since November of last year and, in preparation for writing this post, I again contacted Clancy and requested the data. Hearing nothing back from her, I contacted PLOS ONE's data team (as per the data policy) and asked for their assistance. I was soon informed that my request had been "escalated" to the editorial team, who eventually informed me that I would not be getting the data. The reason I was given was that the data I was asking for (namely, the survey data) did not support the claims in the paper I was asking about (the incidence of rape in the sample).* Rather, I was told, the interviews supported the relevant claims and these were confidential. This seems to belie the impression left by the paper itself, namely, that the conclusions therein are based solely on the surveys. This is important to keep in mind.

At a recent NYU panel (video here, see 00:06:05f.), Clancy described the study as a "Fuck You!" to reviewers of an earlier abstract who weren't persuaded by merely listening to women's "experiences" and wanted something more "empirical" instead. The paper, however, makes it clear that the "survey data neither allow us to estimate the rate of these experiences among our trainees and colleagues, nor do they allow any estimation of the prevalence of field sites with a hostile work environment and/or systematic abuse." This has not prevented either the authors or their readers from taking the results as an indication that science has a significant harassment problem. "[T]he large number of respondents from across dozens of disciplines and high prevalence of harassment and assaults," the paper tells us, "suggests that the results presented here are likely not attributable to only a handful of hostile field sites." In a 2014 podcast Clancy emphasized that they had "absolute numbers of hundreds of women saying they were harassed and assaulted". Despite its own stated limitations, then, the study is clearly being used to support claims about prevalence. The question is whether those claims hold up.

The paper claims that the

survey revealed that conducting research in the field exposes scientists to a number of negative experiences as targets and as bystanders. The experiences described by our respondents ranged from inadvertent alienating behavior, to unwanted verbal and physical sexual advances, to, most troublingly, sexual assault including rape.

I, for one, do not think that SAFE13 provides an empirical basis for saying that field work "exposes" scientists to any particular risk of harassment or assault and, especially, that this exposure includes a significant risk of rape. This is not just because the sample has an (acknowledged) self-selection bias, but because the measuring instrument (the questionnaire) is far too imprecisely designed. Moreover, no attempt has been made to compare the result to any estimate of baseline risk, though this point is somewhat moot since the imprecision of the instrument gives us nothing very definite to compare.

It cannot be stressed enough that the survey questionnaire itself did not afford an opportunity to describe behavior. Rather, as the supplementary material shows, the respondents answered yes-or-no questions about what they had experienced. These, arguably, included alienating behavior, unwanted verbal and physical sexual advances, and sexual assault including rape, but there were only two catch-all questions, one for non-physical and the other for physical harassment:

32. Have you ever personally experienced inappropriate or sexual remarks, comments about physical beauty, cognitive sex differences, or other jokes, at an anthropological field site?

39. Have you ever experienced physical sexual harassment, unwanted sexual contact, or sexual contact in which you could not or did not give consent or felt it would be unsafe to fight back or not give your consent at an anthropological field site?

Answering "yes" to question 32 was counted as a report of "sexual harassment", while a "yes" to question 39 was counted as "sexual assault" (presumably, "including rape"). In so far as experiences of alienating behavior, unwanted advances, assault and rape were actually described by the respondents, then, this must have been done in the interviews, i.e., by only 26 of the 666 respondents. (Like I say, this appears to have been confirmed by the authors, albeit only through the editors of PLOS ONE, after ignoring my question for months.)

We do not know how many of the 26 interviews described rapes. But it should be possible to provide the de-indentified** extracts from the interviews that were coded “alienating”, “assault”, “rape”, etc. The definition of “rape” in the social sciences, it should be noted, is somewhat elastic. The University of Texas, for example, recently announced that 15% of its female undergraduates on the Austin campus had been “raped”. By comparison, the US Department of Justice puts the baseline rate of “rape or sexual assault” among 18-24 year-old women at about 0.7%.

Now, since the rape(s) and assaults reported in the SAFE13 study can only have been described in the interviews, the survey can have counted at most 26 of them. I think a count should be in fact be provided, and the data that underpins that count (i.e., the de-identified description of the behavior that got it coded as “assault” or “rape”) should be openly available. At the very least, the coding methodology should be made available, and the authors should be willing to explain their choices to critics like me.

Otherwise the claim that 26% of the women (and 6% of the men!) in the survey experienced “sexual assault including rape” is simply not supported by the data. To be sure, if the interview data were made available, there would still be a legitimate criticism to be made about the representativeness of the interviews with respect to the overall sample; but this would not be a formal criticism of the relationship between the claims in the paper and the data, since being open about this would allow for discussion, which is all that is needed.

As far as I can tell, then, SAFE13 does not actually support the conclusion that working in the field "exposes scientists” to any particular rate or range of negative experiences (i.e., experiences that they are not already exposed to the possibility of simply by being human). If a woman is at a particular risk of being assaulted anywhere else, then SAFE13 does not provide a rate of assault among field scientists that can be meaningfully compared to it. It is entirely possible that any given woman is safer while in the field than she would be in her own neighborhood of a major US city.

"Science doesn't have a sexual assault problem," Clancy has rightly said; "life has a sexual assault problem." But what she, as well as her fellow authors and many of their readers,*** fail to consider is that the prevalence of harassment and assault—including, indeed, rape—may be lower in science than in all other social and professional spheres. If it is then, not only does science not have a harassment problem, it has a solution. This, I dare say, is an empirical question. And I hope that SAFE17, or an equivalent study, deploys a methodology that lets us answer it.

Update: See also "The Responsibility of Anthropologists"

__________
*In my request, I stated, out of what I considered courtesy, my reasons for wanting the data, but I also made clear that, in addition to the question of the basis of claims about rape, I had a "general" interest in the data. I wanted to see what the data set looked like, and be able to consider alternative interpretations of it. Though I have repeated my request, I have still not seen the actual data and can therefore not even confirm that it exists.
**The "data availability statement" at the beginning of the paper makes clear that data that could lead to the identification of respondents will not be available, but "limited, de-identified data may be available by contacting the corresponding author".
***Monica Byrne, for example, reads the SAFE13 study as showing that female scientists are exposed to a notable risk of being raped by a colleague.