In her Masursky lecture (video 406.01), Christina Richey emphasizes that the 2015 CSWA workplace climate survey is an attempt to replace "anecdotal" evidence with "social science". For this reason she collaborated with the anthropologist Kate Clancy at the University of Illinois.
Clancy recently appeared at the press conference introducing Jackie Speier's Federal Funding Accountability for Sexual Harassers Act, which also featured the astronomer Sarah Ballard. Speier's act has been proposed as a solution to "rampant" harassment in the sciences, and Clancy said at the press conference that she "can't help but take it personally". She described her 2013 Survey of Academic Field Experiences (SAFE) study as an attempt to find out "to what extent sexual harassment and assault occur at field sites" and she summarized the findings as follows:
In our sample, 71% of women and 41% of men reported experiencing sexual harassment. 26% of women and 6% of men in our sample reported experiencing sexual assault including rape.
I had a look at the paper, and found something I think is strange. It appears that Clancy's statement is based on responses to two yes/no questions in the survey. Surprisingly, these are the only two out of 45 questions that measure the extent of harassment and assault. They are:
32. Have you ever personally experienced inappropriate or sexual remarks, comments about physical beauty, cognitive sex differences, or other jokes, at an anthropological field site?
39. Have you ever experienced physical sexual harassment, unwanted sexual contact, or sexual contact in which you could not or did not give consent or felt it would be unsafe to fight back or not give your consent at an anthropological field site?
As far as I can tell, answering "yes" to question 32 is counted as what Clancy described as "reported experiencing sexual harassment" at the press conference. (The 71% just means that 361 of the 512 women in the study answered yes.) I can safely say that, if I answered yes to that question (as 56 men apparently did), I would not thereby mean to suggest (i.e., would not thereby "report") that I had been sexually harassed. As a measure of harassment, then, question 32 leaves a lot to be desired.
But question 39 is much more puzzling. Answering yes to this question, it would seem, is taken by Clancy as a report of "sexual assault including rape." Now, to be sure, the question as worded does cover rape. If a respondent had been raped, they would have to answer yes to this question. But answering yes could also mean that they were merely groped or kissed by an overconfident suitor. Indeed, presumably, Sarah Ballard's last encounter with Geoff Marcy would warrant an affirmative response to this question. He merely touched her neck or shoulder in a gesture of support. Ballard herself has described this as the "physical" moment of her "harassment story". Would she say that she was "harassed or raped" for good measure? I don't think she would.
Suppose we asked Clancy how many people her survey found had been raped. ("Of the 131 women who reported being physically harassed," we might ask, "how many were raped?") As far as I can tell, she simply has no basis for answering that question. (It is possible that some of her phone interviewees told of rape, but this is not made clear in the paper, nor would it provide us with a way to determine the relevant number.)* Certainly, it is misleading to say that 26% of women experienced assault including rape if no effort was made in designing the survey instrument to make it able to measure incidence of, specifically, rape. It is highly doubtful that the 8 yes-answers among the men, for example, "include" an instance of rape. And, yet, this is actually what Clancy claimed at the press conference: "...6% of men in our sample reported experiencing sexual assault including rape." (Similar wording can be found in the discussion section of the paper, though without any further explanation.)
This is a really frustrating thing about this survey, and, at least for me, completely undermines its credibility. Of the 45 questions respondents were asked, only two actually measure the unwanted behavior it makes conclusions about, grouping them together in two large classes under one yes/no question each. Why did Clancy not take each of the types of behavior on the list and ask about them specifically? Indeed, Katie Hinde, who co-authored the study, has remarked that "the absolute WORST way to measure sexual misconduct is to query 'Have you been sexually harassed?' or 'Have you been sexually assaulted?'" Instead, she says, quoting an NDRI report, "measurement of these constructs requires multiple, behaviorally specific questions." While questions 32 and 39 are only arguably "behaviorally specific", a single yes/no question is certainly not "multiple". Question 39, meanwhile, seems to be phrased in "the absolute WORST possible way."
It seems to me that this instrument was designed to be as blunt as possible, promising the highest possible number of affirmative responses, without being able to distinguish mild from severe behavior (i.e., without being able to exclude rape from a specific number of the 131 "reported" cases of "assault"). In my opinion, the study simply does not allow any conclusions to be drawn about the incidence of harassment and assault, certainly not "including rape." Given the way the survey was designed, to say that any of the women in the study "reported experiencing assault including rape" is as meaningless as saying that a certain amount of people have been "shot, punched or insulted" because they answered "yes" to the question "Have you ever been shot, punched or insulted?" on a survey. Unfortunately, this is what passes for "social science" these days.
________
*Update: Here is what the paper says about the interviews: "Survey respondents could indicate whether they were willing to be contacted for a subsequent, 30-minute phone interview; 26 interviews were completed between the two waves and all were conducted by [Kate Clancy]. Interviews were designed to allow respondents to describe their range of experiences at the field sites where they had trained or worked; these data will be described in subsequent publications." As far as I know, the "subsequent publications" have not yet materialized. (The 12 citations of this paper that PLOS-ONE identifies through Scopus do not include a paper authored by Clancy.) If any of the 26 interviews contained an account of a rape, this would have been easy to state. Clancy et al. do not do so, but there is in any case no reason to think that the 26 interviews are representative of the 131 women who reported physical harassment, nor the 187 men and women who did so. While it might be possible, with the use of the interviews, to say that "26% reported being sexually assaulted, and at least one had been raped," the study simply does not allow us to conclude that any proportion of the sample experienced "sexual assault including rape." Again, I want to emphasize that it could easily have been designed to allow such an analysis.
No comments:
Post a Comment