The CSWA Workplace Climate Survey has finally been published. I've been following it since early 2016 and, since its authors wouldn't answer any of my questions, I've been impatiently waiting for the report. Well, here it is:
Clancy, K. B. H., K. M. N. Lee, E. M. Rodgers, and C. Richey (2017), Double jeopardy in astronomy and planetary science: Women of color face greater risks of gendered and racial harassment, J. Geophys. Res. Planets, 122, doi:10.1002/2017JE005256.
The PR push appears to be well-organized. But I notice that the subheadings of the UIL press release and the EOS interview both get the results wrong. UIL says the survey found "widespread bias"; EOS says it "reveal[s] the prevalence" of harassment in astronomy. The paper, however, says that "these data cannot provide a direct assessment of prevalence”, noting that "prevalence studies are exceedingly uncommon in research of this nature," which is true. (To their credit, the AGU and AAS get this right in their joint press release.)
In lieu of determining prevalence, the authors say* they tested four hypotheses, which I can't distinguish from the null or prior I would construct in such a study:
1. Female respondents will report more verbal and physical harassment than men.
2. Respondents of color will report more verbal and physical harassment than white respondents.
3. Trainees will report more verbal and physical harassment than those scientists of a higher rank.
4. Women of color experience double jeopardy where they are especially at risk for verbal and physical harassment compared to white women or men of color.
This isn't something that stands in need of empirical evidence. What we want to know is how astronomy compares with other fields of human endeavor. That is, we want to know whether astronomy provides a more or (as I suspect) less hostile environment for women of color than other fields. Indeed, we'd probably just be testing whether astronomy is generally less hostile for humans than other contexts. It's not going to ensure your safety 100% but it's probably a pretty good choice if it's hostility you're trying to avoid. Especially, indeed, if you're a woman of color.
Finally, it looks like a great deal is going to be made of the finding that "88% of respondents reported hearing negative language from peers". But this number does not distinguish between reports of hearing this sort of language "rarely", "sometimes" or "often". That is, the great majority of respondents reported that it is heard rarely or never. I'm going to look more closely at this in the days to come. (It actually seems a bit more complicated to disaggregate this particular result than the preliminary ones.) I just wanted to get my initial reaction out there now to encourage people to be critical in their reception of this survey. After all, the greatest respect you can pay to a scientific result is to critique it.
_________
*I'm suspicious about whether these hypotheses had been stated explicitly before the survey was designed. They were not part of Christina Richey's 2015 and 2016 presentations of the data. If I'm right about this, there are some pretty serious "degrees of freedom" in their framing. Since the authors do emphasize their p-values, there's a risk that these hypotheses are a result of p-hacking their data.
Number of times the word "assault" appears in the paper: 16 (plus one more in a reference title).
ReplyDeleteNumber of times it appears in the survey questionnaire: 0 (except in an introductory paragraph):
You will be asked about sexual harassment or assault experiences you may have had in the past, albeit in a general way. These questions may be triggering, and stimulation of these memories may make you uncomfortable. If you are worried that thinking about or sharing your experiences about sexual harassment or assault will be too mentally harmful, do not continue with this survey.
The survey respondents read "PHYSICALLY harassed" in Q9 and Q10, and then the researchers interpreted those, or re-worded those, as "assault" in the paper.
Yes, that appears to follow the conventions in this kind of research. In Clancy's 2014 paper about the SAFE survey, "physical sexual harassment, unwanted sexual contact, or sexual contact in which you could not or did not give consent or felt it would be unsafe to fight back or not give your consent" was interpreted in the analysis as "sexual assault including rape".
ReplyDeleteThere seems to be a rule that any unwanted physical interaction must be described as an "assault" on pain of not "respecting" the experience of the respondent. As Katharine Lee put it in the EOS interview:
"The data for this study of harassment and hostile climate were difficult to deal with at times. Every time I encountered a survey response where someone felt unsafe or reported being verbally or physically harassed, I heard them. Every person who participated was important for this research. I want to thank everyone who participated in the study, and I want to support them by respectfully reporting their experiences."
I'm glad the author correctly note that their survey does not provide any information regarding the prevalence of abuse or harassment. But if that's true, then how can they draw conclusions regarding the relative prevalence among different groups (i.e. minorities). All of their four hypotheses are based on one group experiencing more harassment than another group. So if there are sampling effects that won't allow us to estimate prevalence for the group as a whole, why aren't those same effects true for each sub-group relative to one another?
ReplyDeleteThe following press article has good discussion of the research article.
ReplyDeletehttps://www.space.com/37473-women-in-astronomy-discrimination-study.html
It has this to say:
There are about 10,000 people working in astronomy and related fields in the U.S. and about 20,000 worldwide, according to the American Astronomical Society. Clancy said that from a social scientist perspective, the results of the survey cannot be said to represent the entire astronomy community.
"It would be really erroneous of us to dare to say that 450 people speak for 20,000. That would be a real error in our work," she said.
"So instead, what we can do is we can speak from this population, and we can say that this is not a nonrandom sample," she said, meaning the survey was open to anyone in the community and the respondents were not selected by the researchers. "At the same time, there's a good chance [the survey] is fairly representative."
An glib rejoinder is "representative of what? Because 2/3 of the respondents reported their gender as female, whereas females are said to be a minority of astronomers overall."
Another quote:
"Literally almost half of our sample of women of color didn't feel safe in their workplace. Just sit with that for a minute," Clancy told Space.com.
That's a good point. Department chairpersons should sit with that and think, "what can I do?" I would say the first thing is find out if it is true in their own department, by asking people directly or indirectly. Second thing is ask them how/what the department can do to help.
If these were children, and they said they didn't feel safe on the playground because of their race or gender, we'd expect the adults to do something about it.
On the other hand, if you go around your department and have these conversations, and it turns out that a typical response is, "I don't attend the annual holiday party after that one time that a drunk grad student asked me to go home with him after the party" then that's different than, "...after that party included skits of male graduate students pretending to be scantily clad women sitting on Santa's lap and cooing for no-cost-extensions, and no one ever said or did anything but laugh, at the time or the next week," and that would be different than "I switched out of astronomy because my research required me to work alone, late at night, in the unlocked remote observatory out by the gun range." And that's different than, "The last department chair told me that if I got pregnant, then that would not be good for my tenure case."
A few other observations:
ReplyDeleteThe data / analysis in the supplemental documents do not appear to tease out physical vs. verbal harassment. It is not clear how physical harassment is folded into the analysis.
The final section of the article contains the following statement: "At this time, we want to remind the reader that the findings of this study cannot be attributed to events from long ago: respondents were asked to only report experiences they had had in their current career position over the last 5 years." This does not seem to correlate with the instructions in the questionnaire as provided in the supplement. Specifically, the instructions did _not_ say to limit responses to experiences in the last five years but "Please respond to Section 1 regarding your personal experiences in your current position. If you have changed career positions in the past five years, then please also complete Section 2 regarding your personal experiences in your previous position. Section 3 requests demographic information for all respondents." That is, respondents who have been in the same position for 5+ are not limited to reporting experiences from longer ago and respondents who have changed positions within the last five years can report experiences in the previous position from more than five years ago (at the time of taking the survey), even under the assumption that all respondents read, understood, remembered, and followed the instructions, which seem to have only been provided once at the beginning of the survey.
I agree with your point about misinterpreting the last five years. But I figured the distinction would only matter much for senior astronomers; otherwise job changes tend to be often, so the look-back time would be approximately five years even if the respondent didn't pay much attention to the exact wording. Using their Table 2, I find the fraction of senior scientists in each category is 11%, 25%, 38%, and 25%, respectively for WOC, WW, WM, and MOC. So the stats are not going to be much affected by your concern that a person might have reported harassment from 20 years ago because that happened at their current position.
ReplyDeleteIn looking at Table 2, though, something did catch my eye. The fraction of trainees (I guess this means grad students and postdocs) is 71%, 49%, 41%, and 44%. The high proportion (71%) of the WOC group is noteworthy but I don't know how and how much it matters. The paragraph about hypothesis 3 seems to suggest that trainees reported verbal harassment more than their more senior colleagues. So is the "double jeopardy" of WOC an artifact of their sample being more trainees than the other groups? I don't know.
I do believe WOC could and do experience the double jeopardy described, but I'm not confident that this survey is much more than a great deal more anecdotes than ever before assembled.
Thomas, you wrote, "Finally, it looks like a great deal is going to be made of the finding that "88% of respondents reported hearing negative language from peers".
ReplyDeleteOne could spin the positive side: 87% of female respondents reported that they never felt sufficiently unsafe to warrant their missing even a single class, meeting, conference, field work opportunity, or other opportunity.
Still, the associated fraction for males was 98%, i.e. 11% higher than 87%.
I'm grateful for the comments. I've been struggling with the paper for a few days now. And it does seem like there are some obvious problems.
ReplyDeleteI also found the "within five years" claims strange. It really doesn't seem to have been limited that way.
I remember reading a parapsychology study once that failed to demonstrate telekinetic abilities in the sampled population, but nonetheless claimed to have a found a "significant" gender difference in the distribution of that ability. I have a similar feeling about this study.