The CSWA Survey in Plain Language

It's going to take a bit of work to properly critique the the CSWA Workplace Climate Survey as published in the Journal of Geophysical Research. In this post, I want to take a sentence-by-sentence look at the plain language summary. In later posts, I will elaborate on each point by way of a critique of the substance of the paper. I have already pointed out that the press releases spin the survey in ways that the paper itself belies. As it turns out, this spin is also present in the summary. Indeed, it is also present in the abstract, but slightly more subtly. Like I say, I will go through it one sentence at a time.

Women generally, and women of color specifically, have reported hostile workplace experiences in astronomy and related fields for some time.

This is, of course, true. As Kate Clancy has noted elsewhere, it's true of every field of human endeavor, and it is true of all races and genders. Everyone has experienced hostility at work. Work is done by humans in human environments and hostility is a human capacity. Indeed, humans are "capable" of hostility in both directions: they can dish it out and they can take. In short, the first sentence is a truism. The second sentence gestures at something less trivial.

However, little is known of the extent to which individuals in these disciplines experience inappropriate remarks, harassment, and assault.

It's true that this specific question hasn't been studied directly in astronomy. But there is some evidence to suggest that astronomy and related fields are not especially hostile places, specifically to women. (Women of color are, as is often noted, very underrepresented in astronomy and do seem to get lost in such studies.) One study found that women don't think about leaving the discipline more frequently than men; another found that, while they are 1.64 times more likely to have negative experiences than men, the average level of hostility was on the order of occasionally hearing a sexist joke. But it must be granted that the extent to which individuals have particular experiences is not well understood. The next sentence implies that this study will do something about this gap in our knowledge.

We conducted an internet-based survey of the workplace experiences of 474 astronomers and planetary scientists between 2011 and 2015.*

What they here imply is misleading since the paper explicitly states that "these data cannot provide a direct assessment of prevalence". That is, their "plain language summary", presumably intended for the non-expert (or journalist) gets the reader to think that they have done a study to gain the knowledge we lack, even though the authors are well aware that the study was precisely not designed to gain that knowledge, i.e., knowledge of the "the extent" (prevalence) of hostility in astronomy. That is of course also why they don't present general findings of prevalence, only comparisons of groups within the sample:

In this sample, in nearly every significant finding, women of color experienced the highest rates of negative workplace experiences, including harassment and assault.

This may seem like a quibble, but it is worth noting: the survey asked about "verbal harassment" and "physical harassment", not harassment and assault. They don't actually know the extent to which people in their sample were reporting assaults, except on a definition of assault in which any unwanted touching constitutes an assault (I'll discuss this in another post). Note that we are not told whether they experienced these things at a generally high or low rate, mainly because the study sets no threshold to make such judgments. The next sentence does report some alarming levels of hostility, however.

Further, women of color reported feeling unsafe in the workplace as a result of their gender or sex 40% of the time, and as a result of their race 28% of the time.

This sentence is simply a misinterpretation of the relevant result. It distorts and exaggerates their actual finding, as stated in the abstract: "40% of women of color reported feeling unsafe in the workplace as a result of their gender or sex, and 28% of women of color reported feeling unsafe as a result of their race." That is, it is not true that respondents felt unsafe 40% of the time; rather, 40% of respondents felt unsafe some of the time—or, more accurately, had felt unsafe at some time in the past. Indeed, they were specifically asked whether they had "ever felt unsafe" in their current position (see also footnote*). Answering "yes" here says nothing about how long or how often they felt unsafe. If 40% of respondents had ever felt unsafe, surely the population doesn't feel unsafe 40% of the time.

Finally, 18% of women of color, and 12% of white women, skipped professional events because they did not feel safe attending, identifying a significant loss of career opportunities due to a hostile climate.

As far as I can tell, this is a completely accurate summary of the result. I have said before that this is an important one, since it shows that there is a difference between feeling unsafe and doing something about it. It has been established in other studies that women feel more unsafe than men (even when both sexes feel very safe) and it stands to reason that this would translate into more absenteeism among women. It needs to be stressed that the survey found that only 9% of respondents reported "physical harassment", i.e., arguably an actual violation of personal safety. This suggests that women generally feel less safe than they are. This isn't a particularly surprising result, especially in a climate where women are told (by scientists and politicians) that harassment is "rampant". This study, of course, is one of the things that might be making women feel unsafe. Indeed, the authors say women are unsafe explicitly:

Our results suggest that certain community members may be at additional risk of hostile workplace experiences due to their gender, race, or both.

My standing objection to this way of putting it is that it does not account for the fact that "certain community members" would be in other environments if they were not doing astronomy. The authors don't give us any way to decide what the comparative ("additional") risk of hostile work experiences would be if they went into banking, politics or even another scientific discipline. As I said at the outset, there is a risk of hostility in any human environment. If a woman of color took away from this study that she best stay out of astronomy and choose another line of work then that would be a reasonable, if tragic, conclusion to draw from the "plain language" of this summary for the public. But, since the study itself eschews any claims about prevalence, it is not actually a reasonable conclusion to draw from the survey itself. I think that is a serious problem in the communication of this result to the public. It is not only astronomers that should take issue with this; the whole ear of the public is rankly abused.

Like I say, I will be looking at the paper more closely to support these various points of criticism and raise a few more in future posts. As is my custom, I will also be asking the authors for comment. To my knowledge, there has so far not been any serious criticism of the study in the press or the science blogs. It would be to Clancy's credit if she engaged with at least one critic as part of the discussion she so insists it is important to have. But I am not holding my breath.

____________
*I will cover this in a separate post, but as a commenter on my last post pointed out, it does not seem true that the survey asked respondents to confine themselves to the years 2011-2015. As I read the questionnaire, the respondents might well have thought they were being asked "Have you ever experienced...?" I believe that the authors thought they had limited the responses in this way. But I don't think the respondents would generally understand it as limited to five years.