Monday, July 24, 2017

Was the CSWA survey confined to recent experiences?

As I mentioned in my last post, a commenter on my first post on its publication pointed out that it does not seem true that the CSWA Workplace Climate Survey asked respondents to confine themselves to the years 2011-2015 when answering the questions. Clancy et al. are, it should be noted, adamant on this score:

At this time, we want to remind the reader that the findings of this study cannot be attributed to events from long ago: respondents were asked to only report experiences they had had in their current career position over the last 5 years. The events and experiences reported in this survey happened between 2011 and 2015. (p. 11)

But here is what respondents were told in the cover letter to the survey:

The survey is designed to request information during the respondent's current position and previous position (if the respondent has changed positions within the last five years). [...]

Directions: Please respond to Section 1 regarding your personal experiences in your current position. If you have changed career positions in the past five years, then please also complete Section 2 regarding your personal experiences in your previous position.

This does not seem to me to confine responses to the years 2011-2015. (While it's ultimately a moot point, do notice that "the last 5 years" would cover 2010-2014, not 2011-2015, since the survey was administered in early 2015.) If a respondent had been working somewhere for, say, 15 years they would reasonably interpret this as covering the whole time there. If they have changed position within five years, they are being asked to answer also for the entire time of their previous position. In other words, they are being asked to think back at least five years, not at most.

This strikes me as a serious issue, especially given Clancy et al.'s "reminder" to the reader. If the frequency of experiences is really spread over, say, 20 years and the analysis assumes they are concentrated within 5 years, this will strongly distort the result. If anyone knows what I'm getting wrong here, do let me know in the comments.

No comments: