Wednesday, April 01, 2015

Error and Blame, part 4

"Coming to recognize you are wrong is like coming to recognize you are sick. You feel bad long before you admit you have any of the symptoms and certainly long before you are willing to take your medicine." (Norman Maclean)

[Part 1 here]
[Part 3 here]

I have found that sensemaking scholars in general have a hard time engaging with direct criticism. It is as if telling them that they're wrong about something is merely a signal to them that you're not the sort of person they should be talking to. Many years ago, one young sensemaking scholar seriously told me that, unlike me, he was brought up to see the good in people before the bad. He suggested I start by pointing out all the good things in his paper before offering my criticisms. In what has turned out to be a longer series of posts than I originally anticipated, I'm beginning to understand my feelings about that reaction to my work.

Consider Karl Weick's own response. He has offered only two dismissive "rejoinders" (see here and here) to my critiques. He's probably said less than 500 words on the matter. Granted, I've been pretty hard on the guy. I've called him a plagiarist and a poor scholar. I've accused him of getting a great many empirical facts wrong, in part because he can't distinguish between evidence and anecdote. I've even suggested he can't tell an anecdote from a joke. It's understandable that he's unhappy with me, but it is simply unacceptable that, as a leading figure in his field, he does not directly address criticism of his work. If I'm right, he owes it to his readers to acknowledge it; if I'm wrong he owes it to my readers to set me straight. My reflections about his analysis of the Mann Gulch disaster is making it clear to me why I think this is so important.

In his analysis he blames the disaster on "multiple failures of leadership". His only source of facts about the event—Norman Maclean's Young Men and Fire, which he distorts beyond recognition—reaches the opposite conclusion, namely, that the tragedy was the result of a "conflagration" of cultural and natural, organizational and infernal factors, for which no one could be held responsible, and about which nothing, at the end of the day, could have reasonably been done. Weick's source says "the men did not panic"; Weick says they did. Weick's source says that their leader performed brilliantly; Weick says he lacked wisdom. The truth is that the young men were unlucky and their leader was a tragic figure. Wagner Dodge suffered the loss of thirteen young men under his command—an experience that is unimaginable to most of us, especially those of us who have chosen less physically dangerous work.

Tim Vogus has offered a reason sensemaking scholars don't think my criticism should be acknowledged: "the right/wrong frame that you apply to thinking about emotional ambivalence," he tells me, "is unnecessarily limited and restricts our ability to integrate work on mindfulness across the setting[s we are interested in]". At the end of his remarks, he expresses the hope that his response "helps to advance the conversation and moves us away from 'tussles' and binary thinking in favor of carefully contextualized arguments and collaborative synthesis." That is, it's the very idea that there is a "right" and "wrong" here that he thinks is counterproductive. He believes that there will always be some "context" in which, say, Weick's analysis might hold up. He thinks I should be looking for a "collaborative synthesis" of my understanding of events with Weick's, not a correction of his mistakes. He thinks, I guess, that I should begin with everything Weick gets right.*

It's just that, for all intents and purposes, Weick is completely wrong about what happened in Mann Gulch. I mean this in the sense that only saying that the whole thing took place in a snowstorm in the Swiss Alps could be worse. He's wrong about what happened and why it happened. He's wrong about why it happened because he's wrong about what happened. Indeed, one suspects that since he wanted the disaster to have resulted from a "collapse of sensemaking" he distorted his so-called "data" (which not he but Maclean collected) to fit his theory. He cooked the data. He fabricated events to suit his hypothesis. Whether he did this intentionally, or was just enormously careless, doesn't matter. It's the worst thing you can do as a scholar,** especially when you are sitting in judgment where lives were lost.

"Can't you take it down a notch, Thomas?" I imagine some will say. Well, remember that I'm not using language that is any stronger than Weick himself uses when talking about the firefighters. Weick uses the "right/wrong binary" to call Wag Dodge a failure as a leader. I'm using the same binary to say that Weick is a failure as a scholar. Dodge had to defend his actions before a tribunal at an inquest, and live with them the rest of his life. Maclean worked on the problem until his death, too, stopped him. It took eight months, by contrast, to get Weick's paper from conception to publication, and it has since been widely celebrated as the victory of his style over stuffy, "academic" scholarship. (See John Van Maanen's famous argument with Jeremy Pfeffer.)

One last thing: Weick ends his paper by invoking the authority of his source, stopping "just about where Maclean would want us to end." On the contrary, I think Maclean would have wanted the conversation to go on at least long enough to discover Weick's errors. Indeed, if my sense of Maclean's intellectual courage is correct, he would have wanted us to go on long enough discover Maclean's errors too. But since Weick presents his analysis as entirely supported by Maclean's, which he then gets wrong, we have not even begun that critical part of the process of interpretation. After all, Weick went public with his analysis only a few months after he had begun reading Maclean's book. And then it's as if he wants to declare his paper the last word on the matter.

*This is very reminiscent of the regard that some social scientists have for the work of Malcolm Gladwell. I wrote about this back in 2010. As Teppo Felin pointed out in that discussion, Kathleen Sutcliffe and Timothy Wintermute made the connection to Weick's work explicitly almost ten years ago.
**[Update, 03/04/15: I'm sure this strong claim is debatable and I'm very willing to have that debate. I definitely think it's worse to fabricate your data than to plagiarise another's prose; and it's worse to massage your data than to patchwrite your theory. When I say that what Weick has done is "the worst thing" a scholar can do I mean, in part, that the most serious error in social science is to mischaracterise what the people you have studied have done. But I also mean something else. It's pretty bad to publish as your own something you haven't written (i.e., to commit plagiarism) for the sake of advancing your career; but surely it's worse to publish a result that is not supported by your data (i.e., to engage in fabrication) for the sake promoting your theory. In the first case, you're misleading your employer about your abilities. In the second, you are misleading all of us about how the world works.]


Andrew Gelman said...


I don't know if this helps, but nobody I talk with about this stuff has ever heard of Karl Weick. He's an obscure figure and sensemaking is an obscure field. He does teach in a business school and so he's probably paid a lot, and I guess he's on speaking terms with various business executives, but I don't know how influential his stories are.

P.S. The University of Michigan is a public university so you can look up his pay. Salary for recent year is not given, I guess he's retired. Most recent year is 2011-2012, when he was paid . . . $298,500. Ulp!

Maybe if his stories had been true, the admin would've raised his salary to a round $300K?

Anyway, the point is . . . the b-school connection gives him the $ and some authority in our current culture. If he were doing the same thing but was, I dunno, a professor of social work somewhere getting paid $50,000 a year, the glamour wouldn't be there.

But, from my perspective, Weick only exists because I heard about him from you!

Thomas said...

I wouldn't call sensemaking an "obscure" field. It's just an ordinary specialty, a sub-discipline. Maybe a bit a like Bayesian statistics or Lorquian poetics. I'm sure that whatever names I know in either of those fields I wouldn't know except through you and Jonathan Mayhew, respectively.

As for Weick's influence, I'm increasingly confident that the "sensemaking mindset" has been widely adopted in business and public administration. I know it's anecdotal, but it seems to me that when Daniel Kahneman calls you a "famous organisational psychologist" then you're no longer obscure.

Also, he may actually be bigger in Europe than the U.S. said...

Thomas, your posts remind me of moments I have lived when attending philosophy seminars and the discourse turns to “possible worlds”. Since I have no formal training in the philosophy of science, these moments are rather disconcerting; I do not know what it means to admit the existence of worlds where propositions are true in some and not in others.

A Weickian world permits an epistemology where knowledge may not meet a standard of justified true belief. And there appears to be some social ontology in W world that accepts everything he writes and some less-well defined similar work (Gladwell,…) as a set of social facts that can be used in some institutionalized version of sociological research. In this possible world, it appears (e.g. from Czarniawska) that style counts as theory. If the theory consists of a set of propositions, then they are true if they are in the proper style.

A world which I (and you?) find more comfortable has a different epistemological tradition. Stories can be illustrative. Stories can count as justification of a proposition only if they are true (and replicable? and confirmed? and generalizable?). Fables, false accounts, and metaphors do not lead to justified true belief. In this world, analogical reasoning may have scientific value, but the account has some minimal requirements. The phenomena linked by the analogy must be close enough to justify the comparison. Can the phenomena be separated in a meaningful manner from the context? Is firefighting an analogue to university administration; that is, what are the phenomena (behaviors, facts, objects) that are analogous? Otherwise we are left with literary metaphors, which if they meet the requisite style, make for sound epistemology in W world.

Thomas said...

Yes, it's a peculiar world, at least on the face of it. Maybe that famous old story "Flatland" provides an analogy. It's as though there's a whole dimension missing in W world. Rhetorically, you can only ever move forward, "carefully" re-contextualizing the arguments other people have made to bring about a new "collaborative synthesis", but never backward, as it were, to correct them where they went wrong.

It's a world in which contradictions are not resolved (by rejecting one or both of the contradictory statements) but actively fostered (in the interested of greater "mindfulness" through "emotional ambivalence"). You are always allowed to add a new claim to the body of knowledge, no matter how poorly it fits in with what's already believed. You are never allowed to remove one.

Like you say, it's disorienting. In Flatland there's no room to "turn over"; in W world there's no way to "turn around". It feels terribly constraining.

Andrew Gelman said...


I like that analogy. We got 4 forward gears here but no reverse.