Something You Should Almost Be Able to Do in Your Sleep

Like you, I'm suspicious of the phrase "studies show". All too often we read a newspaper story about a study that shows something apparently interesting and then a few months later we read on Andrew Gelman's blog that its methodology is about as bad as "us[ing] a bathroom scale to weigh a feather—and the feather is resting loosely in the pouch of a kangaroo that is vigorously jumping up and down." These studies aren't showing anything at all, but they sometimes make a lot of noise in the "tabloids". An unfortunate amount of resources in the social sciences these days seems to be devoted to the production of disposable significance for use by the popular press.

Still, it is important to remember that, as a scholar, you are responsible for knowing what the current state of your field is. "Being an expert," as Timothy Burke once pointed out to Niall Ferguson, "means you guide an audience through what is known and said about a subject with some respect for the totality of that knowing and saying before favoring your own interpretation." That is, if your study seems to "show" something, it is your responsibility to know how that result fits into the larger body of results that constitutes your research tradition. If your N=40 study shows something that might help us to rid the world of racism and sexism in our sleep, you need to tell us what other work (and please not just your own previous work) either supports or counters yours. You need to show us that you know what, exactly, was believed by your peers about this effect (if anything) before you set out to measure it yourself.

Ezra Zuckerman has a good way of putting this: you need to construct a "compelling null". (Andrew would probably talk in terms of an "informative prior".) This is very important, you'll note, when we think of the "newsworthiness" of a result. If ten N=1000 studies have failed to find an effect of some kind in the data, your single N=40 study isn't going to immediately rock the scientific establishment on its heels. Or, at least, it shouldn't. Unfortunately, it will sometimes make the front page of a newspaper.

I'm saying all this as a sort of long-winded preamble to a simple exercise. Suppose that "everyone knows" that unconscious gender bias can be corrected through conscious "sensitivity training". This means both that (a) people generally believe this and (b) that it is largely true. Suppose you know enough about this subject to complete that first exercise I talked about. This means you can tell us exactly how prevalent unconscious bias about gender is in society and how much can be done about it through explicit training. Presumably, the problem isn't total (whatever that would mean) and things are getting better. Presumably, conscious efforts are part of the reason for the improvement. Presumably, the situation is also improving just because human societies are making general progress "towards the light", etc. Presumably, unconscious gender bias is a serious social problem that needs to be corrected. All this, merely to say that this is presumably a subject that is worthy of study and you, as an expert in the area, are capable of explaining this worthiness clearly and articulately, even to people who are already "on board".

Now, suppose you've conducted a study to see if the effect of conscious training can be reinforced by playing particular sounds to people while they sleep. The exercise I want to suggest is simply that of explaining, in a single paragraph, what you think your peer reader expects such a study to show. That reader will already know how big and lasting the effect of conscious training is. And they will also have some notion of what unconscious training can accomplish. Presumably, the effect of playing all kinds of sounds to people while sleeping has been tested in the interest of helping people lose weight, quit smoking, or cure stress? The effect here would be similar, and you, as an expert, should be able to write intelligently about it.

I'm being very specific about this because I've just read Nick Brown's "pinball" post, but I'm sure you can come up with an exercise that asks you to do the same thing of an effect you're studying. As an expert, it should be easy to do this. (The hard part of research is doing the study, making the discovery.) Remember to stay focused and specific, and anchored in the literature of your field, of course. Talk about the effect in a way that your peers (and therefore any potential reviewer) would recognize. Write at least six sentences and at most two-hundred words (in exactly 27 minutes) every now and then to make clear what you think your peer's think about what you think.

Tomorrow, I'll suggest an exercise to train your articulateness on that last point, i.e., your ability to speak your mind. I will not be suggesting you do very much in your sleep, except that I do recommend that you get some.